This seems to be a ban from target website. I see that you have Crawlera susbcription. Please enable the Crawlera addon through UI or use Crawlera middleware. Please refer our article Using Crawlera with Scrapy to know more about integrating Crawlera.
Kongo site is Javascript based site , crawlera does not render javascript hence there is no data in ""section._588b5_3MtNs > ul.b49ee_2pjyI > li.bbe45_3oExY > div > div.a2cf5_2S5q5 > div._4941f_1HCZm " .
You would need to use some headless browser like Selenium or Splash with Crawlera to render the page.
Please note that paying customers can get help from Support team and have a faster resolution. Please navigate to Dashboard > Help > Contact Support to create a Support ticket.
a
ayza
said
over 3 years ago
Hello Thriveni,
Its Critical Situation.
Please help to make proper spider file for crawl data from konga site. I have attached my spider code in attachment. Please review at once and let me suggest if i forgot anything.
Please check if the CSS/Xpath elements are correctly chosen for the Items. You can make use of Scrapy Shell to check if the CSS/Xpath expressions work as expected.
a
ayza
said
over 3 years ago
I have fixed the Authorization error as per as your suggestion.but still
i am unable to crawl data. Please view the spider file and logs messag,
and suggest if anything i missing.
This seems to be a ban from target website. I see that you have Crawlera susbcription. Please enable the Crawlera addon through UI or use Crawlera middleware. Please refer our article Using Crawlera with Scrapy to know more about integrating Crawlera.
ayza
Hello Team,
I have created a spider to crawl data from konga.com site but getting error like
[scrapy.spidermiddlewares.httperror] Ignoring response <403 https://www.konga.com/category/laptops-5230>: HTTP status code is not handled or not allowed
can anyone view this issue and suggest me any solutions.
Hello,
This seems to be a ban from target website. I see that you have Crawlera susbcription. Please enable the Crawlera addon through UI or use Crawlera middleware. Please refer our article Using Crawlera with Scrapy to know more about integrating Crawlera.
Regards,
Thriveni
- Oldest First
- Popular
- Newest First
Sorted by Newest Firstayza
Thank you very much for confirmation.
thriveni
Kongo site is Javascript based site , crawlera does not render javascript hence there is no data in ""section._588b5_3MtNs > ul.b49ee_2pjyI > li.bbe45_3oExY > div > div.a2cf5_2S5q5 > div._4941f_1HCZm " .
You would need to use some headless browser like Selenium or Splash with Crawlera to render the page.
Please note that paying customers can get help from Support team and have a faster resolution. Please navigate to Dashboard > Help > Contact Support to create a Support ticket.
ayza
Hello Thriveni,
Its Critical Situation.
Please help to make proper spider file for crawl data from konga site. I have attached my spider code in attachment. Please review at once and let me suggest if i forgot anything.
ayza
Hello Team,
I have created a spider to crawl data from konga site but unable to crawl any products. While spider successfully finished.
Can any one view my spider file and suggest ,what is mission in my code (spider file attached )
thriveni
Hello,
The requests were successful from Crawlera this time. The recent job stat shows response code of 200 https://app.scrapinghub.com/p/442308/77/6/stats.
Please check if the CSS/Xpath elements are correctly chosen for the Items. You can make use of Scrapy Shell to check if the CSS/Xpath expressions work as expected.
ayza
thriveni
407 indicates Authorization error. Please ensure the Crawlera API key is provided in Crawlera addon settings.
The Crawlera APIKey can be found in the setup instruction of the crawlera account like here https://app.scrapinghub.com/o/324974/crawlera/setup?username=ayza.
ayza
I have enabled the crawlera using below configuration (attachment file) , but still getting error.
[scrapy.spidermiddlewares.httperror] Ignoring response <407 https://www.konga.com/category/laptops-5230>: HTTP status code is not handled or not allowed
thriveni
Hello,
This seems to be a ban from target website. I see that you have Crawlera susbcription. Please enable the Crawlera addon through UI or use Crawlera middleware. Please refer our article Using Crawlera with Scrapy to know more about integrating Crawlera.
Regards,
Thriveni
aurish_hammad_hafeez
Hi,
Please check https://www.konga.com/robots.txt
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 395 topics