If that also does not help it would mean that target website is banning the requests. To overcome it you would need to use Crawlera our proxy rotator. Do refer Crawlera Articles to know about Crawlera.
If that also does not help it would mean that target website is banning the requests. To overcome it you would need to use Crawlera our proxy rotator. Do refer Crawlera Articles to know about Crawlera.
anigamy
I am testing a spyder but in the log i can see
[scrapy.spidermiddlewares.httperror] Ignoring response <403 >: HTTP status code is not handled or not allowed
I think this is because i cant set up the user-agent in portia.
Do you know how can i do?
You can add User Agent through the settings for the spider through UI as given in Customizing Scrapy Settings in Scrapy Cloud.
If that also does not help it would mean that target website is banning the requests. To overcome it you would need to use Crawlera our proxy rotator. Do refer Crawlera Articles to know about Crawlera.
Regards,
Thriveni Patil
thriveni
You can add User Agent through the settings for the spider through UI as given in Customizing Scrapy Settings in Scrapy Cloud.
If that also does not help it would mean that target website is banning the requests. To overcome it you would need to use Crawlera our proxy rotator. Do refer Crawlera Articles to know about Crawlera.
Regards,
Thriveni Patil
-
Unable to select Scrapy project in GitHub
-
ScrapyCloud can't call spider?
-
Unhandled error in Deferred
-
Item API - Filtering
-
newbie to web scraping but need data from zillow
-
ValueError: Invalid control character
-
Cancelling account
-
Best Practices
-
Beautifulsoup with ScrapingHub
-
Delete a project in ScrapingHub
See all 453 topics