If that also does not help it would mean that target website is banning the requests. To overcome it you would need to use Crawlera our proxy rotator. Do refer Crawlera Articles to know about Crawlera.
If that also does not help it would mean that target website is banning the requests. To overcome it you would need to use Crawlera our proxy rotator. Do refer Crawlera Articles to know about Crawlera.
I am testing a spyder but in the log i can see
[scrapy.spidermiddlewares.httperror] Ignoring response <403 >: HTTP status code is not handled or not allowed
I think this is because i cant set up the user-agent in portia.
Do you know how can i do?
0 Votes
thriveni posted almost 7 years ago Admin Best Answer
You can add User Agent through the settings for the spider through UI as given in Customizing Scrapy Settings in Scrapy Cloud.
If that also does not help it would mean that target website is banning the requests. To overcome it you would need to use Crawlera our proxy rotator. Do refer Crawlera Articles to know about Crawlera.
Regards,
Thriveni Patil
0 Votes
1 Comments
thriveni posted almost 7 years ago Admin Answer
You can add User Agent through the settings for the spider through UI as given in Customizing Scrapy Settings in Scrapy Cloud.
If that also does not help it would mean that target website is banning the requests. To overcome it you would need to use Crawlera our proxy rotator. Do refer Crawlera Articles to know about Crawlera.
Regards,
Thriveni Patil
0 Votes
Login to post a comment