videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic
Answered

Can I alter Crawler default request limits in proxy settings?

I want to crawl a few websites only and I am using C100 in my Scrapy settings. Can I alter the default request limits from 5 a second to something much longer? 


I need to crawl websites only once and it does not matter how long it takes as long as I get a result.


Thanks very much for any suggestions.


Best Answer

I'm not sure if setting a delay has anything to do with it. I'm converting your forum topic into a private ticket.


Those limits changes are only available to Crawlera Enterprise. Could you explain further what you're trying to accomplish? From what I understand you want to increase the delay between requests, is this correct?

Thanks for your prompt reply. I am trying to download a fixed portion of some websites for some classification work I am doing. I can access the URLs in my browser but my Scrapy crawler working through the Crawlera proxy settings is not able to access and download some of the site I am interested in.


So I reduced concurrent requests to domain to 1, activated auto throttle and set Download_Delay to 23 but still no access.


Any thoughts?

Answer

I'm not sure if setting a delay has anything to do with it. I'm converting your forum topic into a private ticket.

Login to post a comment