How can we help you today?

Start a new topic
Discussions Zyte Smart Proxy Manager (formerly Crawlera) Technical Help
Answered

Can I alter Crawler default request limits in proxy settings?

I want to crawl a few websites only and I am using C100 in my Scrapy settings. Can I alter the default request limits from 5 a second to something much longer? 


I need to crawl websites only once and it does not matter how long it takes as long as I get a result.


Thanks very much for any suggestions.

Best Answer

I'm not sure if setting a delay has anything to do with it. I'm converting your forum topic into a private ticket.

3 CommentsSorted by Oldest First

Those limits changes are only available to Crawlera Enterprise. Could you explain further what you're trying to accomplish? From what I understand you want to increase the delay between requests, is this correct?

Thanks for your prompt reply. I am trying to download a fixed portion of some websites for some classification work I am doing. I can access the URLs in my browser but my Scrapy crawler working through the Crawlera proxy settings is not able to access and download some of the site I am interested in.


So I reduced concurrent requests to domain to 1, activated auto throttle and set Download_Delay to 23 but still no access.


Any thoughts?

Answer

I'm not sure if setting a delay has anything to do with it. I'm converting your forum topic into a private ticket.

Login to post a comment
More topics in Technical Help
See all 395 topics