Can I alter Crawler default request limits in proxy settings?

Posted about 7 years ago by Chris Bose

Post a topic

Answered

Chris Bose

I want to crawl a few websites only and I am using C100 in my Scrapy settings. Can I alter the default request limits from 5 a second to something much longer?

I need to crawl websites only once and it does not matter how long it takes as long as I get a result.

Thanks very much for any suggestions.

0 Votes

nestor posted about 7 years ago Admin Best Answer

I'm not sure if setting a delay has anything to do with it. I'm converting your forum topic into a private ticket.

0 Votes

3 Comments

nestor posted about 7 years ago Admin

Those limits changes are only available to Crawlera Enterprise. Could you explain further what you're trying to accomplish? From what I understand you want to increase the delay between requests, is this correct?

0 Votes

Chris Bose posted about 7 years ago

Thanks for your prompt reply. I am trying to download a fixed portion of some websites for some classification work I am doing. I can access the URLs in my browser but my Scrapy crawler working through the Crawlera proxy settings is not able to access and download some of the site I am interested in.

So I reduced concurrent requests to domain to 1, activated auto throttle and set Download_Delay to 23 but still no access.

Any thoughts?

0 Votes

nestor posted about 7 years ago Admin Answer

I'm not sure if setting a delay has anything to do with it. I'm converting your forum topic into a private ticket.

0 Votes