How can we help you today?

Start a new topic
Discussions Scrapy Cloud Technical Help
Answered

How to improve the request sending request?

Hi team,


I have used subscribed scrapy instance for a period of time, I found the average of request is 500 per hour, it's a little bit slow to my business, how can I improve the speed of request.


Btw, I am not saying the way of adding more instances because each of my website is using a spiede to scrape the data.


Looking forward to your kind response.

Best Answer

I would suggest you turn off AUTOTHROTTLE, it's not recommended to use Crawlera with AUTOTHROTTLE since that results in you not being in control of concurrent requests and most importantly "double throttling".

3 CommentsSorted by Oldest First
Answer

I would suggest you turn off AUTOTHROTTLE, it's not recommended to use Crawlera with AUTOTHROTTLE since that results in you not being in control of concurrent requests and most importantly "double throttling".

image


Yes, you are able to see I had already made it closed before I posted this question.


any further help.


as I described in another topic, 2000+ request almost cost me 8hrs? was that normal?


Regards

Login to post a comment
More topics in Technical Help
See all 453 topics