I have used subscribed scrapy instance for a period of time, I found the average of request is 500 per hour, it's a little bit slow to my business, how can I improve the speed of request.
Btw, I am not saying the way of adding more instances because each of my website is using a spiede to scrape the data.
Looking forward to your kind response.
0 Votes
nestor posted
almost 5 years ago
AdminBest Answer
I would suggest you turn off AUTOTHROTTLE, it's not recommended to use Crawlera with AUTOTHROTTLE since that results in you not being in control of concurrent requests and most importantly "double throttling".
0 Votes
3 Comments
Sorted by
L
Lilu Caoposted
almost 5 years ago
Yes, you are able to see I had already made it closed before I posted this question.
any further help.
as I described in another topic, 2000+ request almost cost me 8hrs? was that normal?
Regards
0 Votes
L
Lilu Caoposted
almost 5 years ago
0 Votes
nestorposted
almost 5 years ago
AdminAnswer
I would suggest you turn off AUTOTHROTTLE, it's not recommended to use Crawlera with AUTOTHROTTLE since that results in you not being in control of concurrent requests and most importantly "double throttling".
Hi team,
I have used subscribed scrapy instance for a period of time, I found the average of request is 500 per hour, it's a little bit slow to my business, how can I improve the speed of request.
Btw, I am not saying the way of adding more instances because each of my website is using a spiede to scrape the data.
Looking forward to your kind response.
0 Votes
nestor posted almost 5 years ago Admin Best Answer
I would suggest you turn off AUTOTHROTTLE, it's not recommended to use Crawlera with AUTOTHROTTLE since that results in you not being in control of concurrent requests and most importantly "double throttling".
0 Votes
3 Comments
Lilu Cao posted almost 5 years ago
Yes, you are able to see I had already made it closed before I posted this question.
any further help.
as I described in another topic, 2000+ request almost cost me 8hrs? was that normal?
Regards
0 Votes
Lilu Cao posted almost 5 years ago
0 Votes
nestor posted almost 5 years ago Admin Answer
I would suggest you turn off AUTOTHROTTLE, it's not recommended to use Crawlera with AUTOTHROTTLE since that results in you not being in control of concurrent requests and most importantly "double throttling".
0 Votes
Login to post a comment