Start a new topic
Answered

spider running on scrapy slow

Hi there, 


I notice sometimes spider runs really slow, like multiple times of what it takes than if I run on my local machine.


I only have one spider running in one unit and doing very simple stuff. Also when I run some bit more complicated spider like a docker container it takes longer like 5 minutes for a simple crawl.


Is there any way to speed it up?


Best,




Best Answer

Scrapy Cloud has AUTOTHROTTLE enabled by default, you'll need to disable it in the UI settings and restart your jobs.

1 Comment

Answer

Scrapy Cloud has AUTOTHROTTLE enabled by default, you'll need to disable it in the UI settings and restart your jobs.

Login to post a comment