I notice sometimes spider runs really slow, like multiple times of what it takes than if I run on my local machine.
I only have one spider running in one unit and doing very simple stuff. Also when I run some bit more complicated spider like a docker container it takes longer like 5 minutes for a simple crawl.
Is there any way to speed it up?
Best,
0 Votes
nestor posted
over 5 years ago
AdminBest Answer
Scrapy Cloud has AUTOTHROTTLE enabled by default, you'll need to disable it in the UI settings and restart your jobs.
0 Votes
1 Comments
nestorposted
over 5 years ago
AdminAnswer
Scrapy Cloud has AUTOTHROTTLE enabled by default, you'll need to disable it in the UI settings and restart your jobs.
Hi there,
I notice sometimes spider runs really slow, like multiple times of what it takes than if I run on my local machine.
I only have one spider running in one unit and doing very simple stuff. Also when I run some bit more complicated spider like a docker container it takes longer like 5 minutes for a simple crawl.
Is there any way to speed it up?
Best,
0 Votes
nestor posted over 5 years ago Admin Best Answer
Scrapy Cloud has AUTOTHROTTLE enabled by default, you'll need to disable it in the UI settings and restart your jobs.
0 Votes
1 Comments
nestor posted over 5 years ago Admin Answer
Scrapy Cloud has AUTOTHROTTLE enabled by default, you'll need to disable it in the UI settings and restart your jobs.
0 Votes
Login to post a comment