I notice sometimes spider runs really slow, like multiple times of what it takes than if I run on my local machine.
I only have one spider running in one unit and doing very simple stuff. Also when I run some bit more complicated spider like a docker container it takes longer like 5 minutes for a simple crawl.
Is there any way to speed it up?
Best,
Best Answer
n
nestor
said
about 5 years ago
Scrapy Cloud has AUTOTHROTTLE enabled by default, you'll need to disable it in the UI settings and restart your jobs.
1 Comment
nestor
said
about 5 years ago
Answer
Scrapy Cloud has AUTOTHROTTLE enabled by default, you'll need to disable it in the UI settings and restart your jobs.
Yingke Yu
Hi there,
I notice sometimes spider runs really slow, like multiple times of what it takes than if I run on my local machine.
I only have one spider running in one unit and doing very simple stuff. Also when I run some bit more complicated spider like a docker container it takes longer like 5 minutes for a simple crawl.
Is there any way to speed it up?
Best,
Scrapy Cloud has AUTOTHROTTLE enabled by default, you'll need to disable it in the UI settings and restart your jobs.
nestor
Scrapy Cloud has AUTOTHROTTLE enabled by default, you'll need to disable it in the UI settings and restart your jobs.
-
Unable to select Scrapy project in GitHub
-
ScrapyCloud can't call spider?
-
Unhandled error in Deferred
-
Item API - Filtering
-
newbie to web scraping but need data from zillow
-
ValueError: Invalid control character
-
Cancelling account
-
Best Practices
-
Beautifulsoup with ScrapingHub
-
Delete a project in ScrapingHub
See all 453 topics