Is there an advantage in adding Crawlera to a scrapy cloud account if I only want to do light scraping?
Does the Scrapy cloud account gets banned regularly?
Best Answer
n
nestor
said
over 6 years ago
The main advantage is to reduce the likelihood of bans by routing your requests through different IPs. A job in Scrapy Cloud uses a single IP, it might not only be banned but also throttled. Some websites might be harder to scrape than others, so it depends on that. I would suggest you start with a C10 at $25/month for 150,000 requests/month and 10 concurrent requests.
1 Comment
nestor
said
over 6 years ago
Answer
The main advantage is to reduce the likelihood of bans by routing your requests through different IPs. A job in Scrapy Cloud uses a single IP, it might not only be banned but also throttled. Some websites might be harder to scrape than others, so it depends on that. I would suggest you start with a C10 at $25/month for 150,000 requests/month and 10 concurrent requests.
sa_scrapy
Is there an advantage in adding Crawlera to a scrapy cloud account if I only want to do light scraping?
Does the Scrapy cloud account gets banned regularly?
The main advantage is to reduce the likelihood of bans by routing your requests through different IPs. A job in Scrapy Cloud uses a single IP, it might not only be banned but also throttled. Some websites might be harder to scrape than others, so it depends on that. I would suggest you start with a C10 at $25/month for 150,000 requests/month and 10 concurrent requests.
nestor
The main advantage is to reduce the likelihood of bans by routing your requests through different IPs. A job in Scrapy Cloud uses a single IP, it might not only be banned but also throttled. Some websites might be harder to scrape than others, so it depends on that. I would suggest you start with a C10 at $25/month for 150,000 requests/month and 10 concurrent requests.
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 395 topics