Is there an advantage in adding Crawlera to a scrapy cloud account if I only want to do light scraping?
Does the Scrapy cloud account gets banned regularly?
0 Votes
nestor posted
about 7 years ago
AdminBest Answer
The main advantage is to reduce the likelihood of bans by routing your requests through different IPs. A job in Scrapy Cloud uses a single IP, it might not only be banned but also throttled. Some websites might be harder to scrape than others, so it depends on that. I would suggest you start with a C10 at $25/month for 150,000 requests/month and 10 concurrent requests.
0 Votes
1 Comments
nestorposted
about 7 years ago
AdminAnswer
The main advantage is to reduce the likelihood of bans by routing your requests through different IPs. A job in Scrapy Cloud uses a single IP, it might not only be banned but also throttled. Some websites might be harder to scrape than others, so it depends on that. I would suggest you start with a C10 at $25/month for 150,000 requests/month and 10 concurrent requests.
Is there an advantage in adding Crawlera to a scrapy cloud account if I only want to do light scraping?
Does the Scrapy cloud account gets banned regularly?
0 Votes
nestor posted about 7 years ago Admin Best Answer
The main advantage is to reduce the likelihood of bans by routing your requests through different IPs. A job in Scrapy Cloud uses a single IP, it might not only be banned but also throttled. Some websites might be harder to scrape than others, so it depends on that. I would suggest you start with a C10 at $25/month for 150,000 requests/month and 10 concurrent requests.
0 Votes
1 Comments
nestor posted about 7 years ago Admin Answer
The main advantage is to reduce the likelihood of bans by routing your requests through different IPs. A job in Scrapy Cloud uses a single IP, it might not only be banned but also throttled. Some websites might be harder to scrape than others, so it depends on that. I would suggest you start with a C10 at $25/month for 150,000 requests/month and 10 concurrent requests.
0 Votes
Login to post a comment