The site we want to crawl only accepts India based IPs. Is India based IP option available in crawlera? I need to confirm this before going for the 7-day trial.
Thanks for the reply. Our use case is very specific. We are crawling only one site and serving to lots of users offering them an API. I have read in other forums that delays are introduced between requests. This seems to suggest it will affect concurrency and user experience for our use case. Do you have any such service?
0 Votes
nestorposted
over 6 years ago
Admin
Delays don't affect concurrency, concurrency is limited by your plan, e.g C10 has a limit of 10 concurrent request, C50 has 50, etc.
Do you have any such service? - Sorry, I don't understand your question here, could you elaborate further? What service are you referring to?
1 Votes
j
justin_pythonposted
over 6 years ago
At scale, we are looking at ~ 1000 concurrent requests under a minute, all serving 1000 different users from the site. All this on a single site. Assuming crawlera doesn't have 1000 different proxy servers, it means the same proxy would be used again under a minute. Will crawlera be able to handle such scale?
0 Votes
nestorposted
over 6 years ago
Admin
Yeah, that should be possible. Just note the plan you subscribe, because maximum the number of concurrent requests depends on the plan.
The site we want to crawl only accepts India based IPs. Is India based IP option available in crawlera? I need to confirm this before going for the 7-day trial.
0 Votes
nestor posted over 6 years ago Admin Best Answer
We do have proxies from India. You can signup to a Crawlera plan and use this guide https://support.scrapinghub.com/support/solutions/articles/22000188398-restricting-crawlera-ips-to-a-specific-region to create an account with just proxies from India.
1 Votes
5 Comments
nestor posted over 6 years ago Admin Answer
We do have proxies from India. You can signup to a Crawlera plan and use this guide https://support.scrapinghub.com/support/solutions/articles/22000188398-restricting-crawlera-ips-to-a-specific-region to create an account with just proxies from India.
1 Votes
justin_python posted over 6 years ago
Thanks for the reply. Our use case is very specific. We are crawling only one site and serving to lots of users offering them an API. I have read in other forums that delays are introduced between requests. This seems to suggest it will affect concurrency and user experience for our use case. Do you have any such service?
0 Votes
nestor posted over 6 years ago Admin
Delays don't affect concurrency, concurrency is limited by your plan, e.g C10 has a limit of 10 concurrent request, C50 has 50, etc.
Do you have any such service? - Sorry, I don't understand your question here, could you elaborate further? What service are you referring to?
1 Votes
justin_python posted over 6 years ago
At scale, we are looking at ~ 1000 concurrent requests under a minute, all serving 1000 different users from the site. All this on a single site. Assuming crawlera doesn't have 1000 different proxy servers, it means the same proxy would be used again under a minute. Will crawlera be able to handle such scale?
0 Votes
nestor posted over 6 years ago Admin
Yeah, that should be possible. Just note the plan you subscribe, because maximum the number of concurrent requests depends on the plan.
1 Votes
Login to post a comment