Scrapinghub: concurrent requests settings appear not to be respected
R
Rodrigo Palacios
started a topic
over 4 years ago
Hi scrapinghub team, we have a high priority crawl job which needs to finish by tomorrow. We brought up CONCURRENT_REQUESTS and CONCURRENT_REQUESTS_PER_DOMAIN settings to 100 each, and we're still only seeing a rough average of 27 requests per min.
Observing the request log, it also appears that our requests aren't being sent out in parallel. Any help is appreciated.
Best Answer
A
Adriana Anghel
said
over 4 years ago
Hi! I have converted this into a support ticket - you should have received an email notification. One of our Support Engineers will be in touch soon. Bear in mind that while you have an active subscription with us, you are able to open support tickets directly from your dashboard, under Help - Contact Support. We respond to support tickets within one business day. The forum is for customers who don't have an active subscription with us.
1 Comment
Adriana Anghel
said
over 4 years ago
Answer
Hi! I have converted this into a support ticket - you should have received an email notification. One of our Support Engineers will be in touch soon. Bear in mind that while you have an active subscription with us, you are able to open support tickets directly from your dashboard, under Help - Contact Support. We respond to support tickets within one business day. The forum is for customers who don't have an active subscription with us.
Rodrigo Palacios
Hi scrapinghub team, we have a high priority crawl job which needs to finish by tomorrow. We brought up CONCURRENT_REQUESTS and CONCURRENT_REQUESTS_PER_DOMAIN settings to 100 each, and we're still only seeing a rough average of 27 requests per min.
Observing the request log, it also appears that our requests aren't being sent out in parallel. Any help is appreciated.
Hi! I have converted this into a support ticket - you should have received an email notification. One of our Support Engineers will be in touch soon. Bear in mind that while you have an active subscription with us, you are able to open support tickets directly from your dashboard, under Help - Contact Support. We respond to support tickets within one business day. The forum is for customers who don't have an active subscription with us.
Adriana Anghel
Hi! I have converted this into a support ticket - you should have received an email notification. One of our Support Engineers will be in touch soon. Bear in mind that while you have an active subscription with us, you are able to open support tickets directly from your dashboard, under Help - Contact Support. We respond to support tickets within one business day. The forum is for customers who don't have an active subscription with us.
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 381 topics