Scrapy & Crawlera: not using all concurrent requests?
e
empyreone
started a topic
about 3 years ago
This has been an ongoing issue with crawlera... i never seem to reach the concurrent requests on my plan (it says i have 100). I have set CONCURRENT_REQUESTS = 100 and even tried 1024 in my settings.py and it does not change anything.
When i check Usage Stats on crawlera web app, it says i am only using 8 concurrency. So i'm paying for a plan that i'm not even using, even though i've look at all documentation and help questions and tried it. Please address this issue.
@nestor thanks for the response. I have enabled that and see that it works... but still not reaching my plan which is 100 concurrent req. I am reaching only 60, even though i've changed the settings to 100.
thriveni
said
about 3 years ago
I see the latest concurrency is around 99, we have also made some configuration changes for handling the load for this domain.
For any issues further, I suggest please contact Support team through Dashboard > Help > Contact Support team. Support is available for weekdays for paying customers. They would surely assist you with your queries or technical assistance.
empyreone
This has been an ongoing issue with crawlera... i never seem to reach the concurrent requests on my plan (it says i have 100). I have set CONCURRENT_REQUESTS = 100 and even tried 1024 in my settings.py and it does not change anything.
When i check Usage Stats on crawlera web app, it says i am only using 8 concurrency. So i'm paying for a plan that i'm not even using, even though i've look at all documentation and help questions and tried it. Please address this issue.
The setting CONCURRENT_REQUESTS_PER_DOMAIN has a value of 8 by default: https://docs.scrapy.org/en/latest/topics/settings.html#concurrent-requests-per-domain
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstnestor
The setting CONCURRENT_REQUESTS_PER_DOMAIN has a value of 8 by default: https://docs.scrapy.org/en/latest/topics/settings.html#concurrent-requests-per-domain
empyreone
@nestor thanks for the response. I have enabled that and see that it works... but still not reaching my plan which is 100 concurrent req. I am reaching only 60, even though i've changed the settings to 100.
thriveni
I see the latest concurrency is around 99, we have also made some configuration changes for handling the load for this domain.
For any issues further, I suggest please contact Support team through Dashboard > Help > Contact Support team. Support is available for weekdays for paying customers. They would surely assist you with your queries or technical assistance.
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 365 topics