Is there any socket connection limitation on free plan?
Posted almost 7 years ago by jxltom
Post a topicPeople who like this
Delete Comment
This post will be deleted permanently. Are you sure?
Code Snippet
Html
Html
Css
JavaScript
Sass
Xml
Ruby
PHP
Java
C#
C++
ObjectiveC
Perl
Python
VB
SQL
Generic Language
My spider has a high AUTOTHROTTLE_TARGET_CONCURRENCY and CONCURRENT_REQUESTS, the concurrent transferring requests should be high (in my local PC it is very high indeed). But in scrapyinghub, I found the concurrent transferring requests can not be larger than 4. All other active requests are waiting to be transferred.
So is there any limitation on maximum socket connection for one unit container?
Currently I am using the free plan.
THANKS a lot.
0 Votes
2 Comments
thriveni posted almost 7 years ago Admin
From the logs can see that AUTOTHROTTLE_ENABLED is set to True. For the concurrent requests to be used autothrottle needs to be disabled.
But do note to crawl the websites too fast as that would increase the probability of being blocked by target site or your Scrapinghub account getting suspended.
Regards,
Thriveni Patil
0 Votes
jxltom posted almost 7 years ago
Hi thriveni,
Thanks for your reply.
But I have set a high value of AUTOTHROTTLE_TARGET_CONCURRENCY , so even I enable AUTOTHROTTLE_ENABLED I should also have high concurrent requests which number should be the value of AUTOTHROTTLE_TARGET_CONCURRENCY.
Can you explain why I should disable autothrottle but not instead to set a high AUTOTHROTTLE_TARGET_CONCURRENCY to achieve concurrent requests?
Or is there any special treatment with autothrottle in Scrapinghub which does not respect the behavious of scrapy's default?
THANKS.
0 Votes
Login to post a comment