What the settings for scrapy and crawlera?

Posted almost 6 years ago by aserhii

Post a topic
Answered
a
aserhii

What the settings I need add to scrapy project if I want to use crawlera? for example c 50 plan? If I want quick speed scrape webpages?

0 Votes

nestor

nestor posted almost 6 years ago Admin Best Answer

Start with Crawlera best practices:  https://support.scrapinghub.com/support/solutions/articles/22000188397-crawlera-best-practices 
And if you have any issues, you can open a ticket from the dashboard: Help > Contact support.

0 Votes


2 Comments

Sorted by
thriveni

thriveni posted almost 6 years ago Admin

You can increase the speed my increasing concurrency. You can set concurrency using CONCURRNT_REQUESTS and disabling Autothrottle as given in https://support.scrapinghub.com/solution/articles/22000188399-using-crawlera-with-scrapy.

0 Votes

nestor

nestor posted almost 6 years ago Admin Answer

Start with Crawlera best practices:  https://support.scrapinghub.com/support/solutions/articles/22000188397-crawlera-best-practices 
And if you have any issues, you can open a ticket from the dashboard: Help > Contact support.

0 Votes

Login to post a comment