3 things to evaluate to discover the best way for you to access web data.Register for the webinar
Start a new topic
Answered

Need a quick help

Hello,


I wanted to use put my code on to scrapy cloud and run that lakhs of times in single day and the execution time of my code is taking an avg of 45sec.. is it possible to do in scrapy cloud? If yes can you tell me the procedure?


Best Answer

Hello,


You can schedule periodic jobs to run the spider at specific intervals as given in https://support.scrapinghub.com/a/solutions/articles/22000200419-scheduling-periodic-jobs. Please note that this feature is available only for Scrapy cloud paid plans. You need to be subscribed to atleast 1 Scrapy Unit to schedule periodic jobs.

1 Comment

Answer

Hello,


You can schedule periodic jobs to run the spider at specific intervals as given in https://support.scrapinghub.com/a/solutions/articles/22000200419-scheduling-periodic-jobs. Please note that this feature is available only for Scrapy cloud paid plans. You need to be subscribed to atleast 1 Scrapy Unit to schedule periodic jobs.

Login to post a comment