Scheduling back to back spiders in Scrapy cloud Periodic Jobs

Posted about 3 years ago by Lawrence Li

Post a topic
Un Answered
L
Lawrence Li

Hi, I just purchased 1 unit for Scrapy Cloud and want to take advantage of the periodic jobs functionality.  I have six spiders - and because I only have 1 unit, I want to run these spiders back to back.  What's the right way of scheduling these spiders?  Each of these spiders take from 2 min up to 45 min.  I've currently scheduled them to run every day at 8:01 AM, 8:02 AM, 8:03 AM, 8:04 AM, 8:05 AM, and 8:06 AM.  However, is this the right/preferred way of scheduling the spiders?  Is there a different way or will my configuration cause issues?

0 Votes


0 Comments

Login to post a comment