Hi, I just purchased 1 unit for Scrapy Cloud and want to take advantage of the periodic jobs functionality. I have six spiders - and because I only have 1 unit, I want to run these spiders back to back. What's the right way of scheduling these spiders? Each of these spiders take from 2 min up to 45 min. I've currently scheduled them to run every day at 8:01 AM, 8:02 AM, 8:03 AM, 8:04 AM, 8:05 AM, and 8:06 AM. However, is this the right/preferred way of scheduling the spiders? Is there a different way or will my configuration cause issues?
Hi, I just purchased 1 unit for Scrapy Cloud and want to take advantage of the periodic jobs functionality. I have six spiders - and because I only have 1 unit, I want to run these spiders back to back. What's the right way of scheduling these spiders? Each of these spiders take from 2 min up to 45 min. I've currently scheduled them to run every day at 8:01 AM, 8:02 AM, 8:03 AM, 8:04 AM, 8:05 AM, and 8:06 AM. However, is this the right/preferred way of scheduling the spiders? Is there a different way or will my configuration cause issues?
0 Votes
0 Comments
Login to post a comment