videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic

Add ability to offset Periodic jobs.

Currently we have the ability to schedule jobs every 1, 2, 3, 4, 5, 15, 30 minutes.

If a have alot of spiders running on every 15 minutes, they would all try to run at the exact same time and block each other since there's a limit to how many Units i have.


One way to work around this is do create 4 jobs with the exact minutes of the hour I want to run the spider on as to spread them out over the hour. However, it's not a very neat solutions since it requires 4 jobs instead of 1 per spider.


I would like to have to ability to in the scheduling tool also select an offset in minutes. So if I selected every 15 minutes and offset 3 minutes, it would run at :03, :18, :33 and :48.

Login to post a comment