Scheduling Periodic Jobs

Modified on Wed, 26 Oct, 2022 at 3:06 PM

⚠️  Note: Starting Oct 10th, 2017 Periodic Jobs is a feature available only to Scrapy Cloud paid plans.


You can configure a Periodic Job to run a spider at specific intervals. To add a periodic job, navigate to the project and click the Periodic Jobs link on the left sidebar.


On the Periodic Jobs page click on the Add periodic job button, then provide the details of the required schedule. In the below example, the spider is being set to run every Thursday at 3.00 UTC time.


You can provide Tags and Arguments that should be used to launch the job.



Notes: 

1. There's a limit of 1000 periodic jobs per project and 10000 periodic jobs per organisation.
2. Assigning an arbitrary number of Scrapy Cloud Units for a periodic job to be run with is not currently possible. Please use Default job units section instead -- available in the settings of each project (app.zyte.com/p/<PROJECT_ID>/settings) 
or spider (app.zyte.com/p/<PROJECT_ID>/<SPIDER_ID>):

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select at least one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article