In addition to Scrapy spiders, you can also run custom, standalone python scripts on Scrapy Cloud. They need to be declared in the scripts
section of your project setup.py
file.
⚠ Note that the project deployed still needs to be a Scrapy project. This is a limitation that will be removed in the future.
Here is a setup.py
example for a project that ships a hello.py
script:
from setuptools import setup, find_packages
setup(
name = 'myproject',
version = '1.0',
packages = find_packages(),
scripts = ['bin/hello.py'],
entry_points = {'scrapy': ['settings = myproject.settings']},
)
After you deploy your project, you will see the py:hello.py
script on the Zyte dashboard, in the Run pop-up dialog and in the Add periodic job pop-up dialog.
It’s also possible to schedule a script via the Scrapy Cloud API:
curl -u API_KEY: -X POST https://app.zyte.com/api/schedule.json -d "project=123" -d "spider=py:hello.py" -d "cmd_args=-a --loglevel=10 x y"
And with the python-scrapinghub library:
from scrapinghub import Connection
conn = Connection('API_KEY')
project = conn[123]
project.schedule('py:hello.py', cmd_args='-a --loglevel=10 x y')
Was this article helpful?
That’s Great!
Thank you for your feedback
Sorry! We couldn't be helpful
Thank you for your feedback
Feedback sent
We appreciate your effort and will try to fix the article