How can we help you today?

Start a new topic
Discussions Scrapy Cloud Technical Help
Answered

Unique identifier for a scrapy job

Hi,


If I want to get one unique Id for each scrapy job I run, and use that Id as part of the item I yield which gets dump to the database after scraping.


Is this possible and what will be the id?

Best Answer

There's an ENV variable SHUB_JOBKEY, in the format "project_id/spider_id/job_id"

There's also job_key in the Python Scrapinghub client https://python-scrapinghub.readthedocs.io/en/latest/client/apidocs.html

1 Comment
Answer

There's an ENV variable SHUB_JOBKEY, in the format "project_id/spider_id/job_id"

There's also job_key in the Python Scrapinghub client https://python-scrapinghub.readthedocs.io/en/latest/client/apidocs.html

Login to post a comment
More topics in Technical Help
See all 446 topics