Start a new topic
Answered

Unique identifier for a scrapy job

Hi,


If I want to get one unique Id for each scrapy job I run, and use that Id as part of the item I yield which gets dump to the database after scraping.


Is this possible and what will be the id?


Best Answer

There's an ENV variable SHUB_JOBKEY, in the format "project_id/spider_id/job_id"

There's also job_key in the Python Scrapinghub client https://python-scrapinghub.readthedocs.io/en/latest/client/apidocs.html

1 Comment

Answer

There's an ENV variable SHUB_JOBKEY, in the format "project_id/spider_id/job_id"

There's also job_key in the Python Scrapinghub client https://python-scrapinghub.readthedocs.io/en/latest/client/apidocs.html

Login to post a comment