Question: Is API always returning response without 'items' key when there are 0 jobs? No matter if script failed because of an error/ 500 responses or just finished and no items were found?
The API returns only the fields that have been updated during runtime, it doesn't have a predefined set of fields, this is why items key is not returned if the item count is 0.
I've created a ticket as a feature request, but for now, if you need that key with 0 value then you will need to add it manually.
Thank you, It's not urgent. I just wanted to make sure that it does not happen only in specific case like job finished because of error or something like this.
nestor
said
over 5 years ago
Answer
The API returns only the fields that have been updated during runtime, it doesn't have a predefined set of fields, this is why items key is not returned if the item count is 0.
I've created a ticket as a feature request, but for now, if you need that key with 0 value then you will need to add it manually.
Lukasz Jaworowski
Hi,
I am using storage.scrapinghub.com/jobq/{proj_id}/lis to get list of jobs finished in specific period of time.
I am using startts, endts and state=finished filter.
The problem is sometimes 'items' key is missing.
Examples:
1 When there are items scraped.
{'close_reason': 'finished', 'pending_time': XXXX, 'spider': 'XXX', 'running_time': XXXX, 'state': 'finished', 'version': 'XXXX', 'finished_time': XXXX, 'ts': XXXX, 'elapsed': XXXX, 'key': 'XXXX/XXXX/XXXX', 'items': 18, 'pages': 1, 'logs': 17}
2 When there are no items scraped?
{'version': 'XXX', 'finished_time': XX, 'state': 'XXX', 'pending_time': XX, 'running_time': XXX, 'close_reason': 'finished', 'spider': 'XXX', 'ts': XXXX, 'elapsed': XXX, 'key': 'XXX/XX/XX', 'logs': 17, 'pages': 2} (no 'items' key)
Question: Is API always returning response without 'items' key when there are 0 jobs? No matter if script failed because of an error/ 500 responses or just finished and no items were found?
Is it impossible to get 'items':0 response?
There is no info about that field in official api https://doc.scrapinghub.com/api/jobq.html#jobq-project-id-list
The API returns only the fields that have been updated during runtime, it doesn't have a predefined set of fields, this is why items key is not returned if the item count is 0.
I've created a ticket as a feature request, but for now, if you need that key with 0 value then you will need to add it manually.
- Oldest First
- Popular
- Newest First
Sorted by Newest FirstLukasz Jaworowski
Thank you, It's not urgent. I just wanted to make sure that it does not happen only in specific case like job finished because of error or something like this.
nestor
The API returns only the fields that have been updated during runtime, it doesn't have a predefined set of fields, this is why items key is not returned if the item count is 0.
I've created a ticket as a feature request, but for now, if you need that key with 0 value then you will need to add it manually.
-
Unable to select Scrapy project in GitHub
-
ScrapyCloud can't call spider?
-
Unhandled error in Deferred
-
Item API - Filtering
-
newbie to web scraping but need data from zillow
-
ValueError: Invalid control character
-
Cancelling account
-
Best Practices
-
Beautifulsoup with ScrapingHub
-
Delete a project in ScrapingHub
See all 453 topics