Question: Is API always returning response without 'items' key when there are 0 jobs? No matter if script failed because of an error/ 500 responses or just finished and no items were found?
The API returns only the fields that have been updated during runtime, it doesn't have a predefined set of fields, this is why items key is not returned if the item count is 0.
I've created a ticket as a feature request, but for now, if you need that key with 0 value then you will need to add it manually.
The API returns only the fields that have been updated during runtime, it doesn't have a predefined set of fields, this is why items key is not returned if the item count is 0.
I've created a ticket as a feature request, but for now, if you need that key with 0 value then you will need to add it manually.
L
Lukasz Jaworowski
said
over 5 years ago
Thank you, It's not urgent. I just wanted to make sure that it does not happen only in specific case like job finished because of error or something like this.
Lukasz Jaworowski
Hi,
I am using storage.scrapinghub.com/jobq/{proj_id}/lis to get list of jobs finished in specific period of time.
I am using startts, endts and state=finished filter.
The problem is sometimes 'items' key is missing.
Examples:
1 When there are items scraped.
{'close_reason': 'finished', 'pending_time': XXXX, 'spider': 'XXX', 'running_time': XXXX, 'state': 'finished', 'version': 'XXXX', 'finished_time': XXXX, 'ts': XXXX, 'elapsed': XXXX, 'key': 'XXXX/XXXX/XXXX', 'items': 18, 'pages': 1, 'logs': 17}
2 When there are no items scraped?
{'version': 'XXX', 'finished_time': XX, 'state': 'XXX', 'pending_time': XX, 'running_time': XXX, 'close_reason': 'finished', 'spider': 'XXX', 'ts': XXXX, 'elapsed': XXX, 'key': 'XXX/XX/XX', 'logs': 17, 'pages': 2} (no 'items' key)
Question: Is API always returning response without 'items' key when there are 0 jobs? No matter if script failed because of an error/ 500 responses or just finished and no items were found?
Is it impossible to get 'items':0 response?
There is no info about that field in official api https://doc.scrapinghub.com/api/jobq.html#jobq-project-id-list
The API returns only the fields that have been updated during runtime, it doesn't have a predefined set of fields, this is why items key is not returned if the item count is 0.
I've created a ticket as a feature request, but for now, if you need that key with 0 value then you will need to add it manually.
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstnestor
The API returns only the fields that have been updated during runtime, it doesn't have a predefined set of fields, this is why items key is not returned if the item count is 0.
I've created a ticket as a feature request, but for now, if you need that key with 0 value then you will need to add it manually.
Lukasz Jaworowski
Thank you, It's not urgent. I just wanted to make sure that it does not happen only in specific case like job finished because of error or something like this.
-
Unable to select Scrapy project in GitHub
-
ScrapyCloud can't call spider?
-
Unhandled error in Deferred
-
Item API - Filtering
-
newbie to web scraping but need data from zillow
-
ValueError: Invalid control character
-
Cancelling account
-
Best Practices
-
Beautifulsoup with ScrapingHub
-
Delete a project in ScrapingHub
See all 453 topics