Start a new topic
Answered

Api response question: storage.scrapinghub.com/jobq/{proj_id}/list

Hi, 

I am using storage.scrapinghub.com/jobq/{proj_id}/lis  to get list of jobs finished in specific period of time. 

I am using startts, endts and state=finished filter. 

The problem is sometimes 'items' key is missing. 


Examples:

1 When there are items scraped.

{'close_reason': 'finished', 'pending_time': XXXX, 'spider': 'XXX', 'running_time': XXXX, 'state': 'finished', 'version': 'XXXX', 'finished_time': XXXX, 'ts': XXXX, 'elapsed': XXXX, 'key': 'XXXX/XXXX/XXXX', 'items': 18, 'pages': 1, 'logs': 17}


2 When there are no items scraped?

{'version': 'XXX', 'finished_time': XX, 'state': 'XXX', 'pending_time': XX, 'running_time': XXX, 'close_reason': 'finished', 'spider': 'XXX', 'ts': XXXX, 'elapsed': XXX, 'key': 'XXX/XX/XX', 'logs': 17, 'pages': 2} (no 'items' key)


Question: Is API always returning response without 'items' key when there are 0 jobs? No matter if script failed because of an error/ 500 responses or just finished and no items were found? 

Is it impossible to get 'items':0 response? 

There is no info about that field in official api  https://doc.scrapinghub.com/api/jobq.html#jobq-project-id-list 



Best Answer

The API returns only the fields that have been updated during runtime, it doesn't have a predefined set of fields, this is why items key is not returned if the item count is 0.

I've created a ticket as a feature request, but for now, if you need that key with 0 value then you will need to add it manually.


Answer

The API returns only the fields that have been updated during runtime, it doesn't have a predefined set of fields, this is why items key is not returned if the item count is 0.

I've created a ticket as a feature request, but for now, if you need that key with 0 value then you will need to add it manually.

Thank you, It's not urgent. I just wanted to make sure that it does not happen only in specific case like job finished because of error or something like this.

Login to post a comment