Start a new topic
Answered

Api response question: storage.scrapinghub.com/jobq/{proj_id}/list

Hi, 

I am using storage.scrapinghub.com/jobq/{proj_id}/lis  to get list of jobs finished in specific period of time. 

I am using startts, endts and state=finished filter. 

The problem is sometimes 'items' key is missing. 


Examples:

1 When there are items scraped.

{'close_reason': 'finished', 'pending_time': XXXX, 'spider': 'XXX', 'running_time': XXXX, 'state': 'finished', 'version': 'XXXX', 'finished_time': XXXX, 'ts': XXXX, 'elapsed': XXXX, 'key': 'XXXX/XXXX/XXXX', 'items': 18, 'pages': 1, 'logs': 17}


2 When there are no items scraped?

{'version': 'XXX', 'finished_time': XX, 'state': 'XXX', 'pending_time': XX, 'running_time': XXX, 'close_reason': 'finished', 'spider': 'XXX', 'ts': XXXX, 'elapsed': XXX, 'key': 'XXX/XX/XX', 'logs': 17, 'pages': 2} (no 'items' key)


Question: Is API always returning response without 'items' key when there are 0 jobs? No matter if script failed because of an error/ 500 responses or just finished and no items were found? 

Is it impossible to get 'items':0 response? 

There is no info about that field in official api  https://doc.scrapinghub.com/api/jobq.html#jobq-project-id-list 



Best Answer

The API returns only the fields that have been updated during runtime, it doesn't have a predefined set of fields, this is why items key is not returned if the item count is 0.

I've created a ticket as a feature request, but for now, if you need that key with 0 value then you will need to add it manually.


Thank you, It's not urgent. I just wanted to make sure that it does not happen only in specific case like job finished because of error or something like this.

Answer

The API returns only the fields that have been updated during runtime, it doesn't have a predefined set of fields, this is why items key is not returned if the item count is 0.

I've created a ticket as a feature request, but for now, if you need that key with 0 value then you will need to add it manually.

Login to post a comment