Spider is Returning IndexError for some URLs and does not return all URL Items
s
shaping2startups
started a topic
about 5 years ago
I am summarising the points in a new thread:
While running a spider with Scrapinghub, I am receiving the following error for a number of requests:
IndexError: list index out of range
Does anyone know what could be the reason for this error as other urls in the list are returning results?
Next to this, the search only seems to return items for about half of the urls in my list (in the given case, I have 500+ urls in my list but only 238 items returned). It keeps sending out requests for the remaining urls but does not return/add any additional items. What could that relate to?
shaping2startups
I am summarising the points in a new thread:
While running a spider with Scrapinghub, I am receiving the following error for a number of requests:
IndexError: list index out of range
Does anyone know what could be the reason for this error as other urls in the list are returning results?
Next to this, the search only seems to return items for about half of the urls in my list (in the given case, I have 500+ urls in my list but only 238 items returned). It keeps sending out requests for the remaining urls but does not return/add any additional items. What could that relate to?