With Scrapy, I have a job that has generated a few errors due to data in the response not being in the format the scraper was expecting. What is the best way to store those responses so i can go back and look at them afterwards? Is there a setting i can do for this?
With Scrapy, I have a job that has generated a few errors due to data in the response not being in the format the scraper was expecting. What is the best way to store those responses so i can go back and look at them afterwards? Is there a setting i can do for this?
https://app.scrapinghub.com/p/212215/addons/page_storage seems to be a bit different. In my case there wasn't an error with the response itself, it is simply that my code didn't handle it correctly.
0 Votes
vaz posted over 7 years ago Best Answer
Hi,
The best way is to check on the logs of your particular job and filter by errors, check this example:
https://app.scrapinghub.com/p/212215/1/32/log?filterAndHigher&filterType=error
Best,
Pablo
0 Votes
1 Comments
vaz posted over 7 years ago Answer
Hi,
The best way is to check on the logs of your particular job and filter by errors, check this example:
https://app.scrapinghub.com/p/212215/1/32/log?filterAndHigher&filterType=error
Best,
Pablo
0 Votes
Login to post a comment