My spider doesn't work on scrapinghub, but it works on any IDE. The two errors appear:
[scrapy.downloadermiddlewares.robotstxt] Error downloading <GET https: //my_site/robots.txt>: Response.meta not available, this response is not tied to any request and [scrapy.downloadermiddlewares.robotstxt] Error downloading <GET https: //my_site/robots.txt>: Response.meta not available, this response is not tied to any request
My spider doesn't work on scrapinghub, but it works on any IDE. The two errors appear:
[scrapy.downloadermiddlewares.robotstxt] Error downloading <GET https: //my_site/robots.txt>: Response.meta not available, this response is not tied to any request and [scrapy.downloadermiddlewares.robotstxt] Error downloading <GET https: //my_site/robots.txt>: Response.meta not available, this response is not tied to any request
settings.py
0 Votes
0 Comments
Login to post a comment