My spider doesn't work on scrapinghub, but it works on any IDE. The two errors appear:
[scrapy.downloadermiddlewares.robotstxt] Error downloading <GET https: //my_site/robots.txt>: Response.meta not available, this response is not tied to any request and [scrapy.downloadermiddlewares.robotstxt] Error downloading <GET https: //my_site/robots.txt>: Response.meta not available, this response is not tied to any request
My spider doesn't work on scrapinghub, but it works on any IDE. The two errors appear:
[scrapy.downloadermiddlewares.robotstxt] Error downloading <GET https: //my_site/robots.txt>: Response.meta not available, this response is not tied to any request and [scrapy.downloadermiddlewares.robotstxt] Error downloading <GET https: //my_site/robots.txt>: Response.meta not available, this response is not tied to any request
settings.py
DOWNLOADER_MIDDLEWARES = {
'scrapy_crawlera.CrawleraMiddleware': 610,
'stack.middlewares.RobotsTxtMiddleware': 100,
'stack.middlewares.StackoverflowDownloaderMiddleware': 543,
}
0 Votes
0 Comments
Login to post a comment