scrapy.downloadermiddlewares.robotstxt] Error downloading

Posted almost 4 years ago by levi_junior

Post a topic
Un Answered
l
levi_junior

My spider doesn't work on scrapinghub, but it works on any IDE. The two errors appear:

[scrapy.downloadermiddlewares.robotstxt] Error downloading <GET https: //my_site/robots.txt>: Response.meta not available, this response is not tied to any request and [scrapy.downloadermiddlewares.robotstxt] Error downloading <GET https: //my_site/robots.txt>: Response.meta not available, this response is not tied to any request


settings.py

DOWNLOADER_MIDDLEWARES = {
'scrapy_crawlera.CrawleraMiddleware': 610,
'stack.middlewares.RobotsTxtMiddleware': 100,
'stack.middlewares.StackoverflowDownloaderMiddleware': 543,

}

0 Votes


0 Comments

Login to post a comment