2017-12-07 20:52:32 INFO Log opened. 2017-12-07 20:52:32 INFO [scrapy.log] Scrapy 1.4.0 started 2017-12-07 20:52:32 INFO [scrapy.utils.log] Scrapy 1.4.0 started (bot: farnell) 2017-12-07 20:52:32 INFO [scrapy.utils.log] Overridden settings: {'AUTOTHROTTLE_ENABLED': True, 'BOT_NAME': 'farnell', 'LOG_ENABLED': False, 'LOG_LEVEL': 'INFO', 'MEMUSAGE_LIMIT_MB': 5950, 'NEWSPIDER_MODULE': 'farnell.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['farnell.spiders'], 'STATS_CLASS': 'sh_scrapy.stats.HubStorageStatsCollector', 'TELNETCONSOLE_HOST': '0.0.0.0', 'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36'} 2017-12-07 20:52:32 INFO [scrapy.middleware] Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats', 'scrapy.extensions.spiderstate.SpiderState', 'scrapy.extensions.throttle.AutoThrottle', 'scrapy.extensions.debug.StackTraceDump', 'sh_scrapy.extension.HubstorageExtension'] 2017-12-07 20:52:32 INFO [scrapy.middleware] Enabled downloader middlewares: ['sh_scrapy.diskquota.DiskQuotaDownloaderMiddleware', 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats', 'sh_scrapy.middlewares.HubstorageDownloaderMiddleware'] 2017-12-07 20:52:32 INFO [scrapy.middleware] Enabled spider middlewares: ['sh_scrapy.diskquota.DiskQuotaSpiderMiddleware', 'sh_scrapy.middlewares.HubstorageSpiderMiddleware', 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2017-12-07 20:52:32 INFO [scrapy.middleware] Enabled item pipelines: [] 2017-12-07 20:52:32 INFO [scrapy.core.engine] Spider opened 2017-12-07 20:52:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 20:52:32 INFO TelnetConsole starting on 6023 2017-12-07 20:53:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 20:54:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 20:55:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 20:56:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 20:57:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 20:58:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 20:59:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 21:00:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 21:01:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 21:01:32 ERROR [scrapy.downloadermiddlewares.robotstxt] Error downloading : User timeout caused connection failure: Getting http://uk.farnell.com/robots.txt took longer than 180.0 seconds.. Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1297, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/local/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request defer.returnValue((yield download_func(request=request,spider=spider))) File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 651, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/local/lib/python3.6/site-packages/scrapy/core/downloader/handlers/http11.py", line 320, in _cb_timeout raise TimeoutError("Getting %s took longer than %s seconds." % (url, timeout)) twisted.internet.error.TimeoutError: User timeout caused connection failure: Getting http://uk.farnell.com/robots.txt took longer than 180.0 seconds.. 2017-12-07 21:02:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 21:03:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 21:04:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 21:05:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 21:06:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 21:07:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 21:08:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 21:09:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 21:10:32 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2017-12-07 21:10:32 ERROR [scrapy.core.scraper] Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1297, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/local/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request defer.returnValue((yield download_func(request=request,spider=spider))) File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 651, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/local/lib/python3.6/site-packages/scrapy/core/downloader/handlers/http11.py", line 320, in _cb_timeout raise TimeoutError("Getting %s took longer than %s seconds." % (url, timeout)) twisted.internet.error.TimeoutError: User timeout caused connection failure: Getting http://uk.farnell.com took longer than 180.0 seconds.. 2017-12-07 21:10:33 INFO [scrapy.core.engine] Closing spider (finished)