2019-12-04 12:37:32 INFO Log opened. 2019-12-04 12:37:32 INFO [scrapy.log] Scrapy 1.3.3 started 2019-12-04 12:37:32 WARNING [sh_scrapy.settings] Addon import error spidermon_dash.extensions.SpidermonDash: No module named 'core' 2019-12-04 12:37:32 INFO [scrapy.utils.log] Scrapy 1.3.3 started (bot: mdx_crawl) 2019-12-04 12:37:32 INFO [scrapy.utils.log] Overridden settings: {'AUTOTHROTTLE_ENABLED': True, 'BOT_NAME': 'mdx_crawl', 'FEED_EXPORT_ENCODING': 'utf-8', 'LOG_ENABLED': False, 'LOG_LEVEL': 'INFO', 'MEMUSAGE_ENABLED': True, 'MEMUSAGE_LIMIT_MB': 950, 'NEWSPIDER_MODULE': 'mdx_crawl.spiders', 'SPIDER_MODULES': ['mdx_crawl.spiders'], 'STATS_CLASS': 'sh_scrapy.stats.HubStorageStatsCollector', 'TELNETCONSOLE_HOST': '0.0.0.0'} 2019-12-04 12:37:33 INFO [scrapy.middleware] Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.logstats.LogStats', 'scrapy.extensions.spiderstate.SpiderState', 'scrapy.extensions.throttle.AutoThrottle', 'scrapy.extensions.debug.StackTraceDump', 'sh_scrapy.extension.HubstorageExtension'] 2019-12-04 12:37:33 INFO [scrapy.middleware] Enabled downloader middlewares: ['sh_scrapy.diskquota.DiskQuotaDownloaderMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats', 'sh_scrapy.middlewares.HubstorageDownloaderMiddleware'] 2019-12-04 12:37:33 INFO [scrapy.middleware] Enabled spider middlewares: ['sh_scrapy.diskquota.DiskQuotaSpiderMiddleware', 'sh_scrapy.middlewares.HubstorageSpiderMiddleware', 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2019-12-04 12:37:33 INFO [scrapy.middleware] Enabled item pipelines: [] 2019-12-04 12:37:33 INFO [scrapy.core.engine] Spider opened 2019-12-04 12:37:33 INFO [scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2019-12-04 12:37:33 INFO TelnetConsole starting on 6023 2019-12-04 12:38:33 INFO [scrapy.extensions.logstats] Crawled 65 pages (at 65 pages/min), scraped 55 items (at 55 items/min) 2019-12-04 12:39:33 INFO [scrapy.extensions.logstats] Crawled 137 pages (at 72 pages/min), scraped 128 items (at 73 items/min) 2019-12-04 12:40:33 INFO [scrapy.extensions.logstats] Crawled 217 pages (at 80 pages/min), scraped 201 items (at 73 items/min) 2019-12-04 12:41:33 INFO [scrapy.extensions.logstats] Crawled 291 pages (at 74 pages/min), scraped 275 items (at 74 items/min) 2019-12-04 12:42:33 INFO [scrapy.extensions.logstats] Crawled 372 pages (at 81 pages/min), scraped 348 items (at 73 items/min) 2019-12-04 12:43:33 INFO [scrapy.extensions.logstats] Crawled 430 pages (at 58 pages/min), scraped 407 items (at 59 items/min) 2019-12-04 12:44:33 INFO [scrapy.extensions.logstats] Crawled 509 pages (at 79 pages/min), scraped 478 items (at 71 items/min) 2019-12-04 12:45:20 ERROR --- Logging error --- 2019-12-04 12:45:20 ERROR Unhandled error in Deferred: 2019-12-04 12:45:20 ERROR --- Logging error --- 2019-12-04 12:45:20 CRITICAL [twisted] Unhandled error in Deferred: 2019-12-04 12:45:20 ERROR Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1199, in run self.mainLoop() File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1208, in mainLoop self.runUntilCurrent() File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 828, in runUntilCurrent call.func(*call.args, **call.kw) File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 671, in _tick taskObj._oneWorkUnit() --- --- File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 517, in _oneWorkUnit result = next(self._iterator) File "/usr/local/lib/python3.6/site-packages/scrapy/utils/defer.py", line 63, in work = (callable(elem, *args, **named) for elem in iterable) File "/usr/local/lib/python3.6/site-packages/scrapy/utils/defer.py", line 106, in iter_errback errback(failure.Failure(), *a, **kw) File "/usr/local/lib/python3.6/site-packages/scrapy/core/scraper.py", line 158, in handle_spider_error extra={'spider': spider} File "/usr/local/lib/python3.6/logging/__init__.py", line 1337, in error self._log(ERROR, msg, args, **kwargs) File "/usr/local/lib/python3.6/logging/__init__.py", line 1444, in _log self.handle(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 1454, in handle self.callHandlers(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 1516, in callHandlers hdlr.handle(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 865, in handle self.emit(record) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/log.py", line 94, in emit self.handleError(record) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/log.py", line 100, in handleError super(HubstorageLogHandler, self).handleError(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 918, in handleError traceback.print_exception(t, v, tb, None, sys.stderr) File "/usr/local/lib/python3.6/traceback.py", line 104, in print_exception type(value), value, tb, limit=limit).format(chain=chain): File "/usr/local/lib/python3.6/traceback.py", line 522, in __init__ self._load_lines() File "/usr/local/lib/python3.6/traceback.py", line 534, in _load_lines self.__context__._load_lines() File "/usr/local/lib/python3.6/traceback.py", line 534, in _load_lines self.__context__._load_lines() File "/usr/local/lib/python3.6/traceback.py", line 532, in _load_lines frame.line File "/usr/local/lib/python3.6/traceback.py", line 286, in line self._line = linecache.getline(self.filename, self.lineno).strip() File "/usr/local/lib/python3.6/linecache.py", line 16, in getline lines = getlines(filename, module_globals) File "/usr/local/lib/python3.6/linecache.py", line 47, in getlines return updatecache(filename, module_globals) File "/usr/local/lib/python3.6/linecache.py", line 103, in updatecache data = cache[filename][0]() builtins.UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 2578: invalid start byte 2019-12-04 12:45:20 ERROR Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 517, in _oneWorkUnit result = next(self._iterator) 2019-12-04 12:45:20 ERROR StopIteration 2019-12-04 12:45:20 ERROR During handling of the above exception, another exception occurred: 2019-12-04 12:45:20 ERROR Traceback (most recent call last): File "/usr/local/lib/python3.6/linecache.py", line 95, in updatecache stat = os.stat(fullname) 2019-12-04 12:45:20 ERROR NotADirectoryError: [Errno 20] Not a directory: '/app/__main__.egg/mdx_crawl/spiders/mojalekaren_sk.py' 2019-12-04 12:45:20 ERROR During handling of the above exception, another exception occurred: 2019-12-04 12:45:20 ERROR Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sh_scrapy/log.py", line 88, in emit message = self.format(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 840, in format return fmt.format(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 585, in format record.exc_text = self.formatException(record.exc_info) File "/usr/local/lib/python3.6/logging/__init__.py", line 535, in formatException traceback.print_exception(ei[0], ei[1], tb, None, sio) File "/usr/local/lib/python3.6/traceback.py", line 104, in print_exception type(value), value, tb, limit=limit).format(chain=chain): File "/usr/local/lib/python3.6/traceback.py", line 522, in __init__ self._load_lines() File "/usr/local/lib/python3.6/traceback.py", line 534, in _load_lines self.__context__._load_lines() File "/usr/local/lib/python3.6/traceback.py", line 534, in _load_lines self.__context__._load_lines() File "/usr/local/lib/python3.6/traceback.py", line 534, in _load_lines self.__context__._load_lines() [Previous line repeated 1 more time] File "/usr/local/lib/python3.6/traceback.py", line 532, in _load_lines frame.line File "/usr/local/lib/python3.6/traceback.py", line 286, in line self._line = linecache.getline(self.filename, self.lineno).strip() File "/usr/local/lib/python3.6/linecache.py", line 16, in getline lines = getlines(filename, module_globals) File "/usr/local/lib/python3.6/linecache.py", line 47, in getlines return updatecache(filename, module_globals) File "/usr/local/lib/python3.6/linecache.py", line 103, in updatecache data = cache[filename][0]() 2019-12-04 12:45:20 ERROR UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 2578: invalid start byte 2019-12-04 12:45:20 ERROR Call stack: File "/usr/local/bin/start-crawl", line 10, in sys.exit(main()) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 221, in main _launch() File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 174, in _launch _run_usercode(job['spider'], args, _get_apisettings, loghdlr) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 148, in _run_usercode _run(args, settings) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 103, in _run _run_scrapy(args, settings) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 111, in _run_scrapy execute(settings=settings) File "/usr/local/lib/python3.6/site-packages/scrapy/cmdline.py", line 142, in execute _run_print_help(parser, _run_command, cmd, args, opts) File "/usr/local/lib/python3.6/site-packages/scrapy/cmdline.py", line 88, in _run_print_help func(*a, **kw) File "/usr/local/lib/python3.6/site-packages/scrapy/cmdline.py", line 149, in _run_command cmd.run(args, opts) File "/usr/local/lib/python3.6/site-packages/scrapy/commands/crawl.py", line 58, in run self.crawler_process.start() File "/usr/local/lib/python3.6/site-packages/scrapy/crawler.py", line 280, in start reactor.run(installSignalHandlers=False) # blocking call File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1199, in run self.mainLoop() File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1208, in mainLoop self.runUntilCurrent() File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 828, in runUntilCurrent call.func(*call.args, **call.kw) File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 671, in _tick taskObj._oneWorkUnit() File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 519, in _oneWorkUnit self._completeWith(TaskDone(), self._iterator) File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 487, in _completeWith d.callback(deferredResult) File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 457, in callback self._startRunCallbacks(result) File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 565, in _startRunCallbacks self._runCallbacks() File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 651, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 457, in callback self._startRunCallbacks(result) File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 565, in _startRunCallbacks self._runCallbacks() File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 878, in __del__ debugInfo=debugInfo) File "/usr/local/lib/python3.6/site-packages/twisted/logger/_logger.py", line 178, in failure self.emit(level, format, log_failure=failure, **kwargs) File "/usr/local/lib/python3.6/site-packages/twisted/logger/_logger.py", line 132, in emit self.observer(event) File "/usr/local/lib/python3.6/site-packages/twisted/logger/_observer.py", line 131, in __call__ observer(event) File "/usr/local/lib/python3.6/site-packages/twisted/logger/_legacy.py", line 93, in __call__ self.legacyObserver(event) File "/usr/local/lib/python3.6/site-packages/twisted/python/log.py", line 595, in emit _publishNew(self._newObserver, eventDict, textFromEventDict) File "/usr/local/lib/python3.6/site-packages/twisted/logger/_legacy.py", line 154, in publishToNewObserver observer(eventDict) File "/usr/local/lib/python3.6/site-packages/twisted/logger/_stdlib.py", line 116, in __call__ stdlibLevel, StringifiableFromEvent(event), exc_info=excInfo) File "/usr/local/lib/python3.6/logging/__init__.py", line 1374, in log self._log(level, msg, args, **kwargs) File "/usr/local/lib/python3.6/logging/__init__.py", line 1444, in _log self.handle(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 1454, in handle self.callHandlers(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 1516, in callHandlers hdlr.handle(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 865, in handle self.emit(record) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/log.py", line 94, in emit self.handleError(record) 2019-12-04 12:45:20 ERROR Message: 2019-12-04 12:45:20 ERROR Arguments: () 2019-12-04 12:45:33 INFO [scrapy.extensions.logstats] Crawled 577 pages (at 68 pages/min), scraped 546 items (at 68 items/min) 2019-12-04 12:46:33 INFO [scrapy.extensions.logstats] Crawled 641 pages (at 64 pages/min), scraped 603 items (at 57 items/min) 2019-12-04 12:47:33 INFO [scrapy.extensions.logstats] Crawled 708 pages (at 67 pages/min), scraped 669 items (at 66 items/min) 2019-12-04 12:48:33 INFO [scrapy.extensions.logstats] Crawled 779 pages (at 71 pages/min), scraped 734 items (at 65 items/min) 2019-12-04 12:49:33 INFO [scrapy.extensions.logstats] Crawled 859 pages (at 80 pages/min), scraped 813 items (at 79 items/min) 2019-12-04 12:50:33 INFO [scrapy.extensions.logstats] Crawled 934 pages (at 75 pages/min), scraped 882 items (at 69 items/min) 2019-12-04 12:51:33 INFO [scrapy.extensions.logstats] Crawled 1010 pages (at 76 pages/min), scraped 958 items (at 76 items/min) 2019-12-04 12:52:33 INFO [scrapy.extensions.logstats] Crawled 1077 pages (at 67 pages/min), scraped 1025 items (at 67 items/min) 2019-12-04 12:53:33 INFO [scrapy.extensions.logstats] Crawled 1141 pages (at 64 pages/min), scraped 1082 items (at 57 items/min) 2019-12-04 12:54:33 INFO [scrapy.extensions.logstats] Crawled 1201 pages (at 60 pages/min), scraped 1142 items (at 60 items/min) 2019-12-04 12:55:33 INFO [scrapy.extensions.logstats] Crawled 1272 pages (at 71 pages/min), scraped 1206 items (at 64 items/min) 2019-12-04 12:56:33 INFO [scrapy.extensions.logstats] Crawled 1351 pages (at 79 pages/min), scraped 1285 items (at 79 items/min) 2019-12-04 12:57:33 INFO [scrapy.extensions.logstats] Crawled 1438 pages (at 87 pages/min), scraped 1365 items (at 80 items/min) 2019-12-04 12:58:33 INFO [scrapy.extensions.logstats] Crawled 1509 pages (at 71 pages/min), scraped 1436 items (at 71 items/min) 2019-12-04 12:59:33 INFO [scrapy.extensions.logstats] Crawled 1569 pages (at 60 pages/min), scraped 1490 items (at 54 items/min) 2019-12-04 13:00:33 INFO [scrapy.extensions.logstats] Crawled 1630 pages (at 61 pages/min), scraped 1551 items (at 61 items/min) 2019-12-04 13:01:33 INFO [scrapy.extensions.logstats] Crawled 1701 pages (at 71 pages/min), scraped 1616 items (at 65 items/min) 2019-12-04 13:02:04 ERROR --- Logging error --- 2019-12-04 13:02:04 ERROR Unhandled error in Deferred: 2019-12-04 13:02:04 ERROR --- Logging error --- 2019-12-04 13:02:04 CRITICAL [twisted] Unhandled error in Deferred: 2019-12-04 13:02:04 ERROR Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 517, in _oneWorkUnit result = next(self._iterator) 2019-12-04 13:02:04 ERROR Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1199, in run self.mainLoop() File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1208, in mainLoop self.runUntilCurrent() File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 828, in runUntilCurrent call.func(*call.args, **call.kw) File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 671, in _tick taskObj._oneWorkUnit() --- --- File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 517, in _oneWorkUnit result = next(self._iterator) File "/usr/local/lib/python3.6/site-packages/scrapy/utils/defer.py", line 63, in work = (callable(elem, *args, **named) for elem in iterable) File "/usr/local/lib/python3.6/site-packages/scrapy/utils/defer.py", line 106, in iter_errback errback(failure.Failure(), *a, **kw) File "/usr/local/lib/python3.6/site-packages/scrapy/core/scraper.py", line 158, in handle_spider_error extra={'spider': spider} File "/usr/local/lib/python3.6/logging/__init__.py", line 1337, in error self._log(ERROR, msg, args, **kwargs) File "/usr/local/lib/python3.6/logging/__init__.py", line 1444, in _log self.handle(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 1454, in handle self.callHandlers(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 1516, in callHandlers hdlr.handle(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 865, in handle self.emit(record) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/log.py", line 94, in emit self.handleError(record) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/log.py", line 100, in handleError super(HubstorageLogHandler, self).handleError(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 918, in handleError traceback.print_exception(t, v, tb, None, sys.stderr) File "/usr/local/lib/python3.6/traceback.py", line 104, in print_exception type(value), value, tb, limit=limit).format(chain=chain): File "/usr/local/lib/python3.6/traceback.py", line 522, in __init__ self._load_lines() File "/usr/local/lib/python3.6/traceback.py", line 534, in _load_lines self.__context__._load_lines() File "/usr/local/lib/python3.6/traceback.py", line 534, in _load_lines self.__context__._load_lines() File "/usr/local/lib/python3.6/traceback.py", line 532, in _load_lines frame.line File "/usr/local/lib/python3.6/traceback.py", line 286, in line self._line = linecache.getline(self.filename, self.lineno).strip() File "/usr/local/lib/python3.6/linecache.py", line 16, in getline lines = getlines(filename, module_globals) File "/usr/local/lib/python3.6/linecache.py", line 47, in getlines return updatecache(filename, module_globals) File "/usr/local/lib/python3.6/linecache.py", line 103, in updatecache data = cache[filename][0]() builtins.UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 2578: invalid start byte 2019-12-04 13:02:04 ERROR StopIteration 2019-12-04 13:02:04 ERROR During handling of the above exception, another exception occurred: 2019-12-04 13:02:04 ERROR Traceback (most recent call last): File "/usr/local/lib/python3.6/linecache.py", line 95, in updatecache stat = os.stat(fullname) 2019-12-04 13:02:04 ERROR NotADirectoryError: [Errno 20] Not a directory: '/app/__main__.egg/mdx_crawl/spiders/mojalekaren_sk.py' 2019-12-04 13:02:04 ERROR During handling of the above exception, another exception occurred: 2019-12-04 13:02:04 ERROR Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sh_scrapy/log.py", line 88, in emit message = self.format(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 840, in format return fmt.format(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 585, in format record.exc_text = self.formatException(record.exc_info) File "/usr/local/lib/python3.6/logging/__init__.py", line 535, in formatException traceback.print_exception(ei[0], ei[1], tb, None, sio) File "/usr/local/lib/python3.6/traceback.py", line 104, in print_exception type(value), value, tb, limit=limit).format(chain=chain): File "/usr/local/lib/python3.6/traceback.py", line 522, in __init__ self._load_lines() File "/usr/local/lib/python3.6/traceback.py", line 534, in _load_lines self.__context__._load_lines() File "/usr/local/lib/python3.6/traceback.py", line 534, in _load_lines self.__context__._load_lines() File "/usr/local/lib/python3.6/traceback.py", line 534, in _load_lines self.__context__._load_lines() [Previous line repeated 1 more time] File "/usr/local/lib/python3.6/traceback.py", line 532, in _load_lines frame.line File "/usr/local/lib/python3.6/traceback.py", line 286, in line self._line = linecache.getline(self.filename, self.lineno).strip() File "/usr/local/lib/python3.6/linecache.py", line 16, in getline lines = getlines(filename, module_globals) File "/usr/local/lib/python3.6/linecache.py", line 47, in getlines return updatecache(filename, module_globals) File "/usr/local/lib/python3.6/linecache.py", line 103, in updatecache data = cache[filename][0]() 2019-12-04 13:02:05 ERROR UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 2578: invalid start byte 2019-12-04 13:02:05 ERROR Call stack: File "/usr/local/bin/start-crawl", line 10, in sys.exit(main()) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 221, in main _launch() File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 174, in _launch _run_usercode(job['spider'], args, _get_apisettings, loghdlr) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 148, in _run_usercode _run(args, settings) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 103, in _run _run_scrapy(args, settings) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 111, in _run_scrapy execute(settings=settings) File "/usr/local/lib/python3.6/site-packages/scrapy/cmdline.py", line 142, in execute _run_print_help(parser, _run_command, cmd, args, opts) File "/usr/local/lib/python3.6/site-packages/scrapy/cmdline.py", line 88, in _run_print_help func(*a, **kw) File "/usr/local/lib/python3.6/site-packages/scrapy/cmdline.py", line 149, in _run_command cmd.run(args, opts) File "/usr/local/lib/python3.6/site-packages/scrapy/commands/crawl.py", line 58, in run self.crawler_process.start() File "/usr/local/lib/python3.6/site-packages/scrapy/crawler.py", line 280, in start reactor.run(installSignalHandlers=False) # blocking call File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1199, in run self.mainLoop() File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 1208, in mainLoop self.runUntilCurrent() File "/usr/local/lib/python3.6/site-packages/twisted/internet/base.py", line 828, in runUntilCurrent call.func(*call.args, **call.kw) File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 671, in _tick taskObj._oneWorkUnit() File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 519, in _oneWorkUnit self._completeWith(TaskDone(), self._iterator) File "/usr/local/lib/python3.6/site-packages/twisted/internet/task.py", line 487, in _completeWith d.callback(deferredResult) File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 457, in callback self._startRunCallbacks(result) File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 565, in _startRunCallbacks self._runCallbacks() File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 651, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 457, in callback self._startRunCallbacks(result) File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 565, in _startRunCallbacks self._runCallbacks() File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 878, in __del__ debugInfo=debugInfo) File "/usr/local/lib/python3.6/site-packages/twisted/logger/_logger.py", line 178, in failure self.emit(level, format, log_failure=failure, **kwargs) File "/usr/local/lib/python3.6/site-packages/twisted/logger/_logger.py", line 132, in emit self.observer(event) File "/usr/local/lib/python3.6/site-packages/twisted/logger/_observer.py", line 131, in __call__ observer(event) File "/usr/local/lib/python3.6/site-packages/twisted/logger/_legacy.py", line 93, in __call__ self.legacyObserver(event) File "/usr/local/lib/python3.6/site-packages/twisted/python/log.py", line 595, in emit _publishNew(self._newObserver, eventDict, textFromEventDict) File "/usr/local/lib/python3.6/site-packages/twisted/logger/_legacy.py", line 154, in publishToNewObserver observer(eventDict) File "/usr/local/lib/python3.6/site-packages/twisted/logger/_stdlib.py", line 116, in __call__ stdlibLevel, StringifiableFromEvent(event), exc_info=excInfo) File "/usr/local/lib/python3.6/logging/__init__.py", line 1374, in log self._log(level, msg, args, **kwargs) File "/usr/local/lib/python3.6/logging/__init__.py", line 1444, in _log self.handle(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 1454, in handle self.callHandlers(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 1516, in callHandlers hdlr.handle(record) File "/usr/local/lib/python3.6/logging/__init__.py", line 865, in handle self.emit(record) File "/usr/local/lib/python3.6/site-packages/sh_scrapy/log.py", line 94, in emit self.handleError(record) 2019-12-04 13:02:05 ERROR Message: 2019-12-04 13:02:05 ERROR Arguments: () 2019-12-04 13:02:33 INFO [scrapy.extensions.logstats] Crawled 1771 pages (at 70 pages/min), scraped 1685 items (at 69 items/min) 2019-12-04 13:03:33 INFO [scrapy.extensions.logstats] Crawled 1844 pages (at 73 pages/min), scraped 1752 items (at 67 items/min) 2019-12-04 13:04:33 INFO [scrapy.extensions.logstats] Crawled 1921 pages (at 77 pages/min), scraped 1829 items (at 77 items/min) 2019-12-04 13:05:30 INFO [scrapy.crawler] Received SIGTERM, shutting down gracefully. Send again to force 2019-12-04 13:05:30 INFO [scrapy.core.engine] Closing spider (shutdown) 2019-12-04 13:05:33 INFO [scrapy.extensions.logstats] Crawled 1998 pages (at 77 pages/min), scraped 1900 items (at 71 items/min) 2019-12-04 13:05:44 INFO [scrapy.statscollectors] Dumping Scrapy stats: {'downloader/request_bytes': 919540, 'downloader/request_count': 2013, 'downloader/request_method_count/GET': 2013, 'downloader/response_bytes': 65144359, 'downloader/response_count': 2013, 'downloader/response_status_count/200': 2012, 'downloader/response_status_count/301': 1, 'dupefilter/filtered': 256, 'finish_reason': 'shutdown', 'finish_time': datetime.datetime(2019, 12, 4, 13, 5, 44, 870363), 'item_scraped_count': 1914, 'log_count/CRITICAL': 4, 'log_count/INFO': 36, 'memusage/max': 113332224, 'memusage/startup': 53125120, 'request_depth_max': 15, 'response_received_count': 2012, 'scheduler/dequeued': 2013, 'scheduler/dequeued/disk': 96, 'scheduler/dequeued/memory': 1917, 'scheduler/enqueued': 2103, 'scheduler/enqueued/disk': 102, 'scheduler/enqueued/memory': 2001, 'scheduler/unserializable': 2001, 'start_time': datetime.datetime(2019, 12, 4, 12, 37, 33, 273028)} 2019-12-04 13:05:44 INFO [scrapy.core.engine] Spider closed (shutdown) 2019-12-04 13:05:44 INFO (TCP Port 6023 Closed) 2019-12-04 13:05:44 INFO Main loop terminated.