Crawl "www.reuters.com" working locally, but didn't work when deploying on the cloud

Start a new topic

started a topic about 5 years ago

Below is the log, no any error but seeing scrapy didn't get any response, could anyone make a help on that?

I am trying this cloud and planning to subscribe one, if this situation happens, I can't make my future spiders will be working fine on cloud.

Time (UTC)	Level	Message
0:	2019-02-23 04:15:12	INFO	Log opened.
1:	2019-02-23 04:15:12	INFO	[scrapy.log] Scrapy 1.4.0 started
2:	2019-02-23 04:15:12	INFO	[scrapy.utils.log] Scrapy 1.4.0 started (bot: newscrawl)
3:	2019-02-23 04:15:12	INFO	[scrapy.utils.log] Overridden settings: {'NEWSPIDER_MODULE': 'newscrawl.spiders', 'STATS_CLASS': 'sh_scrapy.stats.HubStorageStatsCollector', 'LOG_LEVEL': 'INFO', 'SPIDER_MODULES': ['newscrawl.spiders'], 'AUTOTHROTTLE_ENABLED': True, 'LOG_ENABLED': False, 'MEMUSAGE_LIMIT_MB': 950, 'BOT_NAME': 'newscrawl', 'TELNETCONSOLE_HOST': '0.0.0.0'}
4:	2019-02-23 04:15:13	INFO	[scrapy.middleware] Enabled extensions: More
5:	2019-02-23 04:15:14	INFO	[scrapy.middleware] Enabled downloader middlewares: More
6:	2019-02-23 04:15:14	INFO	[scrapy.middleware] Enabled spider middlewares: More
7:	2019-02-23 04:15:14	INFO	[scrapy.middleware] Enabled item pipelines: More
8:	2019-02-23 04:15:14	INFO	[scrapy.core.engine] Spider opened
9:	2019-02-23 04:15:14	INFO	[scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
10:	2019-02-23 04:15:14	INFO	[news_reuters] Spider opened: news_reuters
11:	2019-02-23 04:15:14	INFO	TelnetConsole starting on 6023
12:	2019-02-23 04:15:27	INFO	[scrapy.core.engine] Closing spider (finished)
13:	2019-02-23 04:15:27	INFO	[scrapy.statscollectors] Dumping Scrapy stats: More
14:	2019-02-23 04:15:27	INFO	[scrapy.core.engine] Spider closed (finished)
15:	2019-02-23 04:15:27	INFO	(TCP Port 6023 Closed)
16:	2019-02-23 04:15:27	INFO	Main loop terminated.

1 Comment

said about 5 years ago

Hi,

Could you please try disabling the auto-throttle? Please set the value to False AUTOTHROTTLE_ENABLED': False

Zyte Support Center