Start a new topic

Crawl "" working locally, but didn't work when deploying on the cloud

Below is the log, no any error but seeing scrapy didn't get any response, could anyone make a help on that?

I am trying this cloud and planning to subscribe one, if this situation happens, I can't make my future spiders will be working fine on cloud.

Time (UTC)
0:2019-02-23 04:15:12INFO

Log opened.

1:2019-02-23 04:15:12INFO

[scrapy.log] Scrapy 1.4.0 started

2:2019-02-23 04:15:12INFO

[scrapy.utils.log] Scrapy 1.4.0 started (bot: newscrawl)

3:2019-02-23 04:15:12INFO

[scrapy.utils.log] Overridden settings: {'NEWSPIDER_MODULE': 'newscrawl.spiders', 'STATS_CLASS': 'sh_scrapy.stats.HubStorageStatsCollector', 'LOG_LEVEL': 'INFO', 'SPIDER_MODULES': ['newscrawl.spiders'], 'AUTOTHROTTLE_ENABLED': True, 'LOG_ENABLED': False, 'MEMUSAGE_LIMIT_MB': 950, 'BOT_NAME': 'newscrawl', 'TELNETCONSOLE_HOST': ''}

4:2019-02-23 04:15:13INFO

[scrapy.middleware] Enabled extensions:

5:2019-02-23 04:15:14INFO

[scrapy.middleware] Enabled downloader middlewares:

6:2019-02-23 04:15:14INFO

[scrapy.middleware] Enabled spider middlewares:

7:2019-02-23 04:15:14INFO

[scrapy.middleware] Enabled item pipelines:

8:2019-02-23 04:15:14INFO

[scrapy.core.engine] Spider opened

9:2019-02-23 04:15:14INFO

[scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

10:2019-02-23 04:15:14INFO

[news_reuters] Spider opened: news_reuters

11:2019-02-23 04:15:14INFO

TelnetConsole starting on 6023

12:2019-02-23 04:15:27INFO

[scrapy.core.engine] Closing spider (finished)

13:2019-02-23 04:15:27INFO

[scrapy.statscollectors] Dumping Scrapy stats:

14:2019-02-23 04:15:27INFO

[scrapy.core.engine] Spider closed (finished)

15:2019-02-23 04:15:27INFO

(TCP Port 6023 Closed)

16:2019-02-23 04:15:27INFO

Main loop terminated.

1 Comment


Could you please try disabling the auto-throttle? Please set the value to False  AUTOTHROTTLE_ENABLED': False  

Login to post a comment