Crawl "www.reuters.com" working locally, but didn't work when deploying on the cloud

Posted almost 6 years ago by Lilu Cao

Post a topic
Un Answered
L
Lilu Cao

Below is the log, no any error but seeing scrapy didn't get any response, could anyone make a help on that?

I am trying this cloud and planning to subscribe one, if this situation happens, I can't make my future spiders will be working fine on cloud.





Time (UTC)
LevelMessage
0:2019-02-23 04:15:12INFO

Log opened.

1:2019-02-23 04:15:12INFO

[scrapy.log] Scrapy 1.4.0 started

2:2019-02-23 04:15:12INFO

[scrapy.utils.log] Scrapy 1.4.0 started (bot: newscrawl)

3:2019-02-23 04:15:12INFO

[scrapy.utils.log] Overridden settings: {'NEWSPIDER_MODULE': 'newscrawl.spiders', 'STATS_CLASS': 'sh_scrapy.stats.HubStorageStatsCollector', 'LOG_LEVEL': 'INFO', 'SPIDER_MODULES': ['newscrawl.spiders'], 'AUTOTHROTTLE_ENABLED': True, 'LOG_ENABLED': False, 'MEMUSAGE_LIMIT_MB': 950, 'BOT_NAME': 'newscrawl', 'TELNETCONSOLE_HOST': '0.0.0.0'}

4:2019-02-23 04:15:13INFO

[scrapy.middleware] Enabled extensions:

 More
5:2019-02-23 04:15:14INFO

[scrapy.middleware] Enabled downloader middlewares:

 More
6:2019-02-23 04:15:14INFO

[scrapy.middleware] Enabled spider middlewares:

 More
7:2019-02-23 04:15:14INFO

[scrapy.middleware] Enabled item pipelines:

 More
8:2019-02-23 04:15:14INFO

[scrapy.core.engine] Spider opened

9:2019-02-23 04:15:14INFO

[scrapy.extensions.logstats] Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

10:2019-02-23 04:15:14INFO

[news_reuters] Spider opened: news_reuters

11:2019-02-23 04:15:14INFO

TelnetConsole starting on 6023

12:2019-02-23 04:15:27INFO

[scrapy.core.engine] Closing spider (finished)

13:2019-02-23 04:15:27INFO

[scrapy.statscollectors] Dumping Scrapy stats:

 More
14:2019-02-23 04:15:27INFO

[scrapy.core.engine] Spider closed (finished)

15:2019-02-23 04:15:27INFO

(TCP Port 6023 Closed)

16:2019-02-23 04:15:27INFO

Main loop terminated.

0 Votes


1 Comments

tejashri

tejashri posted almost 6 years ago Admin

Hi,


Could you please try disabling the auto-throttle? Please set the value to False  AUTOTHROTTLE_ENABLED': False  

0 Votes

Login to post a comment