I have a scrapy
spider that works fine locally and I am trying to scrape an external
url using crawlera. However, the spider is not going thru crawlera
(middleware not enabled).
I followed this
article but I can’t get it to work. Anything I’m missing?
rmunoz
Hi,
I have a scrapy spider that works fine locally and I am trying to scrape an external url using crawlera. However, the spider is not going thru crawlera (middleware not enabled).
I followed this article but I can’t get it to work. Anything I’m missing?
https://support.scrapinghub.com/support/solutions/articles/22000188399-using-crawlera-with-scrapy
I added scrapy-crawlera 1.6.0
I added the following to the end of settings.py
Here is my partial log
2019-07-30 10:48:36 [scrapy.extensions.telnet] INFO: Telnet Password: 7552880fdb4dd4be
2019-07-30 10:48:36 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats']
2019-07-30 10:48:36 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-07-30 10:48:36 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-07-30 10:48:36 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-07-30 10:48:36 [scrapy.core.engine] INFO: Spider opened
2019-07-30 10:48:36 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
Hello,
As you are a paying customer I am converting this topic to a Support ticket . One of our Support Engineers would get in touch with you soon.
thriveni
Hello,
As you are a paying customer I am converting this topic to a Support ticket . One of our Support Engineers would get in touch with you soon.
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 365 topics