Start a new topic

DOWNLOADER_MIDDLEWARES Issue

Hi, I'm having trouble with my DOWNLOADER_MIDDLEWARES on Scrapy Cloud. I'm using a user-agents library that assigns user agents to various requests, and it works fine on my local computer. However, on Scrapy Cloud, the user-agents are not assigned to each request. 

This is what I have in my setting.py file:

image


My local computer assigns the variables:

image


Scrapy Cloud doesn't assign user-agents to requests:

image



Also, DOWNLOADER_MIDDLEWARES is not a listed field in Spider Setttings.

I did some more research ~ the # assigned to scrapy_user_agents.middlewares.RandomUserAgentMiddleware should be 500. 400 was for older versions of Scrapy, but Scrapy 2.3 has added some more keys to the dict DOWNLOADER_MIDDLEWARES. So for anyone else who runs into this problem, use 500 instead to replace the default scrapy UserAgentMiddleware. 

I can see in the log that scrapy-user-agents is recognized as the main UserAgentMiddleWare:

image


However, Scrapy Cloud doesn't recognize the scrapy-user-agents. On my local computer I get [scrapy.downloader.middlewares.redirect] and [scrapy_user_agents.middlewares] that appear in front of DEBUG: statements. These do not appear in the log for Scrapy Cloud.

Login to post a comment