Hi, I'm having trouble with my DOWNLOADER_MIDDLEWARES on Scrapy Cloud. I'm using a user-agents library that assigns user agents to various requests, and it works fine on my local computer. However, on Scrapy Cloud, the user-agents are not assigned to each request.
This is what I have in my setting.py file:
My local computer assigns the variables:
Scrapy Cloud doesn't assign user-agents to requests:
Also, DOWNLOADER_MIDDLEWARES is not a listed field in Spider Setttings.
I did some more research ~ the # assigned to scrapy_user_agents.middlewares.RandomUserAgentMiddleware should be 500. 400 was for older versions of Scrapy, but Scrapy 2.3 has added some more keys to the dict DOWNLOADER_MIDDLEWARES. So for anyone else who runs into this problem, use 500 instead to replace the default scrapy UserAgentMiddleware.
I can see in the log that scrapy-user-agents is recognized as the main UserAgentMiddleWare:
However, Scrapy Cloud doesn't recognize the scrapy-user-agents. On my local computer I get [scrapy.downloader.middlewares.redirect] and [scrapy_user_agents.middlewares] that appear in front of DEBUG: statements. These do not appear in the log for Scrapy Cloud.