How to set RandomUserAgentMiddleware ?

Posted about 6 years ago by remitoudic

Post a topic

Answered

remitoudic

Hey ,

I am testing Scrapinghub and try to get by . I find all the option really good , I am missing something very important. How can I rotate the user agent ?

Is there a way to set up the DOWNLOADER_MIDDLEWARES in order to rotate the user agent ? If not , is there any alternative ?

Thank you in advance

Rémi

0 Votes

nestor posted about 6 years ago Admin Best Answer

DOWNLOADER_MIDDLEWARES are set in your settings.py

You'll also need to add the library via requirements.txt https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud

0 Votes

6 Comments

vinitha R posted about 3 years ago

I'm getting this middleware error. I don't know how to handle this error. please anyone could me on this.

0 Votes

remitoudic posted about 6 years ago

Hey ,

Today I have again problem with the 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware'.

This time it is an error by running a job on scrapinghub. :(

raise FakeUserAgentError('Maximum amount of retries reached')
	fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached

I don't really understand since I didn't deploy something new and since the same code is running smoothly on my local machine .

Has someone an idea ?

Thank you in advance

best

0 Votes

nestor posted about 6 years ago Admin

It is enabled, check log line 6 of your latest jobs and you'll see: 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware'. I don't think it logs anything else.

1 Votes

remitoudic posted about 6 years ago

Hey , Update on my issue.

I change the stack in scrapinghub,yml to run same the python version that I have locally. The fake-useragent library just did not start and did not log anything . That is why I though it came form scrapinghub.

0 Votes

remitoudic posted about 6 years ago

Thank you for answer nestor . About the dependencies all fine (if the library would not have not been in the requirement file I would have got an error on it .... ) . And about the setting.py , that is also good since I followed the instruction on Pypi (https://pypi.org/project/scrapy-fake-useragent-fix/)

I guess something is messing ......

When I got it I post it here :)

0 Votes

nestor posted about 6 years ago Admin Answer

DOWNLOADER_MIDDLEWARES are set in your settings.py

You'll also need to add the library via requirements.txt https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud

0 Votes