Start a new topic
Answered

How to set RandomUserAgentMiddleware ?

Hey ,

I am testing Scrapinghub and try to get by . I find all the option really good , I am missing something very important.  How can I  rotate the user agent ?

Is there a way to set up the  DOWNLOADER_MIDDLEWARES  in order to rotate  the user agent ?   If not , is there any  alternative ?

Thank you in advance

Rémi


Best Answer

DOWNLOADER_MIDDLEWARES are set in your settings.py

You'll also need to add the library via requirements.txt https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud


Answer

DOWNLOADER_MIDDLEWARES are set in your settings.py

You'll also need to add the library via requirements.txt https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud

Thank you for answer nestor .  About the dependencies all fine (if the library would not have not been in the requirement file I would have got an error on it .... ) .  And about the setting.py , that is also good  since I followed the instruction on Pypi  (https://pypi.org/project/scrapy-fake-useragent-fix/)

I guess  something is messing ......

When I got it I post it here :)

Hey ,  Update on my issue.

I change the stack in scrapinghub,yml to run  same the python  version  that I have locally.  The  fake-useragent  library just did not start and did not log anything . That  is why I though  it came form  scrapinghub. 

It is enabled, check log line 6 of your latest jobs and you'll see: 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware'. I don't think it logs anything else.


1 person likes this

Hey , 

Today I have again problem with the 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware'.

This time  it is  an error  by running  a job  on scrapinghub.    :(


raise FakeUserAgentError('Maximum amount of retries reached')
	fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached


I don't really understand since I didn't deploy  something new and since the same code is running smoothly on my local machine .

Has someone an idea  ?

Thank you in advance

best

image



I'm getting this middleware error. I don't know how to handle this error. please anyone could me on this.

Login to post a comment