I am testing Scrapinghub and try to get by . I find all the option really good , I am missing something very important. How can I rotate the user agent ?
Is there a way to set up the DOWNLOADER_MIDDLEWARES in order to rotate the user agent ? If not , is there any alternative ?
Thank you in advance
Rémi
0 Votes
nestor posted
about 6 years ago
AdminBest Answer
DOWNLOADER_MIDDLEWARES are set in your settings.py
I'm getting this middleware error. I don't know how to handle this error. please anyone could me on this.
0 Votes
r
remitoudicposted
about 6 years ago
Hey ,
Today I have again problem with the 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware'.
This time it is an error by running a job on scrapinghub. :(
raise FakeUserAgentError('Maximum amount of retries reached')
fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached
I don't really understand since I didn't deploy something new and since the same code is running smoothly on my local machine .
Has someone an idea ?
Thank you in advance
best
0 Votes
nestorposted
about 6 years ago
Admin
It is enabled, check log line 6 of your latest jobs and you'll see: 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware'. I don't think it logs anything else.
1 Votes
r
remitoudicposted
about 6 years ago
Hey , Update on my issue.
I change the stack in scrapinghub,yml to run same the python version that I have locally. The fake-useragent library just did not start and did not log anything . That is why I though it came form scrapinghub.
0 Votes
r
remitoudicposted
about 6 years ago
Thank you for answer nestor . About the dependencies all fine (if the library would not have not been in the requirement file I would have got an error on it .... ) . And about the setting.py , that is also good since I followed the instruction on Pypi (https://pypi.org/project/scrapy-fake-useragent-fix/)
I guess something is messing ......
When I got it I post it here :)
0 Votes
nestorposted
about 6 years ago
AdminAnswer
DOWNLOADER_MIDDLEWARES are set in your settings.py
Hey ,
I am testing Scrapinghub and try to get by . I find all the option really good , I am missing something very important. How can I rotate the user agent ?
Is there a way to set up the DOWNLOADER_MIDDLEWARES in order to rotate the user agent ? If not , is there any alternative ?
Thank you in advance
Rémi
0 Votes
nestor posted about 6 years ago Admin Best Answer
DOWNLOADER_MIDDLEWARES are set in your settings.py
You'll also need to add the library via requirements.txt https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud
0 Votes
6 Comments
vinitha R posted about 3 years ago
I'm getting this middleware error. I don't know how to handle this error. please anyone could me on this.
0 Votes
remitoudic posted about 6 years ago
Hey ,
Today I have again problem with the 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware'.
This time it is an error by running a job on scrapinghub. :(
I don't really understand since I didn't deploy something new and since the same code is running smoothly on my local machine .
Has someone an idea ?
Thank you in advance
best
0 Votes
nestor posted about 6 years ago Admin
It is enabled, check log line 6 of your latest jobs and you'll see: 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware'. I don't think it logs anything else.
1 Votes
remitoudic posted about 6 years ago
Hey , Update on my issue.
I change the stack in scrapinghub,yml to run same the python version that I have locally. The fake-useragent library just did not start and did not log anything . That is why I though it came form scrapinghub.
0 Votes
remitoudic posted about 6 years ago
Thank you for answer nestor . About the dependencies all fine (if the library would not have not been in the requirement file I would have got an error on it .... ) . And about the setting.py , that is also good since I followed the instruction on Pypi (https://pypi.org/project/scrapy-fake-useragent-fix/)
I guess something is messing ......
When I got it I post it here :)
0 Votes
nestor posted about 6 years ago Admin Answer
DOWNLOADER_MIDDLEWARES are set in your settings.py
You'll also need to add the library via requirements.txt https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud
0 Votes
Login to post a comment