videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic

Scrapy redirect not working for some scraper

I had a issue with a scraper that was working fine on my local machine but wouldn't run on zyte. I changed the user agent inside the scraper to a Linux user agent and it now works. However on my local machine it is parsing and scraping all items that I am trying. Some links are getting redirected to new links when I run it on my local machine. The same is not happening when I am running it on zyte with the new user agent and the links are throwing errors. I am a beginner and would be grateful if can get help in resolving this issue.

Login to post a comment