Start a new topic

How to fake user agents and ip addresses

 I have a program that needs to scrape several different urls using scrapy and I need it to use the same user agent and IP address for each url every time it scrapes that url. So if I am scraping like 50 urls I need each url to have one unique user agent and ip address that are only used when scraping that url. But the IP address and user agent get changed when the program scrapes the next url.

I have already got it to rotate user agents randomly but now I need to change it so it does not rotate randomly.  I just need to pair user agents with different urls and use those same user agents with same urls each time. As for the IP addresses I cannot even get it to rotate them randomly let alone pair them with one unique url.  I have attached my and files.  Any help would be greatly appreciated!

(1018 Bytes)
(7.86 KB)
Login to post a comment