I have a program that needs to scrape several different urls using
scrapy and I need it to use the same user agent and IP address for each
url every time it scrapes that url. So if I am scraping like 50 urls I need each url to have one
unique user agent and ip address that are only used when scraping that
url. But the IP address and user agent get changed when the program
scrapes the next url.
I have already got it to rotate user agents randomly but now I need to change it so it does not rotate randomly. I just
need to pair user agents with different urls and use those same user
agents with same urls each time. As for the IP addresses I cannot even
get it to rotate them randomly let alone pair them with one unique url. I have attached my settings.py and SplashSpider.py files. Any help would be greatly appreciated!
tnorth2620
I have a program that needs to scrape several different urls using scrapy and I need it to use the same user agent and IP address for each url every time it scrapes that url. So if I am scraping like 50 urls I need each url to have one unique user agent and ip address that are only used when scraping that url. But the IP address and user agent get changed when the program scrapes the next url.
I have already got it to rotate user agents randomly but now I need to change it so it does not rotate randomly. I just need to pair user agents with different urls and use those same user agents with same urls each time. As for the IP addresses I cannot even get it to rotate them randomly let alone pair them with one unique url. I have attached my settings.py and SplashSpider.py files. Any help would be greatly appreciated!