Integrate Scrapy + Selenium scraper to smart proxy manager and schedule daily tasks on prefect
F
Fyonn Oh
started a topic
almost 3 years ago
Hi everyone, I am having trouble trying to figure out how to integratesmart proxy manager with my scraper which uses scrapy and selenium (I have provided a screenshot of the code for my scraper). What I aim to do is to schedule a daily task on prefect which runs my scraper to scrape data with IP rotation using zyte. Please help me understand how I should structure and integrate the three components: scraper and prefect and zyte. Any help is truly appreciated!! :)
Fyonn Oh
Hi everyone, I am having trouble trying to figure out how to integrate smart proxy manager with my scraper which uses scrapy and selenium (I have provided a screenshot of the code for my scraper). What I aim to do is to schedule a daily task on prefect which runs my scraper to scrape data with IP rotation using zyte. Please help me understand how I should structure and integrate the three components: scraper and prefect and zyte. Any help is truly appreciated!! :)