Integrate Scrapy + Selenium scraper to smart proxy manager and schedule daily tasks on prefect

Posted over 3 years ago by Fyonn Oh

Post a topic
Un Answered
F
Fyonn Oh

Hi everyone, I am having trouble trying to figure out how to integrate smart proxy manager with my scraper which uses scrapy and selenium (I have provided a screenshot of the code for my scraper). What I aim to do is to schedule a daily task on prefect which runs my scraper to scrape data with IP rotation using zyte. Please help me understand how I should structure and integrate the three components: scraper and prefect and zyte. Any help is truly appreciated!! :)


0 Votes


0 Comments

Login to post a comment