videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic

Scrapy items from multiple URLs to a dataframe

Hi, I have a spider which access a website and follow the link of each products to get the product details. Is there an option to get the out put of all the products details from many URLs to a single dataframe? Any help will be appreciated. Spider flow start_url >> will go the category and will fetch all the products URLs parse >>> Follow each products and finally will give a dataframe for all the products with product details for that particular category
Login to post a comment