videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic

Best way to do a json request inside a spider?

I'm gathering data using a SitemapSpider, but there is additional data I need to fetch from a JSON query from an URL that I get from each parsed page, a JSON object that has vital data for the page I am currently parsing.


Right now I am using the "requests" package to easily grab the JSON page, and this works just fine. My question, however, is if there is a cleaner/more acceptable way to do this within Scrapy itself without using outside libraries?

Login to post a comment