Start a new topic

Best way to do a json request inside a spider?

I'm gathering data using a SitemapSpider, but there is additional data I need to fetch from a JSON query from an URL that I get from each parsed page, a JSON object that has vital data for the page I am currently parsing.

Right now I am using the "requests" package to easily grab the JSON page, and this works just fine. My question, however, is if there is a cleaner/more acceptable way to do this within Scrapy itself without using outside libraries?

Login to post a comment