I'm gathering data using a SitemapSpider, but there is additional data I need to fetch from a JSON query from an URL that I get from each parsed page, a JSON object that has vital data for the page I am currently parsing.
Right now I am using the "requests" package to easily grab the JSON page, and this works just fine. My question, however, is if there is a cleaner/more acceptable way to do this within Scrapy itself without using outside libraries?
I'm gathering data using a SitemapSpider, but there is additional data I need to fetch from a JSON query from an URL that I get from each parsed page, a JSON object that has vital data for the page I am currently parsing.
Right now I am using the "requests" package to easily grab the JSON page, and this works just fine. My question, however, is if there is a cleaner/more acceptable way to do this within Scrapy itself without using outside libraries?
0 Votes
0 Comments
Login to post a comment