Some fields are populating null values although they are present on website

Posted over 2 years ago by Simran Sharma

Post a topic

Un Answered

Simran Sharma

I am trying to scrape items from this link https://www.amazon.com/s?k=IGBT+Transistors&i=industrial&rh=n%3A306916011&s=price-asc-rank

,I have tried using these set of headers.

headers={

'Accept': 'text/html, */*; q=0.01',

'Accept-Encoding': 'gzip, deflate, br',

'Accept-Language': 'en-US,en;q=0.9',

'Cache-Control': 'no-cache',

'Connection': 'keep-alive',

'device-memory': '8',

'downlink':'6.35',

'dpr':'2',

'ect':'4g',

'Host': 'www.amazon.com',

'Pragma': 'no-cache',

'Referer': 'https://www.amazon.com/IGBT-Transistors-UltraFast-8-25kHz-piece/dp/B00DJY0W2U/ref=sr_1_1?keywords=igbt+transistor&qid=1654595374&refinements=p_n_availability%3A2661601011&s=industrial&sr=1-1',

'rtt':'50',

'sec-ch-device-memory': '8',

'sec-ch-dpr':'2',

'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="101", "Google Chrome";v="101"',

'sec-ch-ua-mobile': '?1',

'sec-ch-ua-platform':'"Android"',

'sec-ch-viewport-width':'918',

'Sec-Fetch-Dest': 'empty',

'Sec-Fetch-Mode': 'cors',

'Sec-Fetch-Site': 'same-origin',

'User-Agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Mobile Safari/537.36',

'viewport-width': '918',

'X-Requested-With':'XMLHttpRequest'

}

and

headers = {

'Referer': 'https://www.amazon.com',

'Host': 'www.amazon.com',

'X-Crawlera-Profile': 'desktop',

'X-Crawlera-Cookies': 'enable',

'Accept-Language': 'en-US,en;q=0.9',

'Cache-Control': 'no-cache',

'Connection': 'keep-alive',

'Pragma': 'no-cache',

'Sec-Fetch-Dest': 'document',

'Sec-Fetch-Mode': 'navigate',

'Sec-Fetch-Site': 'same-origin',

'Upgrade-Insecure-Requests': '1',

'User-Agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Mobile Safari/537.36',

'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="100", "Google Chrome";v="100"',

'sec-ch-ua-mobile': '?1'

}

On one run these are working fine but on the another run some fields are populating null values although they are present on the website.Also the no.of items scraped keeps varying with each run(very less no.of items are scraped in the latest runs as compared to the average number).Can you suggest me the correct headers for this or any other solution for this.

Project id : 585304

0 Votes

0 Comments