Start a new topic

Some fields are populating null values although they are present on website

I am trying to scrape items from this link https://www.amazon.com/s?k=IGBT+Transistors&i=industrial&rh=n%3A306916011&s=price-asc-rank

 

,I have tried using these set of headers.

 

 headers={

        'Accept''text/html, */*; q=0.01',

'Accept-Encoding''gzip, deflate, br',

'Accept-Language''en-US,en;q=0.9',

'Cache-Control''no-cache',

'Connection''keep-alive',

'device-memory''8',

'downlink':'6.35',

'dpr':'2',

'ect':'4g',

'Host''www.amazon.com',

'Pragma''no-cache',

'Referer''https://www.amazon.com/IGBT-Transistors-UltraFast-8-25kHz-piece/dp/B00DJY0W2U/ref=sr_1_1?keywords=igbt+transistor&qid=1654595374&refinements=p_n_availability%3A2661601011&s=industrial&sr=1-1',

'rtt':'50',

'sec-ch-device-memory''8',

'sec-ch-dpr':'2',

'sec-ch-ua''" Not A;Brand";v="99", "Chromium";v="101", "Google Chrome";v="101"',

'sec-ch-ua-mobile''?1',

'sec-ch-ua-platform':'"Android"',

'sec-ch-viewport-width':'918',

'Sec-Fetch-Dest''empty',

'Sec-Fetch-Mode''cors',

'Sec-Fetch-Site''same-origin',

'User-Agent''Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Mobile Safari/537.36',

'viewport-width''918',

'X-Requested-With':'XMLHttpRequest'

    }

and 


     headers = {

  'Referer': 'https://www.amazon.com',

  'Host': 'www.amazon.com',

  'X-Crawlera-Profile': 'desktop',

  'X-Crawlera-Cookies': 'enable',

  'Accept-Language': 'en-US,en;q=0.9',

  'Cache-Control': 'no-cache',

  'Connection': 'keep-alive',

  'Pragma': 'no-cache',

  'Sec-Fetch-Dest': 'document',

  'Sec-Fetch-Mode': 'navigate',

  'Sec-Fetch-Site': 'same-origin',

  'Upgrade-Insecure-Requests': '1',

  'User-Agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Mobile Safari/537.36',

  'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="100", "Google Chrome";v="100"',

  'sec-ch-ua-mobile': '?1'

}


On one run these are working fine but on the another run some fields are populating null values although they are present on the website.Also the no.of items scraped keeps varying with each run(very less no.of items are scraped in the latest runs as compared to the average number).Can you suggest me the correct headers for this or any other solution for this.



Project id : 585304

Login to post a comment