videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic
Answered

Bing

Hi,


I am trying to use Crawlera to get some information about 1000 or so businesses in my local area from Bing.


Originally, I was planning on using the Bing search API. However the API only returns the basic search results. What I am more interested in is the information on the side bar showing links to the TripAdvisor page etc...


If I do a basic search building a query string and using the Crawlera proxy, about 1 in 10 times I get the full page as I expect. The rest of the time, I get half a page. It is a valid html page with a closing </html> tag, but it is missing the content I want and the standard search rows are in a hidden div.


I have attached a "good" and a "bad" result page to illustrate my point.


I have experimented with various query string parameters and tried to use UK proxies only, as I am in UK, but none of this has helped.


Is this something anyone has encountered before? Is Bing just too cleaver to be crawled like this?


Any help would be welcome.


Thanks Bob.

html
(233 KB)
html
(68 KB)

Best Answer

Hello Bob,


Request you to share how you are making the requests to Bing and what headers are being passed.

1 Comment

Answer

Hello Bob,


Request you to share how you are making the requests to Bing and what headers are being passed.

Login to post a comment