Bing

Hi,

I am trying to use Crawlera to get some information about 1000 or so businesses in my local area from Bing.

Originally, I was planning on using the Bing search API. However the API only returns the basic search results. What I am more interested in is the information on the side bar showing links to the TripAdvisor page etc...

If I do a basic search building a query string and using the Crawlera proxy, about 1 in 10 times I get the full page as I expect. The rest of the time, I get half a page. It is a valid html page with a closing </html> tag, but it is missing the content I want and the standard search rows are in a hidden div.

I have attached a "good" and a "bad" result page to illustrate my point.

I have experimented with various query string parameters and tried to use UK proxies only, as I am in UK, but none of this has helped.

Is this something anyone has encountered before? Is Bing just too cleaver to be crawled like this?

Any help would be welcome.

Thanks Bob.