Zyte Support Center

Will using crawlera with the proxy service navigate around the problem of being redirected buy a website when trying to crawl?

I am getting this header =>

HTTP/1.1 200 OK Date: Mon, 26 Jun 2017 20:00:50 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive Vary: Accept-Encoding Expires: Thu, 01 Jan 1970 00:00:01 GMT Cache-Control: no-cache Cache-Control: private, no-cache, no-store, must-revalidate Edge-Control: no-store, bypass-cache Surrogate-Control: no-store, bypass-cache

and then being redirected to => distil_r_blocked.html

Thanks, Simon

So crawlera cannot get around Distil? 

Hi Simon, perhaps the site is using Distil Networks to protect from crawlers and don't allow any extractor method.

If you want, you can let us know more about your needs through our quote request and our developers can provide a free budget to help you further.

Have a nice day!

Pablo