Website redirects

Posted almost 8 years ago by crounauer

Post a topic

Answered

crounauer

Will using crawlera with the proxy service navigate around the problem of being redirected buy a website when trying to crawl?

I am getting this header =>

HTTP/1.1 200 OK Date: Mon, 26 Jun 2017 20:00:50 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive Vary: Accept-Encoding Expires: Thu, 01 Jan 1970 00:00:01 GMT Cache-Control: no-cache Cache-Control: private, no-cache, no-store, must-revalidate Edge-Control: no-store, bypass-cache Surrogate-Control: no-store, bypass-cache

and then being redirected to => distil_r_blocked.html

Thanks, Simon

0 Votes

vaz posted almost 8 years ago Best Answer

Hi Simon, perhaps the site is using Distil Networks to protect from crawlers and don't allow any extractor method.

If you want, you can let us know more about your needs through our quote request and our developers can provide a free budget to help you further.

Have a nice day!

Pablo

0 Votes

2 Comments

dcweeks posted about 7 years ago

So crawlera cannot get around Distil?

1 Votes

vaz posted almost 8 years ago Answer

Hi Simon, perhaps the site is using Distil Networks to protect from crawlers and don't allow any extractor method.

If you want, you can let us know more about your needs through our quote request and our developers can provide a free budget to help you further.

Have a nice day!

Pablo

0 Votes