No recent searches
Popular Articles
Sorry! nothing found for
Posted over 7 years ago by crounauer
Will using crawlera with the proxy service navigate around the problem of being redirected buy a website when trying to crawl?
I am getting this header =>
HTTP/1.1 200 OK Date: Mon, 26 Jun 2017 20:00:50 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive Vary: Accept-Encoding Expires: Thu, 01 Jan 1970 00:00:01 GMT Cache-Control: no-cache Cache-Control: private, no-cache, no-store, must-revalidate Edge-Control: no-store, bypass-cache Surrogate-Control: no-store, bypass-cache
and then being redirected to => distil_r_blocked.html
Thanks, Simon
0 Votes
vaz posted over 7 years ago Best Answer
Hi Simon, perhaps the site is using Distil Networks to protect from crawlers and don't allow any extractor method.
If you want, you can let us know more about your needs through our quote request and our developers can provide a free budget to help you further.
Have a nice day!
Pablo
2 Comments
dcweeks posted over 6 years ago
So crawlera cannot get around Distil?
1 Votes
vaz posted over 7 years ago Answer
Login to post a comment
People who like this
This post will be deleted permanently. Are you sure?
Will using crawlera with the proxy service navigate around the problem of being redirected buy a website when trying to crawl?
I am getting this header =>
HTTP/1.1 200 OK Date: Mon, 26 Jun 2017 20:00:50 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive Vary: Accept-Encoding Expires: Thu, 01 Jan 1970 00:00:01 GMT Cache-Control: no-cache Cache-Control: private, no-cache, no-store, must-revalidate Edge-Control: no-store, bypass-cache Surrogate-Control: no-store, bypass-cache
and then being redirected to => distil_r_blocked.html
Thanks, Simon
0 Votes
vaz posted over 7 years ago Best Answer
Hi Simon, perhaps the site is using Distil Networks to protect from crawlers and don't allow any extractor method.
If you want, you can let us know more about your needs through our quote request and our developers can provide a free budget to help you further.
Have a nice day!
Pablo
0 Votes
2 Comments
dcweeks posted over 6 years ago
So crawlera cannot get around Distil?
1 Votes
vaz posted over 7 years ago Answer
Hi Simon, perhaps the site is using Distil Networks to protect from crawlers and don't allow any extractor method.
If you want, you can let us know more about your needs through our quote request and our developers can provide a free budget to help you further.
Have a nice day!
Pablo
0 Votes
Login to post a comment