Will using crawlera with the proxy service navigate around the problem of being redirected buy a website when trying to crawl?
I am getting this header =>
HTTP/1.1 200 OK Date: Mon, 26 Jun 2017 20:00:50 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive Vary: Accept-Encoding Expires: Thu, 01 Jan 1970 00:00:01 GMT Cache-Control: no-cache Cache-Control: private, no-cache, no-store, must-revalidate Edge-Control: no-store, bypass-cache Surrogate-Control: no-store, bypass-cache
and then being redirected to => distil_r_blocked.html
Thanks, Simon
Hi Simon, perhaps the site is using Distil Networks to protect from crawlers and don't allow any extractor method.
If you want, you can let us know more about your needs through our quote request and our developers can provide a free budget to help you further.
Have a nice day!
Pablo
So crawlera cannot get around Distil?
crounauer
Will using crawlera with the proxy service navigate around the problem of being redirected buy a website when trying to crawl?
I am getting this header =>
HTTP/1.1 200 OK Date: Mon, 26 Jun 2017 20:00:50 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive Vary: Accept-Encoding Expires: Thu, 01 Jan 1970 00:00:01 GMT Cache-Control: no-cache Cache-Control: private, no-cache, no-store, must-revalidate Edge-Control: no-store, bypass-cache Surrogate-Control: no-store, bypass-cache
and then being redirected to => distil_r_blocked.html
Thanks, Simon
Hi Simon, perhaps the site is using Distil Networks to protect from crawlers and don't allow any extractor method.
If you want, you can let us know more about your needs through our quote request and our developers can provide a free budget to help you further.
Have a nice day!
Pablo
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstvaz
Hi Simon, perhaps the site is using Distil Networks to protect from crawlers and don't allow any extractor method.
If you want, you can let us know more about your needs through our quote request and our developers can provide a free budget to help you further.
Have a nice day!
Pablo
dcweeks
So crawlera cannot get around Distil?
1 person likes this
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 381 topics