Start a new topic
Answered

Crawlera 503 Ban

Hi everyone,


I'm using a C10 plan in order to crawl a french website.


But every request are in 503, with those headers :


  

HTTP/1.1 200 OK

HTTP/1.1 503 Service Unavailable
Connection: close
Date: Sun, 25 Jun 2017 01:19:48 GMT
Proxy-Connection: close
Transfer-Encoding: chunked
X-Crawlera-Error: banned
X-Crawlera-Slave: 107.173.143.85:3128
X-Crawlera-Version: 1.30.16-c3ff4a

Website crawl ban

  


What does exactly means ?

Will Crawlera be permanently banned from this domain ?


What can I do ?


Thank you.



Best Answer

Hello,


Whenever Crawlera gets a ban response it retries the request with other IP. By default it re-tries 5 times to get successful response and then gives the 503 error if it still gets unsuccessful response. Hence Crawlera is not permanently banned from the domain.


And from your logs, I can see that you have made successful responses as well.


Answer

Hello,


Whenever Crawlera gets a ban response it retries the request with other IP. By default it re-tries 5 times to get successful response and then gives the 503 error if it still gets unsuccessful response. Hence Crawlera is not permanently banned from the domain.


And from your logs, I can see that you have made successful responses as well.

Last few days i was facing same issues in Crawlera API using c# and Node js

Scarping Domain: HTTPS

Error:

   connection: 'close',

  date: 'Mon, 20 Aug 2018 08:17:37 GMT',

  'proxy-connection': 'close',

  'transfer-encoding': 'chunked',

  'x-crawlera-error''banned',

  'x-crawlera-slave': '185.189.45.61:8800',

  'x-crawlera-version': '1.33.1-68f021' }


still  i am using update version of request npm packkage


Repro Steps:

var options = { method: 'POST',

  url:<url>,

  timeout: 30000,

  ca: crawlera-cs.crt,

  requestCert: true,

  rejectUnauthorized: true,

  headers: 

   { 'Content-Type': 'application/x-www-form-urlencoded',

     'Cache-Control': 'no-cache',

     host: <hostname>,

     Origin: <origin>,

     Cookie: <cookie>,

     'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',

     'X-Crawlera-Max-Retries': 1 },

  form: <formData>}


var new_req = request.defaults({

proxy: "http://<APIKEY>:@<Host>:<PORT>"

});


How do i fix this issue in c# and  node js


1 person likes this

+1

Login to post a comment