Crawlera 503 Ban

T

Tilotiti

started a topic almost 7 years ago

Hi everyone,

I'm using a C10 plan in order to crawl a french website.

But every request are in 503, with those headers :

HTTP/1.1 200 OK

HTTP/1.1 503 Service Unavailable
Connection: close
Date: Sun, 25 Jun 2017 01:19:48 GMT
Proxy-Connection: close
Transfer-Encoding: chunked
X-Crawlera-Error: banned
X-Crawlera-Slave: 107.173.143.85:3128
X-Crawlera-Version: 1.30.16-c3ff4a

Website crawl ban

What does exactly means ?

Will Crawlera be permanently banned from this domain ?

What can I do ?

Thank you.

Best Answer

t

thriveni said almost 7 years ago

Hello,

Whenever Crawlera gets a ban response it retries the request with other IP. By default it re-tries 5 times to get successful response and then gives the 503 error if it still gets unsuccessful response. Hence Crawlera is not permanently banned from the domain.

And from your logs, I can see that you have made successful responses as well.

thriveni

said almost 7 years ago

Answer

Hello,

Whenever Crawlera gets a ban response it retries the request with other IP. By default it re-tries 5 times to get successful response and then gives the 503 error if it still gets unsuccessful response. Hence Crawlera is not permanently banned from the domain.

And from your logs, I can see that you have made successful responses as well.

T

Tech Services

said over 5 years ago

Last few days i was facing same issues in Crawlera API using c# and Node js

Scarping Domain: HTTPS

Error:

{

connection: 'close',

date: 'Mon, 20 Aug 2018 08:17:37 GMT',

'proxy-connection': 'close',

'transfer-encoding': 'chunked',

'x-crawlera-error': 'banned',

'x-crawlera-slave': '185.189.45.61:8800',

'x-crawlera-version': '1.33.1-68f021' }

still i am using update version of request npm packkage

Repro Steps:

var options = { method: 'POST',

url:<url>,

timeout: 30000,

ca: crawlera-cs.crt,

requestCert: true,

rejectUnauthorized: true,

headers:

{ 'Content-Type': 'application/x-www-form-urlencoded',

'Cache-Control': 'no-cache',

host: <hostname>,

Origin: <origin>,

Cookie: <cookie>,

'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',

'X-Crawlera-Max-Retries': 1 },

form: <formData>}

var new_req = request.defaults({

proxy: "http://<APIKEY>:@<Host>:<PORT>"

});

How do i fix this issue in c# and node js

K

Karan Khanna