I'm using a C10 plan in order to crawl a french website.
But every request are in 503, with those headers :
HTTP/1.1 200 OK
HTTP/1.1 503 Service Unavailable
Connection: close
Date: Sun, 25 Jun 2017 01:19:48 GMT
Proxy-Connection: close
Transfer-Encoding: chunked
X-Crawlera-Error: banned
X-Crawlera-Slave: 107.173.143.85:3128
X-Crawlera-Version: 1.30.16-c3ff4a
Website crawl ban
What does exactly means ?
Will Crawlera be permanently banned from this domain ?
What can I do ?
Thank you.
0 Votes
thriveni posted
over 7 years ago
AdminBest Answer
Hello,
Whenever Crawlera gets a ban response it retries the request with other IP. By default it re-tries 5 times to get successful response and then gives the 503 error if it still gets unsuccessful response. Hence Crawlera is not permanently banned from the domain.
And from your logs, I can see that you have made successful responses as well.
0 Votes
3 Comments
Sorted by
K
Karan Khannaposted
almost 6 years ago
+1
0 Votes
T
Tech Servicesposted
about 6 years ago
Last few days i was facing same issues in Crawlera API using c# and Node js
Scarping Domain: HTTPS
Error:
{
connection: 'close',
date: 'Mon, 20 Aug 2018 08:17:37 GMT',
'proxy-connection': 'close',
'transfer-encoding': 'chunked',
'x-crawlera-error': 'banned',
'x-crawlera-slave': '185.189.45.61:8800',
'x-crawlera-version': '1.33.1-68f021' }
still i am using update version of request npm packkage
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',
'X-Crawlera-Max-Retries': 1 },
form: <formData>}
var new_req = request.defaults({
proxy: "http://<APIKEY>:@<Host>:<PORT>"
});
How do i fix this issue in c# and node js
1 Votes
thriveniposted
over 7 years ago
AdminAnswer
Hello,
Whenever Crawlera gets a ban response it retries the request with other IP. By default it re-tries 5 times to get successful response and then gives the 503 error if it still gets unsuccessful response. Hence Crawlera is not permanently banned from the domain.
And from your logs, I can see that you have made successful responses as well.
Hi everyone,
I'm using a C10 plan in order to crawl a french website.
But every request are in 503, with those headers :
What does exactly means ?
Will Crawlera be permanently banned from this domain ?
What can I do ?
Thank you.
0 Votes
thriveni posted over 7 years ago Admin Best Answer
Hello,
Whenever Crawlera gets a ban response it retries the request with other IP. By default it re-tries 5 times to get successful response and then gives the 503 error if it still gets unsuccessful response. Hence Crawlera is not permanently banned from the domain.
And from your logs, I can see that you have made successful responses as well.
0 Votes
3 Comments
Karan Khanna posted almost 6 years ago
+1
0 Votes
Tech Services posted about 6 years ago
Last few days i was facing same issues in Crawlera API using c# and Node js
Scarping Domain: HTTPS
Error:
{
connection: 'close',
date: 'Mon, 20 Aug 2018 08:17:37 GMT',
'proxy-connection': 'close',
'transfer-encoding': 'chunked',
'x-crawlera-error': 'banned',
'x-crawlera-slave': '185.189.45.61:8800',
'x-crawlera-version': '1.33.1-68f021' }
still i am using update version of request npm packkage
Repro Steps:
var options = { method: 'POST',
url:<url>,
timeout: 30000,
ca: crawlera-cs.crt,
requestCert: true,
rejectUnauthorized: true,
headers:
{ 'Content-Type': 'application/x-www-form-urlencoded',
'Cache-Control': 'no-cache',
host: <hostname>,
Origin: <origin>,
Cookie: <cookie>,
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',
'X-Crawlera-Max-Retries': 1 },
form: <formData>}
var new_req = request.defaults({
proxy: "http://<APIKEY>:@<Host>:<PORT>"
});
How do i fix this issue in c# and node js
1 Votes
thriveni posted over 7 years ago Admin Answer
Hello,
Whenever Crawlera gets a ban response it retries the request with other IP. By default it re-tries 5 times to get successful response and then gives the 503 error if it still gets unsuccessful response. Hence Crawlera is not permanently banned from the domain.
And from your logs, I can see that you have made successful responses as well.
0 Votes
Login to post a comment