I read that Crawlera treats a 302 redirect as a successful request, but what if it's actually an anti-spider response from the server? This happened to me when I tried to use the POST method, only to be rebuffed and redirected to an authentication page. Is there a way to manually ask Crawlera to retry with a new IP address when that happens?
We can add the case as a ban rule so that when such a response is received Crawlera would treat it as ban and retry.
However, we would need to check the flow of the pages and if the redirects are valid. Does it need authentication. Does the retry to the request gives successful response.
I read that Crawlera treats a 302 redirect as a successful request, but what if it's actually an anti-spider response from the server? This happened to me when I tried to use the POST method, only to be rebuffed and redirected to an authentication page. Is there a way to manually ask Crawlera to retry with a new IP address when that happens?
0 Votes
1 Comments
thriveni posted over 4 years ago Admin
Hello,
We can add the case as a ban rule so that when such a response is received Crawlera would treat it as ban and retry.
However, we would need to check the flow of the pages and if the redirects are valid. Does it need authentication. Does the retry to the request gives successful response.
0 Votes
Login to post a comment