Start a new topic

Error 404 when using Session ID


I got 404 error for my second request when i use session id.

I am creating Session Id with my first request

headers={'X-Crawlera-Session': 'create'},

getting session id in response

session_id = response.headers.get('X-Crawlera-Session', '')

and sending second request



 headers={'X-Crawlera-Session': session_id}, 



When crawlera is disabled, everything works fine

A 404 typically means URL not found, are you sure that's the error code you're getting?

I got --> [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404...>: HTTP status code is not handled or not allowed

Everything works fine when crawlera is disabled

I see it's POST request. Try adding the header "X-Crawlera-Cookies: disable". If that doesn't work, I would suggest you open a ticket (Help > Contact Support) and provide the data you are sending on the request.

Login to post a comment