How can we help you today?

Start a new topic
Discussions Zyte Smart Proxy Manager (formerly Crawlera) Technical Help
Answered

scrapy FormRequest is not working when I use crawlera else it works fine!

Hi everyone. I just started using crawlera.

I have a scraper written in python scrapy, it goes to a url, logs in and fill search forms and scrape data. And it works fine without crawlera. When I use crawlera middleware, it can't log in to the website.

Please help me in this

Best Answer

You'll probably need to make use of Crawlera Sessions if you need to retain the same IP after login in; by default Crawlera routes every request through a different IP so the login cookie is probably lost. For more on Crawlera Sessions, please see: https://doc.scrapinghub.com/crawlera.html#sessions.


Also if you wish to handle cookies on your side, you can use "X-Crawlera-Cookies: disable" header (https://doc.scrapinghub.com/crawlera.html#x-crawlera-cookies)

1 Comment
Answer

You'll probably need to make use of Crawlera Sessions if you need to retain the same IP after login in; by default Crawlera routes every request through a different IP so the login cookie is probably lost. For more on Crawlera Sessions, please see: https://doc.scrapinghub.com/crawlera.html#sessions.


Also if you wish to handle cookies on your side, you can use "X-Crawlera-Cookies: disable" header (https://doc.scrapinghub.com/crawlera.html#x-crawlera-cookies)

Login to post a comment
More topics in Technical Help
See all 395 topics