scrapy FormRequest is not working when I use crawlera else it works fine!
h
howard1
started a topic
over 6 years ago
Hi everyone. I just started using crawlera.
I have a scraper written in python scrapy, it goes to a url, logs in and fill search forms and scrape data. And it works fine without crawlera. When I use crawlera middleware, it can't log in to the website.
Please help me in this
Best Answer
n
nestor
said
over 6 years ago
You'll probably need to make use of Crawlera Sessions if you need to retain the same IP after login in; by default Crawlera routes every request through a different IP so the login cookie is probably lost. For more on Crawlera Sessions, please see: https://doc.scrapinghub.com/crawlera.html#sessions.
You'll probably need to make use of Crawlera Sessions if you need to retain the same IP after login in; by default Crawlera routes every request through a different IP so the login cookie is probably lost. For more on Crawlera Sessions, please see: https://doc.scrapinghub.com/crawlera.html#sessions.
howard1
Hi everyone. I just started using crawlera.
I have a scraper written in python scrapy, it goes to a url, logs in and fill search forms and scrape data. And it works fine without crawlera. When I use crawlera middleware, it can't log in to the website.
Please help me in this
You'll probably need to make use of Crawlera Sessions if you need to retain the same IP after login in; by default Crawlera routes every request through a different IP so the login cookie is probably lost. For more on Crawlera Sessions, please see: https://doc.scrapinghub.com/crawlera.html#sessions.
Also if you wish to handle cookies on your side, you can use "X-Crawlera-Cookies: disable" header (https://doc.scrapinghub.com/crawlera.html#x-crawlera-cookies)
nestor
You'll probably need to make use of Crawlera Sessions if you need to retain the same IP after login in; by default Crawlera routes every request through a different IP so the login cookie is probably lost. For more on Crawlera Sessions, please see: https://doc.scrapinghub.com/crawlera.html#sessions.
Also if you wish to handle cookies on your side, you can use "X-Crawlera-Cookies: disable" header (https://doc.scrapinghub.com/crawlera.html#x-crawlera-cookies)
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 399 topics