How is it using crawlera for a website which is secured by Cloudflare?
a
andreyuhai
started a topic
about 5 years ago
Hello there,
I have been scraping a website with Cloudflare security system. I would like to subscribe to a plan here for Crawlera but I also saw some bad comments about Crawlera.
Since I am a student I do not want to waste my money which is not so much.
So is there anyone who had experience scraping a well secured website with Crawlera, especially protected with Cloudflare and having captchas?
Cloudfare also employs reCaptcha to weed out bots and in such cases you may need to use something like 2Captcha API with splash/headless chrome to get around it. However as an initial step you can use Crawlera to assess the level of protection employed by the site for requests emanating from a certain region. For example the URL https://nitrogensports.eu/dice/play when accessed from a non-US region redirects to a reCaptcha page. However the same URL can be accessed from a US based IP without having to solve the reCaptcha.
Cloudfare also employs reCaptcha to weed out bots and in such cases you may need to use something like 2Captcha API with splash/headless chrome to get around it. However as an initial step you can use Crawlera to assess the level of protection employed by the site for requests emanating from a certain region. For example the URL https://nitrogensports.eu/dice/play when accessed from a non-US region redirects to a reCaptcha page. However the same URL can be accessed from a US based IP without having to solve the reCaptcha.
andreyuhai
Hello there,
I have been scraping a website with Cloudflare security system. I would like to subscribe to a plan here for Crawlera but I also saw some bad comments about Crawlera.
Since I am a student I do not want to waste my money which is not so much.
So is there anyone who had experience scraping a well secured website with Crawlera, especially protected with Cloudflare and having captchas?
Cheers!
Cloudfare employs 2-3 different flavors of bot protection. Some of them can be addressed by using the `cfscrape` library - https://github.com/Anorov/cloudflare-scrape
Cloudfare also employs reCaptcha to weed out bots and in such cases you may need to use something like 2Captcha API with splash/headless chrome to get around it. However as an initial step you can use Crawlera to assess the level of protection employed by the site for requests emanating from a certain region. For example the URL https://nitrogensports.eu/dice/play when accessed from a non-US region redirects to a reCaptcha page. However the same URL can be accessed from a US based IP without having to solve the reCaptcha.
Adriana Anghel
Cloudfare employs 2-3 different flavors of bot protection. Some of them can be addressed by using the `cfscrape` library - https://github.com/Anorov/cloudflare-scrape
Cloudfare also employs reCaptcha to weed out bots and in such cases you may need to use something like 2Captcha API with splash/headless chrome to get around it. However as an initial step you can use Crawlera to assess the level of protection employed by the site for requests emanating from a certain region. For example the URL https://nitrogensports.eu/dice/play when accessed from a non-US region redirects to a reCaptcha page. However the same URL can be accessed from a US based IP without having to solve the reCaptcha.
1 person likes this
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 395 topics