I could not find on the site if it is possible to use robots that bypass robots.txt through crawlera proxies.
thanks!
Best Answer
n
nestor
said
over 5 years ago
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
s
seuarnaldo
said
over 5 years ago
Sorry.
English: Does the Crawlera accepts ROBOTSTXT_OBEY = False?
seuarnaldo
I could not find on the site if it is possible to use robots that bypass robots.txt through crawlera proxies.
thanks!
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
- Oldest First
- Popular
- Newest First
Sorted by Newest Firstnestor
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
seuarnaldo
Sorry.
English: Does the Crawlera accepts ROBOTSTXT_OBEY = False?
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 395 topics