I could not find on the site if it is possible to use robots that bypass robots.txt through crawlera proxies.
thanks!
Best Answer
n
nestor
said
about 5 years ago
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
English: Does the Crawlera accepts ROBOTSTXT_OBEY = False?
nestor
said
about 5 years ago
Answer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
seuarnaldo
I could not find on the site if it is possible to use robots that bypass robots.txt through crawlera proxies.
thanks!
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstseuarnaldo
Sorry.
English: Does the Crawlera accepts ROBOTSTXT_OBEY = False?
nestor
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 389 topics