I could not find on the site if it is possible to use robots that bypass robots.txt through crawlera proxies.
thanks!
0 Votes
nestor posted
over 6 years ago
AdminBest Answer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
0 Votes
2 Comments
Sorted by
nestorposted
over 6 years ago
AdminAnswer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
0 Votes
s
seuarnaldoposted
over 6 years ago
Sorry.
English: Does the Crawlera accepts ROBOTSTXT_OBEY = False?
I could not find on the site if it is possible to use robots that bypass robots.txt through crawlera proxies.
thanks!
0 Votes
nestor posted over 6 years ago Admin Best Answer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
0 Votes
2 Comments
nestor posted over 6 years ago Admin Answer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
0 Votes
seuarnaldo posted over 6 years ago
Sorry.
English: Does the Crawlera accepts ROBOTSTXT_OBEY = False?
0 Votes
Login to post a comment