I could not find on the site if it is possible to use robots that bypass robots.txt through crawlera proxies.
thanks!
0 Votes
nestor posted
almost 6 years ago
AdminBest Answer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
0 Votes
2 Comments
Sorted by
s
seuarnaldoposted
almost 6 years ago
Sorry.
English: Does the Crawlera accepts ROBOTSTXT_OBEY = False?
0 Votes
nestorposted
almost 6 years ago
AdminAnswer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
I could not find on the site if it is possible to use robots that bypass robots.txt through crawlera proxies.
thanks!
0 Votes
nestor posted almost 6 years ago Admin Best Answer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
0 Votes
2 Comments
seuarnaldo posted almost 6 years ago
Sorry.
English: Does the Crawlera accepts ROBOTSTXT_OBEY = False?
0 Votes
nestor posted almost 6 years ago Admin Answer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
0 Votes
Login to post a comment