I could not find on the site if it is possible to use robots that bypass robots.txt through crawlera proxies.
thanks!
0 Votes
Nestor Toledo posted
almost 8 years ago
AdminBest Answer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
0 Votes
2 Comments
Nestor Toledoposted
almost 8 years ago
AdminAnswer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
0 Votes
s
seuarnaldoposted
almost 8 years ago
Sorry.
English: Does the Crawlera accepts ROBOTSTXT_OBEY = False?
I could not find on the site if it is possible to use robots that bypass robots.txt through crawlera proxies.
thanks!
0 Votes
Nestor Toledo posted almost 8 years ago Admin Best Answer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
0 Votes
2 Comments
Nestor Toledo posted almost 8 years ago Admin Answer
That's up to the spider. Crawlera will just route requests through different proxies. If your spider doesn't obey robots.txt Crawlera would work either way.
0 Votes
seuarnaldo posted almost 8 years ago
Sorry.
English: Does the Crawlera accepts ROBOTSTXT_OBEY = False?
0 Votes
Login to post a comment