If I have a scraper set up to send a request to a URL, scrape a URL from the HTML that gets returned, and then download the file at the URL, does that count as one request towards Crawlera's monthly limit, or two?
0 Votes
peixoto posted
over 5 years ago
AdminBest Answer
Hello,
If you have to issue a separate request to download the content, then it would count as two different requests.
Crawlera accounts for all successful requests it delivers to your spider. If the requests are marked as banned or failed, they do not count.
Hope this is the info you were looking for.
1 Votes
2 Comments
Sorted by
n
nathanwailesposted
over 5 years ago
Thanks!
0 Votes
peixotoposted
over 5 years ago
AdminAnswer
Hello,
If you have to issue a separate request to download the content, then it would count as two different requests.
Crawlera accounts for all successful requests it delivers to your spider. If the requests are marked as banned or failed, they do not count.
If I have a scraper set up to send a request to a URL, scrape a URL from the HTML that gets returned, and then download the file at the URL, does that count as one request towards Crawlera's monthly limit, or two?
0 Votes
peixoto posted over 5 years ago Admin Best Answer
Hello,
If you have to issue a separate request to download the content, then it would count as two different requests.
Crawlera accounts for all successful requests it delivers to your spider. If the requests are marked as banned or failed, they do not count.
Hope this is the info you were looking for.
1 Votes
2 Comments
nathanwailes posted over 5 years ago
Thanks!
0 Votes
peixoto posted over 5 years ago Admin Answer
Hello,
If you have to issue a separate request to download the content, then it would count as two different requests.
Crawlera accounts for all successful requests it delivers to your spider. If the requests are marked as banned or failed, they do not count.
Hope this is the info you were looking for.
1 Votes
Login to post a comment