If I have a scraper set up to send a request to a URL, scrape a URL from the HTML that gets returned, and then download the file at the URL, does that count as one request towards Crawlera's monthly limit, or two?
0 Votes
peixoto posted
almost 6 years ago
AdminBest Answer
Hello,
If you have to issue a separate request to download the content, then it would count as two different requests.
Crawlera accounts for all successful requests it delivers to your spider. If the requests are marked as banned or failed, they do not count.
Hope this is the info you were looking for.
1 Votes
2 Comments
Sorted by
peixotoposted
almost 6 years ago
AdminAnswer
Hello,
If you have to issue a separate request to download the content, then it would count as two different requests.
Crawlera accounts for all successful requests it delivers to your spider. If the requests are marked as banned or failed, they do not count.
If I have a scraper set up to send a request to a URL, scrape a URL from the HTML that gets returned, and then download the file at the URL, does that count as one request towards Crawlera's monthly limit, or two?
0 Votes
peixoto posted almost 6 years ago Admin Best Answer
Hello,
If you have to issue a separate request to download the content, then it would count as two different requests.
Crawlera accounts for all successful requests it delivers to your spider. If the requests are marked as banned or failed, they do not count.
Hope this is the info you were looking for.
1 Votes
2 Comments
peixoto posted almost 6 years ago Admin Answer
Hello,
If you have to issue a separate request to download the content, then it would count as two different requests.
Crawlera accounts for all successful requests it delivers to your spider. If the requests are marked as banned or failed, they do not count.
Hope this is the info you were looking for.
1 Votes
nathanwailes posted almost 6 years ago
Thanks!
0 Votes
Login to post a comment