If I have a scraper set up to send a request to a URL, scrape a URL from the HTML that gets returned, and then download the file at the URL, does that count as one request towards Crawlera's monthly limit, or two?
Best Answer
p
peixoto
said
over 4 years ago
Hello,
If you have to issue a separate request to download the content, then it would count as two different requests.
Crawlera accounts for all successful requests it delivers to your spider. If the requests are marked as banned or failed, they do not count.
nathanwailes
If I have a scraper set up to send a request to a URL, scrape a URL from the HTML that gets returned, and then download the file at the URL, does that count as one request towards Crawlera's monthly limit, or two?
Hello,
If you have to issue a separate request to download the content, then it would count as two different requests.
Crawlera accounts for all successful requests it delivers to your spider. If the requests are marked as banned or failed, they do not count.
Hope this is the info you were looking for.
- Oldest First
- Popular
- Newest First
Sorted by Newest Firstnathanwailes
Thanks!
peixoto
Hello,
If you have to issue a separate request to download the content, then it would count as two different requests.
Crawlera accounts for all successful requests it delivers to your spider. If the requests are marked as banned or failed, they do not count.
Hope this is the info you were looking for.
1 person likes this
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 395 topics