What is the recommended way of handling scenarios where we run out credits? This happened to us recently which meant our scrapers were failing until we upgraded our subscription to C200. We'd like to simply fall back on regular direct connections in such a case.
We use Python with Requests and Scrapy. There doesn't appear to be anything in the docs that suggests how to check for this specific scenario.
0 Votes
nestor posted
about 6 years ago
AdminBest Answer
The recommended way is to upgrade your plan, we send an email once you reach 80% of your request quota.
There's nothing out-of-the-box to do what you are asking. You'll need to implement something that detects the HTTP response code "403 user suspended" and start another crawl process where settings have CRAWLERA_ENABLED=FALSE.
0 Votes
1 Comments
nestorposted
about 6 years ago
AdminAnswer
The recommended way is to upgrade your plan, we send an email once you reach 80% of your request quota.
There's nothing out-of-the-box to do what you are asking. You'll need to implement something that detects the HTTP response code "403 user suspended" and start another crawl process where settings have CRAWLERA_ENABLED=FALSE.
What is the recommended way of handling scenarios where we run out credits? This happened to us recently which meant our scrapers were failing until we upgraded our subscription to C200. We'd like to simply fall back on regular direct connections in such a case.
We use Python with Requests and Scrapy. There doesn't appear to be anything in the docs that suggests how to check for this specific scenario.
0 Votes
nestor posted about 6 years ago Admin Best Answer
The recommended way is to upgrade your plan, we send an email once you reach 80% of your request quota.
There's nothing out-of-the-box to do what you are asking. You'll need to implement something that detects the HTTP response code "403 user suspended" and start another crawl process where settings have CRAWLERA_ENABLED=FALSE.
0 Votes
1 Comments
nestor posted about 6 years ago Admin Answer
The recommended way is to upgrade your plan, we send an email once you reach 80% of your request quota.
There's nothing out-of-the-box to do what you are asking. You'll need to implement something that detects the HTTP response code "403 user suspended" and start another crawl process where settings have CRAWLERA_ENABLED=FALSE.
0 Votes
Login to post a comment