videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic
Answered

429 Client Error: Internal Server Error

Hi,

How to avoid 429 Client Error: Internal Server Error using  crawlera?

Is this just an issues of there aren't enough proxy IPs? How can   I avoid exceeding the rate-limit  in a given period of time?


Best Answer

You need to limit the number of concurrent request to what your plan allows. A C10 allows for up to 10 concurrent requests. If you exceed it, Crawlera will reject the request with a 429 code.

1 Comment

Answer

You need to limit the number of concurrent request to what your plan allows. A C10 allows for up to 10 concurrent requests. If you exceed it, Crawlera will reject the request with a 429 code.

Login to post a comment