Ban is just an HTTP 200 code with specific returned text
A
Arjen Vellinga
started a topic
over 5 years ago
I'm crawling a site which allows a limit number of requests per day. When you reach the limit it just responds with a valid page (HTTP 200 code), but the returned page has a text saying you reached you daily limit. Is there a way to inform Crawlera (via Scrapy) that the requested page should be regarded as a ban? Aka can I extend the ban rules?
Arjen Vellinga
I'm crawling a site which allows a limit number of requests per day. When you reach the limit it just responds with a valid page (HTTP 200 code), but the returned page has a text saying you reached you daily limit. Is there a way to inform Crawlera (via Scrapy) that the requested page should be regarded as a ban? Aka can I extend the ban rules?
Maybe somewhat related:
https://support.scrapinghub.com/support/discussions/topics/22000009296
Hi,
I would request you to Contact Support through Scrapinghub Dashboard > Help.
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstthriveni
Hi,
I would request you to Contact Support through Scrapinghub Dashboard > Help.
Arjen Vellinga
done!
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 395 topics