Start a new topic
Answered

Ban is just an HTTP 200 code with specific returned text

I'm crawling a site which allows a limit number of requests per day. When you reach the limit it just responds with a valid page (HTTP 200 code), but the returned page has a text saying you reached you daily limit. Is there a way to inform Crawlera (via Scrapy) that the requested page should be regarded as a ban? Aka can I extend the ban rules?


Maybe somewhat related:

https://support.scrapinghub.com/support/discussions/topics/22000009296


Best Answer

Hi,


I would request you to Contact Support through Scrapinghub Dashboard > Help.



Answer

Hi,


I would request you to Contact Support through Scrapinghub Dashboard > Help.


done!

Login to post a comment