I'm crawling a site which allows a limit number of requests per day. When you reach the limit it just responds with a valid page (HTTP 200 code), but the returned page has a text saying you reached you daily limit. Is there a way to inform Crawlera (via Scrapy) that the requested page should be regarded as a ban? Aka can I extend the ban rules?
I'm crawling a site which allows a limit number of requests per day. When you reach the limit it just responds with a valid page (HTTP 200 code), but the returned page has a text saying you reached you daily limit. Is there a way to inform Crawlera (via Scrapy) that the requested page should be regarded as a ban? Aka can I extend the ban rules?
Maybe somewhat related:
https://support.scrapinghub.com/support/discussions/topics/22000009296
0 Votes
thriveni posted over 6 years ago Admin Best Answer
Hi,
I would request you to Contact Support through Scrapinghub Dashboard > Help.
0 Votes
2 Comments
Arjen Vellinga posted over 6 years ago
done!
0 Votes
thriveni posted over 6 years ago Admin Answer
Hi,
I would request you to Contact Support through Scrapinghub Dashboard > Help.
0 Votes
Login to post a comment