Hello,
my crawler is downloading images it has found to an AWS S3 bucket.
However, after only 30 seconds of crawling it starts spitting out this warning:
[urllib3.connectionpool] Connection pool is full, discarding connection: <S3 bucket url>
I have configured it according to this documentation:
https://docs.scrapy.org/en/latest/topics/media-pipeline.html
log234
Hello,
my crawler is downloading images it has found to an AWS S3 bucket.
However, after only 30 seconds of crawling it starts spitting out this warning:
[urllib3.connectionpool] Connection pool is full, discarding connection: <S3 bucket url>
I have configured it according to this documentation:
https://docs.scrapy.org/en/latest/topics/media-pipeline.html
2 people have this question