No recent searches
Popular Articles
Sorry! nothing found for
Posted almost 5 years ago by log234
Hello,
my crawler is downloading images it has found to an AWS S3 bucket.
However, after only 30 seconds of crawling it starts spitting out this warning:
[urllib3.connectionpool] Connection pool is full, discarding connection: <S3 bucket url>
I have configured it according to this documentation:
https://docs.scrapy.org/en/latest/topics/media-pipeline.html
2 Votes
0 Comments
Login to post a comment
People who like this
This post will be deleted permanently. Are you sure?
Hello,
my crawler is downloading images it has found to an AWS S3 bucket.
However, after only 30 seconds of crawling it starts spitting out this warning:
[urllib3.connectionpool] Connection pool is full, discarding connection: <S3 bucket url>
I have configured it according to this documentation:
https://docs.scrapy.org/en/latest/topics/media-pipeline.html
2 Votes
0 Comments
Login to post a comment