Connection pool is full, discarding connection

Posted over 4 years ago by log234

Post a topic
Un Answered
l
log234

Hello,

my crawler is downloading images it has found to an AWS S3 bucket.

However, after only 30 seconds of crawling it starts spitting out this warning:

[urllib3.connectionpool] Connection pool is full, discarding connection: <S3 bucket url>


I have configured it according to this documentation:

https://docs.scrapy.org/en/latest/topics/media-pipeline.html


Are there any settings I can change to increase the connection pool size?

Thanks for any help!

2 Votes


0 Comments

Login to post a comment