Hello, recently I found that some of my crawler tasks have crawled overtime, but this is only part of it.
For example, I have https://www.amazon.it Site task, part of the success, part of the failure, prompt me "User timeout caused connection failure: Getting https://www.amazon.it/dp/B089SSJCYR?task_id=4dda6e2c-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds.." situation, what is the reason?
Example：
2021-06-11 01:43:45 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.it/dp/B08CZP1L8G?task_id=4ddab332-ca0d-11eb-b23c-00163e12f905> (failed 2 times): User timeout caused connection failure: Getting https://www.amazon.it/dp/B08CZP1L8G?task_id=4ddab332-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds..
2021-06-11 01:43:50 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.it/dp/B08GG5JVFY?task_id=4ddadee8-ca0d-11eb-b23c-00163e12f905> (failed 2 times): User timeout caused connection failure: Getting https://www.amazon.it/dp/B08GG5JVFY?task_id=4ddadee8-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds..
2021-06-11 01:43:59 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.it/dp/B08BZG4Y1L?task_id=4ddaa6b2-ca0d-11eb-b23c-00163e12f905> (failed 2 times): User timeout caused connection failure: Getting https://www.amazon.it/dp/B08BZG4Y1L?task_id=4ddaa6b2-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds..
2021-06-11 01:44:01 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.it/dp/B08F5579ML?task_id=4ddac584-ca0d-11eb-b23c-00163e12f905> (failed 2 times): User timeout caused connection failure: Getting https://www.amazon.it/dp/B08F5579ML?task_id=4ddac584-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds..
2021-06-11 01:44:03 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.it/dp/B08BNJ3D8F?task_id=4dda953c-ca0d-11eb-b23c-00163e12f905> (failed 2 times): User timeout caused connection failure: Getting https://www.amazon.it/dp/B08BNJ3D8F?task_id=4dda953c-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds..
2021-06-11 01:44:18 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.it/dp/B089SSJCYR?task_id=4dda6e2c-ca0d-11eb-b23c-00163e12f905> (failed 1 times): User timeout caused connection failure: Getting https://www.amazon.it/dp/B089SSJCYR?task_id=4dda6e2c-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds..
We look forward to your reply.
luke.chen
Hello, recently I found that some of my crawler tasks have crawled overtime, but this is only part of it.
For example, I have https://www.amazon.it Site task, part of the success, part of the failure, prompt me "User timeout caused connection failure: Getting https://www.amazon.it/dp/B089SSJCYR?task_id=4dda6e2c-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds.." situation, what is the reason?
Example：
2021-06-11 01:43:45 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.it/dp/B08CZP1L8G?task_id=4ddab332-ca0d-11eb-b23c-00163e12f905> (failed 2 times): User timeout caused connection failure: Getting https://www.amazon.it/dp/B08CZP1L8G?task_id=4ddab332-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds..
2021-06-11 01:43:50 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.it/dp/B08GG5JVFY?task_id=4ddadee8-ca0d-11eb-b23c-00163e12f905> (failed 2 times): User timeout caused connection failure: Getting https://www.amazon.it/dp/B08GG5JVFY?task_id=4ddadee8-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds..
2021-06-11 01:43:59 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.it/dp/B08BZG4Y1L?task_id=4ddaa6b2-ca0d-11eb-b23c-00163e12f905> (failed 2 times): User timeout caused connection failure: Getting https://www.amazon.it/dp/B08BZG4Y1L?task_id=4ddaa6b2-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds..
2021-06-11 01:44:01 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.it/dp/B08F5579ML?task_id=4ddac584-ca0d-11eb-b23c-00163e12f905> (failed 2 times): User timeout caused connection failure: Getting https://www.amazon.it/dp/B08F5579ML?task_id=4ddac584-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds..
2021-06-11 01:44:03 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.it/dp/B08BNJ3D8F?task_id=4dda953c-ca0d-11eb-b23c-00163e12f905> (failed 2 times): User timeout caused connection failure: Getting https://www.amazon.it/dp/B08BNJ3D8F?task_id=4dda953c-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds..
2021-06-11 01:44:18 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.it/dp/B089SSJCYR?task_id=4dda6e2c-ca0d-11eb-b23c-00163e12f905> (failed 1 times): User timeout caused connection failure: Getting https://www.amazon.it/dp/B089SSJCYR?task_id=4dda6e2c-ca0d-11eb-b23c-00163e12f905 took longer than 190 seconds..
We look forward to your reply.
1 person has this question