Spider is using requests module to crawl more then 100 k items (POST method). After different number of succesefull requests (between 3000 and 29000), the spider creashes into this error: self.chunk_left = int(line, 16) ValueError: invalid literal for int() with base 16: b'' from response.py.
priceedge
Spider is using requests module to crawl more then 100 k items (POST method). After different number of succesefull requests (between 3000 and 29000), the spider creashes into this error: self.chunk_left = int(line, 16) ValueError: invalid literal for int() with base 16: b'' from response.py.
Any sugestion could help. Thank you!