Start a new topic

Slow spider


Can someone help me with my scrapy spider ? I'm beginner in python. I have wrote a scrapy spider which retrieve 100 urls from a rest api, scrape each url and extract the data, then post the items to another rest endpoint through the pipelines.

The problem is it is very slow, for 100 urls the jobs sometimes take 1 minutes but sometime 10 minutes to finish. All the urls are on diffrents domains/websites so there is no probleme gettings ban. Each websites receive only one request.

What could be the possible issue ?

Thank you.

Login to post a comment