Spider pipeline not running in scrapy cloud

Posted about 5 years ago by Vuelos 5555

Post a topic
Un Answered
V
Vuelos 5555

 Hi,


I have a project with a spider using a pipeline.


Something like this:

class MySpider(scrapy.Spider):

name = "myspider"
custom_settings = {
'ITEM_PIPELINES': {
'project.pipelines.MyPipeline.MyPipeline': 400
}
}


When running locally, spider collects items and I see the pipeline processing items (pipeline is summarizing data from the items, then making a

requests.patch

call to make an update in a separate API


Locally, everything works as expected.. but when I deployed that using "shub", spider runs in Scrapy Cloud but I don't see the pipeline being invoked.


How can I troubleshoot that?


Thanks!

0 Votes


1 Comments

V

Vuelos 5555 posted about 5 years ago

 nvm.. I tested again and it seems to be running, not sure what the problem was

0 Votes

Login to post a comment