Start a new topic

Spider pipeline not running in scrapy cloud

 Hi,


I have a project with a spider using a pipeline.


Something like this:

class MySpider(scrapy.Spider):

name = "myspider"
custom_settings = {
'ITEM_PIPELINES': {
'project.pipelines.MyPipeline.MyPipeline': 400
}
}


When running locally, spider collects items and I see the pipeline processing items (pipeline is summarizing data from the items, then making a

requests.patch

call to make an update in a separate API


Locally, everything works as expected.. but when I deployed that using "shub", spider runs in Scrapy Cloud but I don't see the pipeline being invoked.


How can I troubleshoot that?


Thanks!

1 Comment

 nvm.. I tested again and it seems to be running, not sure what the problem was

Login to post a comment