Start a new topic

Spider pipeline not running in scrapy cloud


I have a project with a spider using a pipeline.

Something like this:

class MySpider(scrapy.Spider):

name = "myspider"
custom_settings = {
'project.pipelines.MyPipeline.MyPipeline': 400

When running locally, spider collects items and I see the pipeline processing items (pipeline is summarizing data from the items, then making a


call to make an update in a separate API

Locally, everything works as expected.. but when I deployed that using "shub", spider runs in Scrapy Cloud but I don't see the pipeline being invoked.

How can I troubleshoot that?


1 Comment

 nvm.. I tested again and it seems to be running, not sure what the problem was

Login to post a comment