Hi,
I have a project with a spider using a pipeline.
Something like this:
class MySpider(scrapy.Spider): name = "myspider" custom_settings = { 'ITEM_PIPELINES': { 'project.pipelines.MyPipeline.MyPipeline': 400 } }
When running locally, spider collects items and I see the pipeline processing items (pipeline is summarizing data from the items, then making a
requests.patch
call to make an update in a separate API
Locally, everything works as expected.. but when I deployed that using "shub", spider runs in Scrapy Cloud but I don't see the pipeline being invoked.
How can I troubleshoot that?
Thanks!
nvm.. I tested again and it seems to be running, not sure what the problem was
Vuelos 5555
Hi,
I have a project with a spider using a pipeline.
Something like this:
When running locally, spider collects items and I see the pipeline processing items (pipeline is summarizing data from the items, then making a
requests.patch
call to make an update in a separate API
Locally, everything works as expected.. but when I deployed that using "shub", spider runs in Scrapy Cloud but I don't see the pipeline being invoked.
How can I troubleshoot that?
Thanks!