No recent searches
Popular Articles
Sorry! nothing found for
Posted about 5 years ago by Vuelos 5555
Hi,
I have a project with a spider using a pipeline.
Something like this:
class MySpider(scrapy.Spider): name = "myspider" custom_settings = { 'ITEM_PIPELINES': { 'project.pipelines.MyPipeline.MyPipeline': 400 } }
When running locally, spider collects items and I see the pipeline processing items (pipeline is summarizing data from the items, then making a
requests.patch
call to make an update in a separate API
Locally, everything works as expected.. but when I deployed that using "shub", spider runs in Scrapy Cloud but I don't see the pipeline being invoked.
How can I troubleshoot that?
Thanks!
0 Votes
1 Comments
Vuelos 5555 posted about 5 years ago
nvm.. I tested again and it seems to be running, not sure what the problem was
Login to post a comment
People who like this
This post will be deleted permanently. Are you sure?
Hi,
I have a project with a spider using a pipeline.
Something like this:
When running locally, spider collects items and I see the pipeline processing items (pipeline is summarizing data from the items, then making a
requests.patch
call to make an update in a separate API
Locally, everything works as expected.. but when I deployed that using "shub", spider runs in Scrapy Cloud but I don't see the pipeline being invoked.
How can I troubleshoot that?
Thanks!
0 Votes
1 Comments
Vuelos 5555 posted about 5 years ago
nvm.. I tested again and it seems to be running, not sure what the problem was
0 Votes
Login to post a comment