I'm trying to customize a Portia project to send the scraped data to another server by calling a REST API.
Basically, I want to create spiders following these steps:
1- Create the spider with Portia.
2- Publish the spider to Scrapy Cloud
3- When a scheduled job is completed, send the results in JSON format to the API.
How can I add a custom feed export to the Portia project? Does Portia overwrite all files when publishing changes?
Is it creating a custom feed export the right way to do it, or should I use a pipeline instead?
Best,
0 Votes
1 Comments
R
Richard Baxterposted
over 6 years ago
I have this problem too; it seems the documentation on setting up feed exports is a bit weak for portia / scrapy cloud users. I'd like to ftp to overwrite a file called feed.json or similar. Any help?
I'm trying to customize a Portia project to send the scraped data to another server by calling a REST API.
Basically, I want to create spiders following these steps:
1- Create the spider with Portia.
2- Publish the spider to Scrapy Cloud
3- When a scheduled job is completed, send the results in JSON format to the API.
How can I add a custom feed export to the Portia project? Does Portia overwrite all files when publishing changes?
Is it creating a custom feed export the right way to do it, or should I use a pipeline instead?
Best,
0 Votes
1 Comments
Richard Baxter posted over 6 years ago
I have this problem too; it seems the documentation on setting up feed exports is a bit weak for portia / scrapy cloud users. I'd like to ftp to overwrite a file called feed.json or similar. Any help?
0 Votes
Login to post a comment