Start a new topic
Answered

Store crawled data into hosted Elasticsearch @ elastic cloud

I am new to the Scrapinghub, do you guys know if there is a plug-in for storing the crawling data on the fly into hosted Elasticsearch @ elastic cloud? FYI. I did this well with Scrapy item pipeline setting @ my desktop. Thanks.


Best Answer

The same pipeline should work in Scrapy Cloud as long as you provide all the required python dependencies in your requirements.txt. See: https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud

1 Comment

Answer

The same pipeline should work in Scrapy Cloud as long as you provide all the required python dependencies in your requirements.txt. See: https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud

Login to post a comment