Zyte Support Center

Hi there!,

I would like to know if there is any option to dump the log entries to an Elastic search instance, or any other kind of database, so I would have a "unified log" within the spider runs.

Brgds,

Dante Henrique

Hi Dante,

Very nice question! It calls the attention of our director Shane Evans. He mention, you can download logs, there is an option in the UI and they can get retrieved via the API. The logs are available in a few formats, including JSON.

But your question is an excellent candidate to be posted as an idea to consider for further developments. I'll turn it into an IDEA in our Forums to get more visibility and got more feedback from the entire community.

Depending on the demand, our team can consider to discuss developing a script that pulls logs into elastic search, because that script doesn’t exist at present.

Our director mention is a good idea to have your question in our Forum: "Integrations are a good idea.. an easy way to get log data into systems that can be used to analyse (like ELK) is certainly something several customers would like"

So, really nice job Dante! Thanks for helping Scrapinghub to become better and better.

Best regards,