Using Filebeat with Bonsai

Bonsai offers support for Filebeat, a lightweight shipper for forwarding & centralizing log data.
Last updated
July 7, 2023

Filebeat is a lightweight shipper for forwarding and centralizing log data. It monitors the log files or locations that you specify, collects log events, and forwards them to Elasticsearch for indexing. A common question is whether Bonsai offers support for it.

The answer is a qualified “yes.” Filebeat is a server-side tool, meaning it runs outside of Bonsai’s infrastructure and Bonsai is not involved in its configuration or management. But as a host, Bonsai is not opinionated about where your cluster’s data comes from. So if you have Filebeat running on your servers, you can configure an output to your Bonsai cluster, and it will work.

To connect Filebeat  to a Bonsai cluster you just need to add your Bonsai URL to the filebeat.yml file like this:

<div class="code-snippet-container"><a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="" loading="lazy" alt=""><img class="copied-image" src="" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">output.elasticsearch:
 hosts: [""]
 protocol: "https"
 username: "aaa" # The randomly-generated username for your cluster
 password: "xxx" # The randomly-generated password for your cluster</code></pre></div></div>

Autocreation and Bonsai

If an application sends data to an index which does not exist, Elasticsearch will create that index and assume its mappings from the data in the payload. This feature is called autocreation, and it is supported in a limited capacity on Bonsai. Certain base names can be used for autocreation. Those base names are:

  • .kibana
  • events
  • filebeat
  • kibana-int
  • logstash
  • requests

This means your Filebeat index must start with one of these index names, or it will not be automatically created.

It is important to note that Filebeat requires the OSS distribution of Elasticsearch, so for this process to work the OSS version of Filebeat needs to be used.

View code snippet
Close code snippet
By clicking “Accept”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.