Using Logstash with Bonsai

Logstash is a data processing tool for ingesting logs into Elasticsearch. It plays a prominent role in the Elastic suite, and a common question is whether Bonsai offers support for it.

The answer is a qualified “yes.” Logstash is a server-side tool, meaning it runs outside of Bonsai's infrastructure and Bonsai is not involved in its configuration or management. But as a host, Bonsai is not opinionated about where your cluster's data comes from. So if you have Logstash running on your servers, you can configure an output to your Bonsai cluster, and it will work.

Connecting your Logstash instance to a Bonsai cluster is as easy as adding an output to the Logstash configuration file like so:

output {
  elasticsearch {
    # https://randomuser:[email protected]
    # would be entered as:
    hosts => ["something-12345.us-east-1.bonsai.io:443"]
    user => "randomuser"
    password => "randompass"
    ssl => true
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

Autocreation and Bonsai

If an application sends data to an index which does not exist, Elasticsearch will create that index and assume its mappings from the data in the payload. This feature is called autocreation, and it is supported in a limited capacity on Bonsai. Certain base names can be used for autocreation. Those base names are:

• .kibana
• events
• filebeat
• kibana-int
• logstash
• requests

This means your Logstash index must start with one of these index names, or it will not be automatically created.

We have made a number of tests and verified that we are fully compatible with Logstash, as of version 1.5+. Older versions of Logstash don't have support for SSL/TLS or HTTP Basic Auth; these older versions can work with Bonsai, but only without the benefits of encryption or authentication.

If you have any issues getting Logstash to pass data along to Bonsai, check the documentation to make sure it's set up correctly. If that doesn't help, feel free to reach out to us at [email protected] and we'll do our best to get you pointed in the right direction.