Whether you are an Elasticsearch beginner or an expert, get helpful Bonsai.io news as well as Elasticsearch tutorials, tips, and updates from our team.

Updates from the Bonsai Elasticsearch team, from One More Cloud: the first and best cloud search platform, since 2009.


Search Snippets


Schedule a free consultation to see how we can create a customized plan to meet your search needs.


Find out how we can help you.

As your operations team, we at Bonsai take uptime—and reporting it—seriously. We&#8217;ve had another great quarter, and look forward to reporting even better numbers throughout the rest of 2016.

2016 Q2 Uptime Report

<div>We’re excited to announce that we now support ES 2.2.2 in addition to 1.7.5 and 1.5. We’ve spent quite a bit of time improving our infrastructure to ensure 2.x will be a stable, blazing-fast build. Two weeks ago we shipped a button that allows anyone to upgrade their cluster to 2.2.2:
Because of backwards incompatibility with 2.x, clusters on 1.7 won’t be able to upgrade if they have data on them. If you have active shards, you will see a warning message and your button will be disabled:
So, make sure to clear your cluster! You can do this by running <code>curl -XDELETE "BONSAI_URL"</code>
Elasticsearch v5 should be ready by the end of the year, and we don&#8217;t expect any updates to 1.x after this release. In early January 2017 we&#8217;ll phase out support for 1.x on non-dedicated plans, so we recommend all of our users to start their process of moving off of 1.x and into 2.x.</div>

Now Supporting 2.x!

<div>As your operations team, we at Bonsai take uptime (and reporting it) seriously. We&#8217;ve had a fantastic first quarter and look forward to reporting even better numbers in the coming quarters.
If you&#8217;re interested, you can read more about how we measure uptime here: <a href="/2014/10/02/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as a Service Provider</a>. We&#8217;ve added even more high-precision monitoring tools to our stack since that post was written, which we plan to write about in upcoming posts.</div>

2016 Q1 Uptime Report

<div>###Access
You can find the logs for your cluster under the Logs tab in your cluster dashboard. Once you click on the tab, logs will populate in real time as requests are made to your cluster.
Each row in the table is one request made to your cluster. You will see the timestamp of the request, the type of request [GET, POST, PUT, UPDATE, DELETE], the URL of the request, the time in milliseconds it took for the request to complete, and the status.
####Color Indicators
The time in milliseconds and status have color indicators that let you know how long it took for the request to complete, and if it was successful.

The top 20 slow requests are shown on a separate tab within the Logs menu, and shows requests that take longer than 200ms. Any yellow or red requests will appear on this tab.
This feature is still in beta, so we would appreciate any feedback you have. Hit us up at <a href="mailto:support@bonsai.io">support@bonsai.io</a> or you can ping us on <a href="https://twitter.com/bonsaisearch" target="_blank" rel="nofollow">Twitter @bonsaisearch</a>.</div>

Introducing: Live Streaming Logs

<div>Whether you&#8217;re a small team of two or part of a large organization, there&#8217;s a good chance you will need to delegate responsibilities of managing your cluster at some point. Even if you just want to include someone else on email communication, or if you want to stop our emails going out to everyone on the group email you signed up with (gasp!), we can help.
##Heroku Contacts
For our Heroku add-on users, we built Heroku Contacts. Being a contact means that you will receive notifications from us about your cluster usage, and overage warnings if you&#8217;re going over your plan limits. You can manage your contacts in the Bonsai Dashboard, which you can access through Heroku, or by typing this in your terminal:
<pre><code>heroku addons:open bonsai -a &lt;your-app-name-here&gt;
</code></pre>
###Contacts page in the Bonsai Dashboard:
Your Heroku app owner email is automatically set up as a contact, as well as the email address of anyone who SSO&#8217;s into the dashboard. If your Heroku owner email is a group email, you will likely want to remove it as a contact and replace it with another email address so we don&#8217;t spam your whole company!
##Cluster Collaborators
For our direct sign-up users, we built Cluster Collaborators. A collaborator is able to login to the Bonsai Dashboard with their own user account, which means no more sharing passwords! To add a collaborator, click on the Collaborators tab from your Bonsai Dashboard.
###Collaborators page in the Bonsai Dashboard:
You have the option of adding either a regular collaborator or an administrator. A collaborator will receive notifications from us about the cluster, be able to view API credentials, and view other collaborators. An administrator has the additional ability to regenerate credentials, add and remove other collaborators, update the cluster plan, and deprovision the cluster.
Once you invite a collaborator, they will be emailed a link to sign up as a user for Bonsai, where they can choose to accept or decline your invitation.
After clicking &#8216;Accept&#8217;, they will be able to see the cluster on their cluster dashboard. A cluster that you are a collaborator on is shown with a collaborator symbol in the upper right hand corner.
##Go Forth, and Collaborate!
We loved building these tools, and we hope you enjoy using them! If you have any suggestions for improvements or you have a business need for something else, hit us up at support@bonsai.io or you can ping us on <a href="https://twitter.com/bonsaisearch" target="_blank" rel="nofollow">Twitter @bonsaisearch</a>.</div>

Teamwork Makes the Dream Work

<div>Sometimes we’re asked if we support Logstash. The relationships between Elasticsearch, Logstash and Kibana (often referred to as the ELK stack) can sometimes foster a confusing mental model, and a simple &#8220;yes&#8221; from our support staff probably isn’t enough to help users get up and running quickly. In this post, we’d like to unveil some of the mystery behind why the ELK stack is so popular.
We whipped up a quick example showing how to set up and use Logstash with Bonsai Elasticsearch and a Kibana 4 instance. This example takes a real world use case and walks the reader through the process of setting up Logstash to pass a particular type of event to Bonsai for indexing, where it can then be analyzed in Kibana. The intent is to help new users get up to speed quickly, without needing to do a lot of Googling for answers.
###So what is Logstash?
Logstash is an awesome tool for log parsing. Logstash can accept inputs from a number of different sources, apply a series of filters to the incoming data, then pass the results to a data store. In combination with Elasticsearch (for data storage, aggregation and retrieval) and Kibana (for analysis and visualization), users can get a very nice high-level view of a data set. It can do many other things, but that is its primary function.
##A real world example
In this example, we&#8217;ll look at the SSH daemon running on an Ubuntu server. SSH logs, particularly failed login attempts, can be extremely useful in investigating break-in attempts, identifying unusual traffic patterns, and performing regular security audits. It&#8217;s part of a proactive security policy, and critical if you&#8217;re a sysadmin in charge of dozens or hundreds of servers.
We could use other input contexts, but since the ‘net is rife with automated attack traffic, we think security is particularly interesting. If you have a machine connected to the Internet, you will eventually be hit by someone. Hackers use bots to scan the IPv4 space looking for particular ports &#8212; 21 for FTP, 22 for SSH, 23 for Telnet, and so on — and when a server responds to a request on a given port, the bot will try everything it can to log in to the system.
Generally, the bot doesn’t know anything about the target, so it just tries a bunch of random usernames and passwords. By studying this behavior, we can perform some counterintelligence gathering on the hacker community and use the ELK stack to aid in our reconnaissance. For another fun read, see how the developer of Elastichoney uses a <a href="http://jordan-wright.com/blog/2015/05/11/60-days-of-watching-hackers-attack-elasticsearch/" target="_blank" rel="nofollow">similar approach</a> to learn how hackers try to exploit Groovy scripting vulnerabilities in Elasticsearch.
###The Game Plan
We&#8217;ll pass the system logs into Logstash, run some filters to make sure we&#8217;re only seeing what we want, send the results along to Bonsai, and finally, we&#8217;ll examine the data using Kibana.
For the purposes of this example, we&#8217;ll make a few assumptions:
<ol>
<li>
SSH activity is being logged to <code>/var/log/auth.log</code>. This is the default setting for Amazon&#8217;s Ubuntu images, which have not (yet) switched over to systemd. Although you can use systemd with Logstash if you like.
</li>
<li>
You have the ability to install Logstash on the server(s), and have already done so. There are ways to do it with a package manager, and Elastic offers Logstash as a .deb on their downloads page, so it can be pretty easily installed. Read their documentation if you need more help getting it installed wherever you&#8217;re trying to use it.
In preparing this example, I downloaded the .deb to an Ubuntu machine, so the instructions are based on that. If you use a package manager or another OS, some files might end up in other places and some commands may need minor tweaks.
</li>
<li>
You already have a Bonsai Elasticsearch cluster provisioned. In this example, the URL of that cluster will be <code>http://user123:pass456@logstash-12345.us-east-1.bonsai.io</code>
</li>
<li>
You have a Kibana 4 instance set up somewhere. Need more info on this? <a href="http://docs.bonsai.io/docs/using-kibana-with-bonsai" target="_blank" rel="nofollow">We&#8217;ve got you covered</a>.
</li>
</ol>
###Laying the groundwork
Logstash can accept streams from a large number of inputs. They also have an ever-expanding list of input plugins to support more streams and file types. One requirement that all inputs have in common is permissions; that is, Logstash must have permission to access the stream in order to read from it. This probably sounds trivial, but it is a pretty common &#8220;gotcha,&#8221; especially when dealing with daemon log files.
When Logstash is installed in Debian, the OS creates a user called <code>logstash</code>. This is the user that will need to be able to access the log file. We can see the permissions of the auth.log file with <code>ls -lAh /var/log/auth.log</code>:
<pre><code>-rw-r----- 1 syslog adm 6.4M Jan 20 17:24 /var/log/auth.log
</code></pre>
We don&#8217;t want to change these permissions, because that&#8217;s not a best practice and could be dangerous. But what we can do is add the <code>logstash</code> user to the <code>adm</code> group. The <code>adm</code> group typically grants members access to read log files and nothing else. Joining this group would give the Logstash process the ability to read the file, but not modify it in any way. This is the minimum necessary access to do the job, and it doesn&#8217;t expose security details to other programs/users on the server, so it&#8217;s the route we&#8217;ll take:
<pre><code>$ sudo usermod -G adm logstash
</code></pre>
The last thing we need to do is create an index on our Bonsai cluster for Logstash to send data. We&#8217;ll call this index <code>logstash</code>:
<pre><code>$ curl -XPUT http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash
</code></pre>
###Configuring Logstash
Now that we know Logstash will be able to read the input stream, we&#8217;ll need to configure it. Go ahead and edit <code>/etc/logstash/conf.d/logstash.conf</code>:
<pre><code>$ sudo nano /etc/logstash/conf.d/logstash.conf
</code></pre>
Copy/paste the settings below (making sure to substitute the correct values for your cluster URL):
<pre><code>input {
 file {
 path =&gt; "/var/log/auth.log"
 start_position =&gt; beginning
 sincedb_path =&gt; "/dev/null"
 }
}

output {
 elasticsearch {
 hosts =&gt; ["logstash-12345.us-east-1.bonsai.io:443"]
 user =&gt; "user123"
 password =&gt; "pass456"
 ssl =&gt; true
 index =&gt; "logstash"
 }
}
</code></pre>
Finally, make sure that Logstash has permission to read its own config file:
<pre><code>$ sudo chown logstash /etc/logstash/conf.d/logstash.conf
</code></pre>
###Test it out!
We&#8217;ve just completed the bare minimum necessary to get Logstash up and running. We&#8217;ve given Logstash access to a log file, defined it as an input stream, created a place in Elasticsearch for the stream to go, and defined that index as an output. Now all we need to do is run it and make sure it works. When I installed the .deb for Logstash, the one thing it didn&#8217;t do was symlink the binary, so I&#8217;ll just use the full path. YMMV:
<pre><code>$ /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/logstash.conf
Logstash startup completed
</code></pre>
Now, if everything worked as expected, I should be able to see the data in my cluster:
<pre><code>$ curl "http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash/_search?pretty&amp;size=1"
{
 "took" : 1,
 "timed_out" : false,
 "_shards" : {
 "total" : 1,
 "successful" : 1,
 "failed" : 0
 },
 "hits" : {
 "total" : 1975,
 "max_score" : 1.0,
 "hits" : [ {
 "_index" : "logstash",
 "_type" : "logs",
 "_id" : "AVJdzwbLfbt0kfwq_Tzs",
 "_score" : 1.0,
 "_source":{"message":"Jan 17 04:30:52 ip-XXX-XXX-XXX-XXX sshd[12170]: Invalid user cms from XXX.XXX.XXX.XXX","@version":"1","@timestamp":"2016-01-17T11:30:52.000Z","host":"localhost","path":"/var/log/auth.log"}
 } ]
 }
}
</code></pre>
Neat! Let&#8217;s look at this in Kibana:
So, a partial success. We can see data in the index, and it is coming out of the auth.log file as expected. But the timestamp associated to the event is not when it occurred in real time, but when it was indexed. Not very helpful.
This is a pretty common gotcha as well. I included it here to underscore it as a pitfall (and maybe save everyone a few support tickets). It&#8217;s easily addressed, we just need to do a bit more work.
##A more complex setup
Let&#8217;s go ahead and delete the logstash index on our cluster:
<pre><code>$ curl -XDELETE http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash
</code></pre>
With that out of the way, we can focus on refining our Logstash configuration. We&#8217;ll use some of the more popular filters to extract the most important details, and add those to fields that we can later query and display with Kibana.
Before we do that, let&#8217;s think about what our goal is here. We want to look for failed login attempts over SSH; we&#8217;re not interested in <code>cron</code> tasks or <code>sudo</code> sessions, or anything else that commonly gets logged to auth.log. Let&#8217;s take a look at a sample line:
<pre><code>Jan 17 04:30:52 ip-XXX-XXX-XXX-XXX sshd[12170]: Invalid user cms from XXX.XXX.XXX.XXX
</code></pre>
What kind of data do we have there? I see a date (in a non-standard format, boo!), a private IP address (useful for identifying what server is being hit), as well as a probe at a user name (&#8220;cms,&#8221; really?) and an IP address from where the attempt originated.
So we want to disregard lines that don&#8217;t match this pattern, and extract some useful information from lines that do match the pattern. We&#8217;ll also have to do something about that date &#8212; it will need to be in a standard format to be useful in Kibana. It would also be cool if we could perform some further analysis on the originating IP address, to see if there are any interesting trends. Who&#8217;s hitting us, and where are they coming from, would be good to know.
Fortunately, Logstash has filters that allow us to do all of this:
<ul>
<li>
Grok. The <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html" target="_blank" rel="nofollow">grok filter</a> allows you to use regular expressions to parse log entries. It has a number of built-in patterns, and allows you to arbitrarily define your own. This is how we&#8217;ll extract the lines of interest.
</li>
<li>
Date. The <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html" target="_blank" rel="nofollow">date filter</a> allows you to extract timestamps from your logs, then using those as the Logstash timestamp for the event. This will fix that &#8220;single bar&#8221; problem in Kibana.
</li>
<li>
GeoIP. The <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html" target="_blank" rel="nofollow">geoip filter</a> allows you to take an IP address, run it through the GeoLiteCity database, and save geographic information to a new field. We can use this info in Kibana to make a heat map of where the nefarious traffic is originating.
</li>
</ul>
Update the Logstash configuration file to this, again changing the Elasticsearch output to match your Bonsai URL :
<pre><code>input {
 file {
 path =&gt; "/var/log/auth.log"
 start_position =&gt; beginning
 }
}

filter {
 grok {
 match =&gt; { "message" =&gt; "(?&lt;logdate&gt;%{MONTH} %{HOUR}.%{MINUTE}.%{SECOND}) (?&lt;private_ip&gt;ip-[0-9]{1,3}-[0-9]{1,3}-[0-9]{1,3}-[0-9]{1,3}) sshd\[[0-9]{1,}\]: (?&lt;sshd_message&gt;Invalid user (?&lt;invalid_user&gt;[a-z0-9]{1,}) from (?&lt;invalid_ip&gt;%{IP}))"}
 }
 if "_grokparsefailure" in [tags] {
 drop { }
 }
 date {
 match =&gt; [ "logdate", "MMM dd HH:mm:ss" ]
 }
 geoip {
 source =&gt; "invalid_ip"
 target =&gt; "geoip"
 database =&gt; "/etc/logstash/conf.d/GeoLiteCity.dat"
 }
}

output {
 elasticsearch {
 hosts =&gt; ["logstash-12345.us-east-1.bonsai.io:443"]
 user =&gt; "user123"
 password =&gt; "pass456"
 ssl =&gt; true
 index =&gt; "logstash"
 }
}
</code></pre>
We&#8217;ll need to make sure that the GeoLiteCity.dat file is in its proper place:
<pre><code>$ cd /etc/logstash/conf.d/
$ sudo curl -O "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz"
$ sudo gunzip GeoLiteCity.dat.gz
$ sudo chown logstash GeoLiteCity.dat
</code></pre>
Finally, we need to make one little tweak to our index before running Logstash. As of this writing, there is a bug where Elasticsearch will dynamically map the <code>geoip.location</code> as a <code>float</code> instead of a <code>geo_point</code>, which makes it impossible for Kibana to create a tile map with the information. We just need to explicitly set this field up first. We&#8217;ll do that when we recreate the index:
<pre><code>$ curl -XPUT http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash -d '{"mappings":{"logs":{"properties":{"geoip":{"properties":{"location":{"type":"geo_point"}}}}}}}'
</code></pre>
Let&#8217;s spin up Logstash again and see what happens:
<pre><code>$ logstash agent -f /etc/logstash.conf
Settings: Default filter workers: 2
Logstash startup completed
</code></pre>
If we check on the data again, we see:
<pre><code>curl "http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash/_search?pretty&amp;size=1"
{
 "took" : 1,
 "timed_out" : false,
 "_shards" : {
 "total" : 1,
 "successful" : 1,
 "failed" : 0
 },
 "hits" : {
 "total" : 144,
 "max_score" : 1.0,
 "hits" : [ {
 "_index" : "logstash",
 "_type" : "logs",
 "_id" : "AVJggJNEfbt0kfwq_U0P",
 "_score" : 1.0,
 "_source":{"message":"Jan 17 04:30:52 ip-XXX-XXX-XXX-XXX sshd[12170]: Invalid user cms from XXX.XXX.XXX.XXX","@version":"1","@timestamp":"2016-01-17T11:30:52.000Z","host":"localhost","path":"/var/log/auth.log","logdate":"Jan 17 04:30:52","private_ip":"ip-XXX-XXX-XXX-XXX","sshd_message":"Invalid user cms from XXX.XXX.XXX.XXX","invalid_user":"cms","invalid_ip":"XXX.XXX.XXX.XXX","geoip":{"ip":"XXX.XXX.XXX.XXX","country_code2":"CN","country_code3":"CHN","country_name":"China","continent_code":"AS","region_name":"12","city_name":"Wuhan","latitude":30.580099999999987,"longitude":114.27339999999998,"timezone":"Asia/Shanghai","real_region_name":"Hubei","location":[114.27339999999998,30.580099999999987]}}
 } ]
 }
}
</code></pre>
Nice! So our filters worked. We&#8217;ve dropped from almost 2K lines to just 144. That&#8217;s a big reduction in noise. Let&#8217;s look in Kibana:
Great, so the timestamps are correct as well. We can see a big spike in activity at one point. From a security standpoint, that would be something to investigate. Ideally, it would also be something we could wire up to an alarm and page a sysadmin to let them know there might be an intrusion attempt.
But let&#8217;s take it further. We can create a whole dashboard around this. We can create a Tile Map using the <code>geoip.location</code> field and see where these requests are originating. We can also look at stats like the user account that was probed; this might help us to determine if these attacks are some random bot or script kiddie, or if there is a more sophisticated adversary targeting specific systems.
I&#8217;m sure you could think of even more ways to use this data. For example, imagine that the source data is coming from multiple servers in your network. You could see which ones are being hit hardest and take corrective action. You could set up an alert that notifies you if someone logs in from an IP address overseas &#8212; perhaps they&#8217;ve been compromised. There is a wealth of intelligence that can be gleaned from looking at a setup like this.
###Wrapping up
People use Logstash for all sorts of things. It really is an amazing tool, and we wish it had a larger following. If you&#8217;re using Logstash and want to host your Elasticsearch cluster with Bonsai, we&#8217;d love to work with you. We fully support both Logstash and Kibana, so&#8230;
##Give us a shout!
Got a cool use case you&#8217;d like to share? Maybe an anecdote or useful tip? Did you find this article tedious and boring? We&#8217;d like to hear about that too. Hit us up at support@bonsai.io and share. You can also ping us at <a href="https://twitter.com/bonsaisearch" target="_blank" rel="nofollow">@bonsaisearch</a> on Twitter if that&#8217;s something you&#8217;re in to.</div>

Logstash and Bonsai and bots, oh my!

<div>A t the core of Elasticsearch’s popularity is the ease and simplicity of setting up a cluster. And with hosted search options like <a href="https://bonsai.io" target="_blank" rel="nofollow">Bonsai.io</a>, powerful, multi-node clusters can be created instantly.
In a lot of ways, ease of use is both a blessing and a curse. When the learning curve isn’t a barrier to entry, it’s easy to start on a path that causes problems later. Many users find that decisions they made early in the product cycle regarding Elasticsearch don’t scale well in production. Some early decisions can become extremely painful or difficult to change later.
At Bonsai, we’ve had the opportunity to work directly with thousands of users around the world. We’ve had the unique experience of getting a peek into projects integrating Elasticsearch at scale, and helping people get the absolute most out of search. This experience has been instrumental in crafting some guidelines for users just starting out.
This is the guide a lot of users wished they’d had in the beginning.
<h3>Why Focus on the Index? Why Not the Cluster?</h3>
The short answer is that you should let your data decide the cluster specifications, not vice-versa. The index will determine how the data is stored and retrieved. If you haven’t thought through what your indices will look like and how they will be configured, you may end up underprovisioning the cluster (hurting performance) or overprovisioning the cluster (hurting your wallet).
This guide will help you understand and reason about the factors that will impact the quality of your search experience over the long term.
<h3>Elasticsearch Has So Many Use Cases, How Relevant is this Advice to Me?</h3>
People use Elasticsearch in so many ways. And each use case is going to benefit from a different set of settings and low level tweaks. However, these are typically changes that can be made at any time, in contrast to immutable settings like primary sharding. In other words, the fine tuning can happen a little later.
Instead, we’re going to focus on a couple of high level characteristics shared by all well-configured indices, and leave the more esoteric stuff for a later series.
<h2>That said, the Ideal Elasticsearch Index is…</h2>
<ol>
<li>Backed by a database</li>
<li>Sharded intelligently</li>
<li>Replicated, but not overreplicated</li>
<li>Has well thought out data structures</li>
</ol>
Let’s look into those in a bit more detail.
<h3>Backing Elasticsearch with a Database</h3>
Most developers have at least a casual understanding of what a database is: some software that stores data in a structured format, and allows applications to interact with that data using a structured query language. And Elasticsearch is basically the same thing, right?
While that is an accurate description of what Elasticsearch does, the devil is in the details. Database software aims for something called ACID compliance. ACID (Atomicity, Consistency, Isolation, Durability) is a set of properties that guarantee validity of data in the event of unexpected service interruptions.
An “ACID-compliant data store” is a fancy way of saying that the technology is highly resistant to corruption and data loss. Some examples of ACID-compliant data stores you may be familiar with: Postgres, SQLite, Oracle, MySQL (with InnoDB), and MongoDB (4.0+). Not listed: Elasticsearch.
Elasticsearch uses a search engine technology called <a href="https://lucene.apache.org/" target="_blank" rel="nofollow">Lucene</a>. Lucene is an information retrieval technology built for speed, not redundancy. It has a radically different architecture that gives it blazing fast performance, at the expense of being more susceptible to data loss.
Data loss can happen in a number of ways, you need to be able to recreate the data if needed. True, Elasticsearch has a snapshot/restore feature, but this process will only ever partially recover you in the event of data loss. Updates made between the most recent snapshot and outage will be lost unless you have another system in place to queue them. Snapshot/restore will also not help in the event of split brain, because there’s no mechanism for reconciling updates to each partition. Updates will just be lost.
This is why every Elasticsearch index you create should be backed by an ACID-compliant data store. The database should be the ultimate source of truth, from which the index is populated. If you ever lose the index — via node loss, outage, or simple fat-finger — you’ll be able to completely recover it.
<h3>Intelligent Sharding</h3>
Are you brand new to Elasticsearch? Start by reading <a href="https://docs.bonsai.io/docs/what-are-shards-and-replicas" target="_blank" rel="nofollow">this article on Elasticsearch shards and replicas</a>.
Every Elasticsearch index is composed of one or more shards. There are two types of shards: primary, and replica. Primary shards are responsible for managing updates to the index, while replica shards are simply copies on the primary and live on other nodes. The idea is that if a primary shard is taken offline, the replica will be able to fill the role and keep search from going down.
Shards are really just abstractions for Lucene indices. When an Elasticsearch index has several primary shards, it can be thought of having the data spread out over several different search engines. This is useful, for example, when you have millions of documents, because it allows queries to be served in parallel, which can save time and IO.
“Oh,” I hear you say, “well then I’ll just create my index with 100 primary shards and be done with it!” That certainly is one approach, but it’s a terrible one. Shards come with overhead; even empty shards require memory and disk. And <a href="https://docs.bonsai.io/docs/what-are-shards-and-replicas" target="_blank" rel="nofollow">tf-idf</a> relevance is calculated on a per-shard basis. Oh, and if you have more shards than CPU cores, they will end up fighting each other for system resources on every request.
In other words, this approach would give you the worst of all worlds unless you already have a lot of data and a lot of nodes.
Another thing to understand is that Elasticsearch uses a naive hashing algorithm to route documents to a given primary shard. This design choice allows documents to be randomly distributed in a reproducible way. This avoids “hot spots” that affect performance and overallocation. However, it has one major downside, which is that the number of primary shards can not be changed after an index has been created. Replicas can be added and removed at will, but the number of primary shards is basically written in stone.
This means that we need to have some sense of our scale before even creating the index. We need a shard scheme that makes sense for our present needs, but with enough headroom to scale up.
<h3>Benchmarking is key</h3>
Benchmarking a subset of your data before you decide on your production settings will give you a sense of how much data will be used per document, both on-disk and in memory. From there, you can use some heuristics to figure out a good launch point.
Something else to think about is the number of nodes you will be using, and what are their capabilities. A production cluster should not have fewer than three nodes, and should almost always have an odd number. With that in mind, here are some heuristics:
If you don’t expect data to grow significantly, then:
<ul>
<li>One primary shard is fine if you have less than 100K documents</li>
<li>One primary shard per node is good if you have over 100K documents</li>
<li>One primary shard per CPU core is good if you have a couple million documents</li>
</ul>
If you do anticipate significant growth, then you’ll want to balance your present needs with your expected needs. At a minimum, you’ll want a number of primary cores equal to the number of nodes you would scale to when horizontal scaling is required.
In other words, if you’re starting out on a three node cluster, create the index with at least five primary shards. If you have a large enough collection of documents, you could also go with the product of your current and future number of nodes (in this case, 3 nodes today x 5 nodes at scale-up=15 primary shards). That assumes that the nodes in your three node cluster have at least five CPU cores.
See why this is an interesting problem?
Thinking about your needs and planning a little ahead will go a long way in creating an Elasticsearch index that performs well over the long term.
<h3>Replicating shards, but not Overreplicating them</h3>
Replica shards are copies of primary shards that live on any node other than the primary. Like primary shards, replicas can serve search requests if needed. Unlike primary shards, replicas do not handle update operations; they simply receive updates from the primary at regular intervals. A nice benefit of this design is that replicas are also not subject to immutability like primary shards. You can add/remove replicas at any time.
Replicas reduce stress on primary shards, and provide protection against data loss, node loss, network partitions, etc. However, too many replicas lead to wasted resources, because shards aren’t free. The cost-benefit ratio of replication gets worse with each new replica shard.
The ideal Elasticsearch index has a replication factor of at least 1. This means for every primary shard — however many there may be — there is at least one replica. And the maximum number of replicas never exceeds (n-1), where n is the number of nodes in the cluster. (This is a common newbie mistake: creating 10 replicas on a three node cluster and then wondering why the cluster state is perpetually “yellow.” Where are all of those extra replicas supposed to go?)
How do you know how many replicas is “enough”? With a minimum of 1 and maximum of (n-1), what is the “sweet spot?” Incremental benchmarking is a good way to narrow in on that. Add replicas incrementally and measure performance improvements with each new replica. If improvements are not noticeable, replication is probably too high.
<h3>Thinking through data structures</h3>
What are “data structures,” and how are they pertinent to Elasticsearch? Elasticsearch has a concept known as <a href="https://www.tutorialspoint.com/elasticsearch/elasticsearch_mapping.htm" target="_blank" rel="nofollow">“mappings.”</a> Mappings to specify how data will be ingested and queried. Mappings define analyzer chains composed of tokenizers and filters; these allow data (and queries) to be tokenized and mutated before being used.
While a well considered mapping can improve your search results, it’s wise to start simply, benchmark, and iterate later. Tweaking mappings and then promoting them to production is easier than you think. That means you don’t have don’t index everything in your database. Don’t try to do All The Things™. Think about what features your application actually needs right now, and worry about new features later.
It’s also carefully considered: don’t use mappings that you don’t fully understand. Don’t just copy/paste from the Internet. Read the documentation, know what each component does and how it will affect performance and system resources. If you don’t know what your mappings look like, take an hour to look at them and comprehend how they will work with your data.
The Ideal Elasticsearch Index isn’t necessarily just implementing default data structures, but has mappings that were honed in small scale testing. A subset of production data can be used to benchmark the performance and resource demands of a mapping. Figure these things out before taking it to scale.
<h4>Understanding how a mapping can affect performance</h4>
If you’re familiar with the <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/pipes-and-filters" target="_blank" rel="nofollow">Pipe and Filter Pattern</a>, then this will make a lot of sense. But just in case, let’s look at the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-standard-tokenizer.html" target="_blank" rel="nofollow">standard tokenizer</a> for a moment. This component takes a text input and breaks it up using the Unicode Text Segmentation algorithm. The short explanation is that this splits the input up on whitespace. So an input like:
<pre><code>It was the best of times, it was the BLURST of times
</code></pre>
is broken up into an array like so:
<pre><code>[It, was, the, best, of, times, it, was, the, BLURST, of, times]
</code></pre>
The elements of this array are called tokens, and they can be filters out and mutated before reaching Lucene. We can use stopwords filters to get rid of tokens that aren’t likely to be relevant to a search:
<pre><code>[best, times, BLURST, times]
</code></pre>
Then we can pass these tokens through a lowercase filter to get rid of the all-caps, and then through a synonyms filter, which maps <code>blurst =&gt; worst</code>. That will leave us with:
<pre><code>[best, times, worst, times]
</code></pre>
These tokens are now stored in the inverted index along with the corresponding document ID, ensuring queries will be much more relevant. This pattern gives developers a lot of flexibility, however some seemingly innocuous choices will have enormous consequences at scale.
Why? Well, mappings are easy to understand conceptually, but can take time to master in practice. Here’s a classic example: <a href="https://en.wikipedia.org/wiki/N-gram" target="_blank" rel="nofollow">n-grams</a>. This is a filter which breaks up tokens into lots of sub-tokens, with set minimum and maximum sizes. Developers often like to use n-grams for substring matching at search time, and subsequently give the tokens a size from 1 to 100 (or something obnoxiously large).
To see why this is a problem, look at what a token like <code>“terrific”</code> turns into:
<pre><code>["t", "e", "r", "r", "i", "f", "i", "c", "te", "er", "rr", "ri", "if", "fi", "ic", "ter", "err", "rri", "rif", "ifi", "fic", "terr", "erri", "rrif", "rifi", "ific", "terri", "errif", "rrifi", "rific", "terrif", "errifi", "rrific", "terrifi", "errific", "terrific"]
</code></pre>
If you work it out, you find that there is a parabolic relationship between the token length and the number of n-grams that it generates:
Worse, this pattern is generated per token. A standard 500-token document will create 500 times more grams, much of them garbage. How many words can you think of with substrings that are also, unrelated word? “Scatter” contains the totally unrelated word “cat.” If a user searches for “cat,” a document with the term “scatter” could plausibly be a match in this scenario.
And if your gram size is 1, then you’re basically indexing every single character in the alphabet to every single document. Pro-tip: this is not a good way to get relevant results or use system resources intelligently. But it’s a great way to consume lots of disk and memory for no reason.
How about another example? Synonym matching is another feature that is easy to implement, and even easier to screw up. Synonyms allow developers to create rules that group or translate related tokens. You might have a rule that replaces “ram” with “memory,” or a rule that takes the token “cat” and expand it out to the tokens: “cat”, “kitty” and “kitten.”
Elasticsearch allows developers to perform synonym expansion at index time, query time, or both. Expanding at index time means that the tokens are being expanded before being written out to disk. This does lead to larger indices, and has the downside of needing to reindex every time you want to change them.
That said, many developers opt instead for query time expansion. The user searches for a term that the synonym filter matches and expands, and those new tokens become part of the Lucene query. But then in production, the users report lots of unexpected and irrelevant results.
Why? Because multi-word synonyms break phrase queries, and the tf-idf scores for rare terms lead to score boosting in ways that are not immediately obvious. The low level details of why this happens is well beyond the scope of this post, but needless to say, you could spend weeks wrangling with relevancy tuning because of an ill-considered choice when working out the mappings.
These are just a few examples; broader coverage of the kinds of subtle mistakes and inefficiencies that are introduced in early stage data structures would be a whole other series of articles. But it’s just a taste of what can happen when choices are made without understanding the consequences and trade offs.
<h2>Conclusion</h2>
Elasticsearch and search engines are incredibly broad topics, and mastery of these subjects can be incredibly difficult. Fortunately, the fundamentals are easy to grasp and implement. The Ideal Elasticsearch Index is a set of design principles common to most use cases.</div>

The Ideal Elasticsearch Index

<div>W
e’ve had Elasticsearch 1.7 out and running in production for a few months now, and, as expected, it’s operating like a champ. This is a minor update that doesn’t change user functionality, so we don’t expect any impact. Next week we’ll begin the process of upgrading all remaining 1.5 clusters onto ES 1.7, from Tuesday through Thursday. We will be posting updates on <a href="https://twitter.com/bonsaisearch" target="_blank" rel="nofollow">twitter</a>.
###Gain Improved Versions of Features Available Through Elasticsearch 1.6
Most user-facing changes to Elasticsearch in 1.6 and 1.7 have been minor or subtle additions to existing functionality, so we’re not expecting any notable changes for clusters being upgraded.
One such example from our friends over at <a href="http://www.wework.com" target="_blank" rel="nofollow">WeWork</a>: the <code>field_value_factor</code> function now lets you specify a default value for a field that is missing on a document. A handy improvement if you’re using <code>function_score_query</code> with inconsistent document values!
<ul>
<li><a href="https://github.com/elastic/elasticsearch/pull/10845" target="_blank" rel="nofollow">elastic/elasticsearch#10845</a></li>
</ul>
Other changes to ES 1.6 and 1.7 have been focused on security and operational improvements. While these kinds of improvements are always nice to have, they weren’t critical for us. The security fixes, in particular, do not affect our systems due to our architectural design choices and other preventative measures.
So if ES 1.5 is working out fine, why risk an upgrade? Good question. These newer versions definitely are better, even if subtly. We think it’s a core part of our value as a service provider to provide continuous managed upgrades to our customers, so that you can enjoy the full benefits of open source community built software.
Imagine, if the cost:benefit ratio had us putting this upgrade on the backburner, how much longer might a busy team focused on their app wait to manage their own upgrade!
###A Smoother Transition for Future Releases: 2.0
More importantly here, Elasticsearch 2.0 is around the corner, and that upgrade will not be so trivial. We’re laying the groundwork now to help prepare for that upgrade some time in the new year, and ES 1.6 and 1.7 also help to provide interim tooling toward that end. You will receive at least a month of notification for this update so you can test your app against this particular upgrade.
###How to opt-out of 1.7
If, out of an abundance of caution, you’d like to opt-out of the upgrade, that’s fine too! We can provide long-term version support for any Elasticsearch version on one of our dedicated cluster plans. Just <a href="mailto:support@bonsai.io">contact support</a> and let us know.
<h2>Recap:</h2>
<ul>
<li>We’re upgrading everyone from 1.5 to 1.7</li>
<li>This will happen next week, Dec 22nd — Dec 24th</li>
<li>1.7 improves security and operations, so we don’t expect any user-facing impact</li>
<li>Upgrades to 2.0 are coming in the new year</li>
</ul></div>

Scheduled Maintenance Upgrade to Elasticsearch 1.7

<div>We sometimes hear from users concerned about their shard counts. They ask how they can instruct Elasticsearch to add multiple indices to a single shard. This is a surprisingly difficult question to answer in full because the terminology can confuse the issue quite a bit.
In Elasticsearch, every index maps to one or more shards, but the reverse is not true for some pretty sound technical reasons. However, changes to data mapping do afford users the ability to logically separate data on a single shard, thus producing the effect of multiple indices per shard for many use cases.
If that explanation sounds confusing, don&#8217;t worry. In this post, we&#8217;re going to lay out the issue and solution as easily as possible.
<h2>The problem</h2>
<a href="https://en.wikipedia.org/wiki/Lucene" target="_blank" rel="nofollow">Lucene</a> is the Java-based information retrieval software that forms the backbone of Elasticsearch. Elasticsearch adds a few layers of abstraction on top of Lucene to provide features that are extremely useful to production applications: aggregations, analysis, distributed search, and more, all controlled through a well-documented and easy to use RESTful API. However, with these features comes a richer glossary of terms, with slightly more subtle definitions.
The biggest point of confusion for users is the word &#8220;index.&#8221; This word means different things, depending on the context. It can be used:
<ul>
<li>As a noun in the context of Lucene</li>
<li>As a noun in the context of Elasticsearch</li>
<li>As a verb in the context of either Lucene or Elasticsearch</li>
</ul>
Catch that? A Lucene index and an Elasticsearch index are very different things, even though Elasticsearch implements Lucene. In the context of Lucene, an index is basically a Lucene instance. In the context of Elasticsearch, an index is&#8230; well, read on.
I mentioned earlier that Elasticsearch offers a rich glossary, with terms like &#8220;shard,&#8221; &#8220;index,&#8221; and &#8220;type,&#8221; which are closely related but distinct concepts. So what distinguishes a shard from an index from a type in Elasticsearch? In the Elasticsearch lexicon:
<ul>
<li>A shard is a single Lucene instance</li>
<li>An index is a logical namespace that points to one or more shards</li>
<li>A type is a piece of metadata used to logically distinguish between documents of different types within an index</li>
<li>A document in Elasticsearch is essentially a Lucene document with some injected metadata</li>
</ul>
To distill this all down:
<blockquote>
An Elasticsearch shard is a Lucene index. An Elasticsearch index is a collection of one or more Lucene indices. An Elasticsearch type is a bit of metadata injected into the document which allows users to organize different types of documents within a single collection of Lucene indices.
</blockquote>
<h2>Pedantry! Just tell me what to do!!</h2>
Let&#8217;s say I have an application and I want to be able to search all of my users. My application has three kinds of users: administrators, moderators and customers. There are a couple of ways I could organize my data. I could create an Elasticsearch index for each type:
<pre><code class="language-bash">PUT localhost:9200/administrators/administrator/1 -d &lt;some fields&gt;
PUT localhost:9200/moderators/moderator/1/ -d &lt;some fields&gt;
PUT localhost:9200/customers/customer/1 -d &lt;some fields&gt;

# Search for customers:
GET localhost:9200/customers/_search
</code></pre>
In this configuration, I have three indices, one for each type of user. Elasticsearch therefore requires a minimum of 3 shards, one for each index. This is kind of a waste, because each index only contains documents of a single type, and Elasticsearch offers the ability have multiple types on an index.
If I want my index to use fewer shards, I could instead create a single index and use types to distinguish between different users. Maybe my index could be called <code>accounts</code>, and my types would be the same as my user types:
<pre><code class="language-bash">PUT localhost:9200/accounts/administrator/1 -d &lt;some fields&gt;
PUT localhost:9200/accounts/moderator/1 -d &lt;some fields&gt;
PUT localhost:9200/accounts/customer/1 -d &lt;some fields&gt;

# Search for customers:
GET localhost:9200/accounts/customer/_search
</code></pre>
In the second configuration, I have one index containing all three types of users. Elasticsearch therefore only requires a minimum of one shard, and uses said types to scope search results to a particular class of user. My goal of reducing the shard count has been completed.
<h2>In closing</h2>
In this post, we focused more on the basic problem and solution to a particular concern without really addressing the underlying question of why you can&#8217;t have multiple indices on a shard. The short answer is that Elasticsearch is magic, and you should just accept this as a reality.
A slightly longer, more technical answer is that it spins up fresh Lucene instances (shards) when an index is created, and handles the coordination of requests between those shards. Elasticsearch takes this approach to make sure that data is spread out for both performance and integrity reasons. That&#8217;s why it doesn&#8217;t make sense to have multiple indices per shard, because that is essentially asking why you can&#8217;t have multiple Lucene instances per Lucene instance, when the Lucene instance is the lowest possible worker unit. That&#8217;s also why types exist. Types are used to achieve the functionality of multiple indices per shard, but the terminology can confuse the issue by masking what&#8217;s going on internally.
Hopefully this has all helped to clarify the issue. If you have questions, concerns, hate mail or songs of praise, feel free to <a href="mailto:support@bonsai.io">gives us a shout</a>!</div>

Can I have multiple indices on a shard?

<div>We often get asked about security at Bonsai. How do you ensure only I can see my data? What kind of encryption do you offer? What if my credentials are compromised?
<h2>1. Unique, Randomized URL</h2>
We offer several levels of security to all customers by default, and it starts
with your cluster&#8217;s hostname. Each cluster is assigned a unique, randomized
subdomain known only to the customer. This is a fairly standard practice for
cloud hosting providers (consider, for example, AWS Elastic Load Balancer or
CloudFront URLs).
The randomized hostname is security by obscurity: a good start, but not the full
story. A useful global identifier against which we can attach more tools.
Clusters are also provided with access credentials, which are required for all
requests. The extra layer of credentials makes it impossible for an attacker to
randomly guess the right combination to your cluster, and also provide a means
of periodically replacing access credentials.
Altogether, this cluster URL with credentials should be treated as sensitively
as any other database password, and only accessed from an application you
control and trust.
<h2>2. TLS/SSL</h2>
We support HTTPS on all of our clusters by default, providing encryption of all
traffic in transit via recent TLS encryption algorithms.
Encryption in transit works alongside your randomized URL and credentials,
helping to keep both private from any prying eyes. Not to mention the contents
of your data and search queries as your application interacts with the cluster.
<h2>3. Firewall</h2>
Like any good network administrators, we operate under the principle of least
privilege. Our servers are all tightly firewalled away from communication with
the wider internet, for obvious security reasons.
Each cluster&#8217;s private URL — with proper authorization over an encrypted
protocol — makes for a tightly controlled access path through our networks
and over to your cluster&#8217;s resources.
This access to Elasticsearch is controlled and mediated by our custom-built,
high-performance layer 7 routing proxy. This proxy serves as a central choke
point where we can authorize and audit all of the traffic incoming to your
traffic.
In addition to logging and access control, this proxy layer applies intelligent
connection management policies to help ensure best access patterns and
interacting with Elasticsearch itself. This lets us protect your cluster against
unexpected traffic surges, but that&#8217;s a whole different kind of security that
deserves its own article!
<h2>4. IP and Network CIDR Whitelisting</h2>
An additional benefit of our proxy layer is that it gives us tools to support
more advanced networking functionality that are not present in a stock
Elasticsearch deployment.
Currently in beta with select customers, we have the ability to also enforce IP
and network CIDR whitelisting. This provides an extra level of comfort (or
security compliance!) for customers whose deployments all route their outgoing
traffic through a predictable network gateway.
<h2>5. Encrypted, Offsite Backups</h2>
Bonsai also ensures regular backups are encrypted and stored in a completely
separate AWS account than the one that runs our servers. We use the bare minimum
in credentials for access to backups to ensure that your data is safe from
scrutiny, and always available for disaster recovery should the worst occur.
<h2>6. Encryption at Rest</h2>
Finally, we offer encryption at rest as a feature of single tenant clusters.
Generally speaking, we also have experience in designing and deploying clusters
built to customer specifications in order to support sensitive data and comply
with a range of regulatory policies, like HIPAA and Safe Harbor. <a href="mailto:support@bonsai.io">Contact
us</a> with any specific questions you might have on
security and compliance.</div>

Six Important Facts about Bonsai Security

<div>At <a href="https://bonsai.io" target="_blank" rel="nofollow">Bonsai</a>, we frequently answer questions about the many
interesting ways to use Elasticsearch.
So to help spread the knowledge, we&#8217;re hosting our first live Q&amp;A next Friday.
This is a great opportunity to talk directly with our founder, Nick Zadrozny,
who has managed some of the world’s biggest deployments of Elasticsearch.
The format is informal. It will take place on Google Hangouts and in a public
Slack channel. All sessions will be recorded and published on our blog at a
later date.
Here are the details:
<ul>
<li>WHAT: Bonsai’s Ask Me Anything</li>
<li>WHO: Bonsai Founder Nick Zadrozny (<a href="https://twitter.com/nz_" target="_blank" rel="nofollow">@nz_</a>)</li>
<li>WHEN: October 30, 2015 at 10-11am PDT / 1-2pm EDT / 1700-1800 UTC</li>
<li>WHERE: Google Hangouts <a href="https://bonsaiioama.wufoo.com/forms/bonsaiio-ama-with-nick-zadrozny/" target="_blank" rel="nofollow">Reserve your spot</a> to get the Hangout link.</li>
</ul>
We hope to see you there!</div>

Do you have questions about Elasticsearch? We have answers!

<div>Earlier today, new versions of Elasticsearch were released. These updates add new features as well as address two security vulnerabilities: CVE-2015-5377 and CVE-2015-5531. Initial assessments show that our systems are not vulnerable to either.
The first is a remote code execution vulnerability in Groovy, which targets Elasticsearch&#8217;s internal binary protocol on port 9300. Bonsai does not support or expose this protocol of port to end-users, it is only made available on a secure private network with access limited to nodes within a given Elasticsearch cluster.
Next is a directory traversal attack in the Elasticsearch Snapshot and Restore API. While we do currently take advantage of the Snapshot and Restore API for our periodic backups of customer data, we do not currently expose this API to end-users directly.
In both cases, our policy of only exposing features and functionality on a whitelist basis has helped to avoid exposing our customers to unexpected vulnerabilities.
Regardless, we&#8217;re accelerating our plans to roll out upgrades to newer versions of Elasticsearch. In addition to security fixes for the above vulnerability, recent releases of Elasticsearch all include a number of useful improvements that will directly help improve performance and reliability. Expect an announcement of our upgrade schedule shortly.
You can read more about the new 1.6.1 and 1.7.0 releases, and their security fixes, on <a href="https://www.elastic.co/blog/elasticsearch-1-7-0-and-1-6-1-released" target="_blank" rel="nofollow">Elastic&#8217;s release announcement</a>.</div>

Status Update on Recent Elasticsearch Security Vulnerabilities

<div>Elasticsearch is much more than just a search engine: it&#8217;s also a powerful analytics tool. One of the awesome things that Elasticsearch provides out of the box is the ability to calculate the distance between geographic points, and order the results by proximity. A common use case for this is an application where a user wants to see search results that are near a given point.
Take an app like OpenTable, for example, which uses Elasticsearch to power its mobile application. Someone using the iPhone app might want to see all of the best nearby restaurants. The application uses the iPhone&#8217;s GPS to get a latitude and longitude for the user, then uses Elasticsearch to quickly return all nearby results within a region, sorted by the closest ones.
However, some users trying to build similar capability find that their search is lightning fast in development, but slows to a crawl in production. We see this at Bonsai from time to time, when users open support tickets asking what gives.
The answer has to do with how search is being implemented. Consider the following snippet, and see if you can spot the problem:
<pre><code class="language-ruby">Restaurant.search(
 "*",
 sort: {
 _geo_distance: {
 location: "#{lat},#{lon}",
 order: "asc",
 unit: "km"
 }
 },
 size: 20
)
</code></pre>
Seems pretty straightforward, right? The code block returns the 20 nearest restaurants to <code>lat,lon</code>, ordered by nearest to furthest. But there is a subtle problem.
See how we&#8217;re searching for all documents (<code>*</code>)? The code is passing along a lat/lon point to Elasticsearch, and a geodistance is being calculated between that point and every other point in _all the documents in the index. If there are a large number of documents, then the time to compute geo_distances on all of them becomes non-trivial.
This is a direct consequence of how search engines sort results. In order to get the top 20 results, the engine has to calculate a score for every point in the set of results. Once that&#8217;s done, it needs to pass the scored results through a sorting algorithm before the top results can be returned.
This process is generally pretty well optimized in Elasticsearch. Calculating geo_distances is more computationally complex than basic TF-IDF relevancy scoring, but the default <code>sloppy_arc</code> algorithm is blazing fast, and over 99% accurate. The sorting algorithms are smart too. For example, it will stop sorting after the first 20 results (or whatever is specified), since we don&#8217;t need to know where a particular document ranks if it&#8217;s not going to be returned to the user anyway.
This explains why sorting by geo_distance is often blazing fast in development, when there is a low document count, but sluggish in production. You&#8217;re wasting too much CPU (and you may be memory-bound too).
<h2>What can I do about it?</h2>
The short answer is to reduce the number of documents you&#8217;re sorting over. Your particular use case will drive exactly how you want to do this, but it typically involves dropping the wildcard character and/or using filters.
Going back to the OpenTable example, if a user in San Francisco wants to see nearby Chinese restaurants, there&#8217;s no reason to calculate a geo_distance for a bistro in Paris or a steakhouse in Texas. The search should be limited to San Francisco, and specifically Chinese joints. Consider the following adaptation of the code above:
<pre><code class="language-ruby">Restaurant.search(
 "*",
 filter: {
	 must: [
 { term: { category: "chinese" }},
 { term: { city: "san francisco" }}
 ]
 },
 sort: {
 _geo_distance: {
 location: "#{lat},#{lon}",
 order: "asc",
 unit: "mi"
 }
 },
 size: 20
)
</code></pre>
This would limit the scope of the search to documents with a category of &#8220;chinese&#8221; and a city of &#8220;san francisco,&#8221; which is all the user wants to see anyway. That would require computing geo_distances and sorting on perhaps a few dozen entries in a specific area rather than millions around the world.
Another solution is to use the <a href="http://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-bounding-box-filter.html" target="_blank" rel="nofollow">geo_bounding_box filter</a>. This is a filter specifically designed for isolating geo_points in a result set. It allows you to define the lat/lon points of a rectangle, and filters by results within that area. This method is a little more elegant, but comes with a cost.
There&#8217;s some discussion over the efficiency of this approach, since it seems to have mixed results (performance-wise) in production. Using a bounding box still requires geographic coordinates to be computed for every document in the result set when the filter is applied. In theory, there is less overhead in determining whether a point resides within an area than computing the distance between two points, but it&#8217;s still overhead. So applying this filter to a wildcard query and nothing else will not be much more efficient than the original code.
<h2>Questions? Comments?</h2>
Using geo_distances in your app? We&#8217;d love to hear how you&#8217;re using it and what kind of performance you&#8217;re seeing. If you have any questions, comments or concerns, <a href="mailto:support@bonsai.io">shoot us an email</a> and we&#8217;ll see how we can help.</div>

Efficient sorting of geo distances in Elasticsearch

<div>We sometimes get support tickets from users asking about this error message. They report that some queries, like <code>hotdog</code>, work just fine while others, like <code>hamburger</code>, return an error like <code>IllegalArgumentException[docID must be &gt;= 0 and &lt; maxDoc=... (got docID=2147483647)]</code>. They&#8217;re confused because it seems to happen randomly and without an underlying cause. They ask: if the index is broken, why do some queries work? If the index is operating normally, why do some queries fail?
The tl;dr answer here is (usually) that the index is operating normally, but that it&#8217;s populated with bad data. This issue is not a bug per se, has been known for <a href="https://github.com/elastic/elasticsearch/issues/2426" target="_blank" rel="nofollow">a while</a> now, and should hopefully be fixed in Elasticsearch 2.0+.
To elaborate a bit, users normally see this error when they&#8217;re using custom scoring scripts to order results. For example, if you have an index of restaurants and want to score results based on the number of reviews each restaurant has, you might have something like this:
<pre><code>curl localhost:9200/restaurants/_search -d '{
 "query": {
 "function_score": {
 "query": {
 "match_all": {}
 },
 "script_score": {
 "script":"1.0 * log(doc[\"review_count\"].value + 2.718281828)"
 }
 }
 }
}
'
</code></pre>
This query uses a script to boost the score of matching documents, based on their number of reviews.
Can you spot the problem? Given our error message which asks for a value that’s greater than or equal to zero, can we find something in here that would be unhappy with a negative number? Here&#8217;s a hint: think back to your high school algebra classes.
Logarithms are only defined over the domain of positive, non-zero real numbers. Using a zero or negative value for the input is undefined, and results in a <code>NaN</code> response from the computer. The problem with the script above is there&#8217;s no logic to deal with negative values for <code>review_count</code>.
What happens if there is an edge case in your application where negative values are assigned for the number of reviews a restaurant has received? Elasticsearch will happily update the <code>review_count</code> field with a negative number. Then, during query time, if the document with the negative value in this field is matched by the query, Elasticsearch will try to calculate a score for it using that custom script. It ends up with an undefined number and complains.
<h2>But why can&#8217;t the error message just say that, instead of babbling incoherently? And what&#8217;s with the whole &#8220;got docID=2147483647&#8221; thing? I don&#8217;t even have that many documents!</h2>
Elasticsearch doesn&#8217;t really have a say in the matter, as the error is actually raised by Lucene (in <a href="http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/index/BaseCompositeReader.java" target="_blank" rel="nofollow">BaseCompositeReader.java</a>). The short explanation here is summarized by a comment deep in the Lucene <a href="http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Scorer.java" target="_blank" rel="nofollow">source code</a>:
<blockquote>
NOTE: The values Float.Nan,
Float.NEGATIVE_INFINITY and Float.POSITIVE_INFINITY are
not valid scores. Certain collectors (eg
TopScoreDocCollector) will not properly collect hits
with these scores.
</blockquote>
When non-numbers and infinities are used as scores, Lucene doesn&#8217;t have a way to deal with it. Instead, there&#8217;s a regression that sets the docID of the problem document to 2147483647 (Lucene instantiates docID as an <code>int</code>, which has a maximum value of 2147483647 in Java), which in turn causes the strange error message to be raised. Interestingly, it seems that Lucene imposes some checks to raise an error complaining about invalid scores, but these checks are not present in the Lucene expressions module, which is why the error is raised when using expressions as custom scoring scripts.
To work around this error, at least one of the following has to happen: 1) Elasticsearch needs to include validity checks with custom scoring scripts; 2) Lucene needs to test expressions for invalid scores and raise a proper warning; or 3) users need to be more cognizant of what values they&#8217;re indexing. Guess which one of these solutions is available today :-)
<h2>What can I do about it?</h2>
Elasticsearch doesn&#8217;t currently support unsigned integers (to be fair, neither does Java), so there isn&#8217;t a direct way to prevent negative numbers from being indexed. Ideally you would want your application to handle the logic of indexing sane values to Elasticsearch. If you see this error coming up in your logs, the first place to do is ensure you&#8217;re using good values.
Try running a <code>curl</code> command against the cluster and see what happens:
<pre><code>curl "localhost:9200/&lt;index&gt;/_search?pretty" -d '
{
 "filter":{
 "range":{
 "review_count":{
 "lt":0
 }
 }
 }
}'
</code></pre>
This search will return any and all documents in your index that have a negative value. This will help identify problem documents, and aid in troubleshooting any regressions on your end.
If you can’t otherwise purge those documents or fix their values, you can always just filter the original search itself to ensure only documents with valid field values are scored.
<pre><code>curl localhost:9200/restaurants/_search -d '{
 "query": {
 "function_score": {
 "query": {
 "match_all": {}
 },
 "script_score": {
 "script":"1.0 * log(doc[\"review_count\"].value + 2.718281828)"
 },
 "filter":{
 "range":{
 "review_count":{
 "gte":0
 }
 }
 }
 }
 }
}
'
</code></pre>
Got a tricky query? <a href="mailto:support@onemorecloud.com">Shoot us an email</a>! We&#8217;ll help you dig into it and identify a fix. We get questions about funky queries and responses all the time, so maybe we’ve already found an answer to your question.</div>

Elasticsearch and the IllegalArgumentException (docID must be >= 0)

<div>This month, some Bonsai customers were impacted by networking problems in AWS&#8217; Oregon region. Our services were not impacted internally, however customers in our Oregon region had issues resolving hostnames and connecting to their clusters. There was also a period of scheduled maintenance in upgrading all customers from Elasticsearch 1.3 to 1.4, which went smoothly for the most part. There have been some unexpected growing pains with 1.4, but we have more or less worked out the kinks.
For these reasons, we&#8217;re reporting our March 2015 availability as 99.991%. We expect that this should improve over the next month as system loads are rebalanced.
For further information on how we think about and report our availability, see our article on <a href="/blog/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider</a>.</div>

March 2015 Uptime: 99.991%

<div>There was a period of scheduled downtime in February, due to a <a href="https://www.elastic.co/blog/elasticsearch-1-4-3-and-1-3-8-released" target="_blank" rel="nofollow">serious vulnerability</a> in Elasticsearch, as well as some scheduled maintenance in rebalancing load across the system. Otherwise, it was a smooth month, and Bonsai is reporting 100% uptime.
The pending upgrade to Elasticsearch 1.4 is expected to mitigate some of the sporadic 503 errors and memory leaks that have been causing problems in 1.3, so we anticipate more monthly reports of high availability to come post-rollout.
For further information on how we think about and report our availability, see our article on <a href="/blog/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider</a>.</div>

February 2015 Uptime: 100.000%

<div>We kicked the year off with an uptime in January of more than 99.999%! Over the past month, we have seen a handful of 503 errors, which we are expecting to dissipate when we roll out Elasticsearch 1.4 to all regions. Stay tuned!
For further information on how we think about and report our availability, see our article on <a href="/blog/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider</a>.</div>

January 2015 Uptime: 99.999%

<div>Our uptime in December was greater than 99.999% after accounting for a very slight increase of 503 errors for the month on a very small number of tenants. We expect an upcoming rollout of Elasticsearch 1.4 will help us eliminate these errors, and until then we are capping our uptime at five nines for the month.
During December we did launch our first Elasticsearch 1.4 clusters into our AWS US East region. All new clusters in that region have been provisioned on Elasticsearch 1.4 to overall positive reception.
Development for Elasticsearch 1.4 had focused specifically on resiliency, and so brings a number of performance and availability improvements. We&#8217;ll be announcing plans for broader upgrades to Elasticsearch 1.4 within the next few weeks.
For further information on how we think about and report our availability, see our article on <a href="/blog/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider</a>.</div>

December 2014 Uptime: 99.999%

<div>Regional cluster upgrades, maintenance and rebalancing saw some brief windows of slightly elevated error rates in November. While our external polling reports 100% uptime, we calculate closer to 99.998% for the month.
For further reading, see our thoughts on <a href="/blog/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider</a> for more information on how we think about and measure total uptime.</div>

November 2014 Uptime: 99.998%

<div>Starting Wednesday, November 12, 2014, existing clusters will be automatically upgraded to version 1.3.5. In addition, new clusters will be provisioned with version 1.4. Customers looking to upgrade their existing clusters to version 1.4 will be able to upgrade starting the following week (November 17th).
1.3.5 Highlights
<pre><code> - Bug fixes for 1.3.x
</code></pre>
1.4 Highlights
<pre><code> - Major theme of 1.4.0 is resiliency: making Elasticsearch more stable and reliable than ever before, with better memory management, improved discovery algorithms, and better detection of corrupted data.
</code></pre>
Both 1.3.5 and 1.4 help us provide customers with a more resilient, faster, and more stable service.
Any customers with questions may email us at <a href="mailto:support@bonsai.io">support@bonsai.io</a>.</div>

Elasticsearch 1.3.5 and 1.4.0 Rollout Schedule

<div>Along with many of your favorite service providers, Bonsai will be taking a firm stance of support for the new <a href="https://www.eugdpr.org/" target="_blank" rel="nofollow">GDPR</a> (General Data Protection Regulation) legislation. We at Bonsai believe this is an important piece of legislation that is designed to strengthen and unify data protection laws for all individuals within the European Union. In essence, the GDPR requires internet companies to expose how they store and share data belonging to any EU citizen. The regulation will become effective and enforceable on May 25, 2018.
While this regulation specifically protects EU citizens, Bonsai has always been committed to the data privacy of its customers. As Internet users and developers ourselves, we appreciate the goals and ideals of this legislation, and find no reason to not apply the features we build for GDPR to everyone.
<h2>What is Bonsai doing about the GDPR?</h2>
From its beginning, Bonsai has been designed to store as little personal information as possible. We generally store things required to communicate with you without sounding like complete robots. Nonetheless, we did a thorough review of our data stores to make sure that nothing was hiding in our database tables.
We personally reviewed every table in our database. For those of you that are using PostgreSQL for your database, this was the SQL I used to manually inspect every column.
<pre><code>SELECT table_name, column_name, *
FROM information_schema.columns
WHERE table_schema = 'public'
ORDER BY table_name;
</code></pre>
Additionally, has collaborated with our lawyers to ensure that we are able to fulfill every aspect of this regulation. Here are the measures we’ve taken in advance of our attorneys to meet the GDPR deadline:
<ol>
<li>We have reviewed all of the data that we track about our customers, and identified what needs to be kept and what can be thrown away. This list will be incorporated into our application soon for your review.</li>
<li>We have internally documented our data flows of this data, and verified that all usage is needed for business operation. We then removed any data that was deemed unnecessary.</li>
<li>We have removed any vendor that is either not planning to support GDPR or has no public stance on GDPR.</li>
<li>We are improving security around the management and display of resource credentials, and are improving administrative workflows around system access for operational maintenance.</li>
<li>We have identified all third party cookies and removed those we no longer need or use. As an example, after review, we have decided to remove Facebook Pixel tracking.</li>
<li>We reviewed our Privacy policy, and are working to further clarify or update it where needed to satisfy new GDPR compliance requirements.</li>
</ol>
<h2>What do Bonsai Customers need to do?</h2>
GDPR identifies two roles for handling data. The role of the Data Processor, a legal entity that processes personal data on behalf of the Data Controller and the role of Data Controller, a legal entity that determines the purposes and means of the processing. As a customer of Bonsai, you will be in the role of Data Controller for the data loaded into Bonsai&#8217;s services, and Bonsai wil be in the role of Data Processor. Its important when you are reviewing GDPR to keep this in mind.
Among many other things, there are two things that you might need to do depending on your situation and jurisdiction. Below are the only impactful changes that we can foresee that might affect you as a result of using Bonsai:
<ol>
<li>Make sure your Terms of Service or Privacy Policy properly communicate to your users how you are using Bonsai (and any other similar services) on your website or app. We recommend you ensure your policies are up to date and clear to your readers.</li>
<li>If you are in the European Union, you’ll likely want to sign a Data Processing Agreement (DPA) with Bonsai. We’re happy to do so. Working with outside counsel, we are building a standard DPA that will be available to all customers.</li>
</ol>
If you have any questions, please don&#8217;t hesitate to contact us at <a href="mailto:support@bonsai.io">support@bonsai.io</a>.</div>

GDPR and Bonsai: An Overview of Your Data Privacy

<div>The newest feature to join the dashboard family is the one we’ve been most excited about: announcing Bonsai’s Elasticsearch operations metrics, available now on your cluster dashboard! All accounts have access to cluster metrics, including those that provision our add-ons through Heroku or Manifold.
These performance metrics are fast, live-updating, and retain up to a month of request data. From platform to UI, they are a technical feat built entirely in-house with great care and attention to detail. You can toggle between local and UTC time for debugging with app logs, explore time ranges, and click and drag sections of time to focus your time window.
<h3>Metrics are built for problem-solving</h3>
At its core, the metrics page shows visibility and transparency into what is coming in and out of your cluster. It reveals not just how users are interacting with your cluster, but how your app is.
The graphs in this page give you a visual, and what they say depends on the question you’re asking. A question like “Is it fast?” is answered by the performance heat map and percentiles. “When are we under the most load during the week?” — zoom out the timeline and look at request count patterns.
The most valuable puzzle these graphs unravel is the question of causation. Each graph retains the same timeline so that trends can be linked across multiple graphs. For example, an increase in request counts can reveal load handling problems, which may mean that you need more resources. An increase in queue time might be linked to higher write traffic, perhaps from an ill-timed indexing task during peak read times.
<h3>Worth the wait</h3>
We’ve long wanted to empower Bonsai customers with visibility into their cluster, but it’s been difficult to find a solution that would scale affordably and remain performant. For many years our operations and support teams have been using things like Grafana and Kibana to manage, debug, and optimize clusters. But our databases that house the huge amount of request data we handle are under limited load: only a small set of engineers have access to them, and during analysis they have the patience to wait for complicated aggregation requests to complete (sometimes up to 30s, depending on the time window length). There was no way we could scale these tools for our thousands of clusters we manage without spending a ton of money on hardware to keep up with our client base. Moreover, there was doubt that those existing platforms would support a good UI experience (namely, a fast one). In fact, our first prototypes worked well if we only showed requests from the last hour. But every time we zoomed out, things slowed to a glacial pace. That was unacceptable.
In many ways, building metrics was a waiting game. Tooling is much better now than when we first launched Bonsai back in 2012. Our current infrastructure supports the high standards we had when we set out to ship this project.
<h3>The technical stuff (or, Make It Fast)</h3>
From the onset of this project, our team had three goals to measure success:
<h4>1. The metrics database must be operationally dependable, affordable, and fast.</h4>
The platform has huge implications on cost, because we need to store
requests for 10k+ clusters and read metrics off of them quickly. That’s around 1B request insertions per day. The first metrics prototypes used Elasticsearch, but we ran into performance issues once we started ran larger aggregations for bigger time slices. Our metrics engineer ultimately replaced Elasticsearch with Cassandra, which is extremely fast, consistent, and scales for long periods of time.
<h4>2. The UI must be fast and responsive, and integrate easily our current javascript environment.</h4>
Web users have high expectations for interface usability. All graphs need to load on the DOM quickly and respond to interaction smoothly. A lot of our UI is React-based, so we first tried building our graphs with open-source React libraries. During prototyping, we weren’t able to customize and simplify the axes and labels. A whole new React-based charting library was developed in-house to give us all the tools we needed to keep our charts Tuftian. Keep on the lookout for more news from our team, as we’ll be open-sourcing this library soon.
<h4>3. The graphical design must be easy to read and relay information accurately.</h4>
Charts must be easy to read — the design should avoid ‘getting in the way’ of answering any questions people have when they come to the Metrics page. In order to build something we’d be proud to look at, we took some lessons from Tufte to keep our designs straightforward and readable.
<h3>Tufte as a baseline</h3>
If you’ve studied data visualization, you know the work of statistician <a href="https://www.edwardtufte.com/tufte/" target="_blank" rel="nofollow">Edward Tufte</a>. Our charts’ design takes two directives from Tufte very seriously. First, never lie. Second, don’t add chart junk.
<h4>1. Never Lie</h4>
Every graph has a measurable lie factor. The biggest contributor is when vertical space leads to incorrect comparisons within the data set. If you don’t start with base zero, the graphs can look drastically different based on the smallest value or zero.
Lie factor can easily fly under the radar to an unsuspecting viewer. While most people are trained from a young age to read charts and graphs, they aren’t taught about designing them or how to choose the appropriate graph type given a data set. It’s easy to give a lot of absolute weight to graphs, because we’re taught that they’re a picture of some truth. But what truth are they telling? Some lies are easy to spot, like this price of crude oil graph:
Others are harder to determine, usually because the display is unconventional.
This is an example of our request count graph, with a y-axis minimum of 0:
Now, let’s take the same graph and push the minimum closer to the minimum of the data set (700):
Forcing a non-zero minimum not only diminishes the ratio of total requests running per minute, but also lies about the ratio of write to read requests.
We mitigate our lie factor by always starting at zero. This means that the ratio between two data points is neither (a) given undue significance or (b) visually downplayed.
<h4>2. Don’t Add Chart Junk</h4>
The ‘stuff’ around charts and graphs can become eye-catching distractions to the thing that matters the most: the data. Sometimes, it’s a function of a overeager designer looking to add more visual interest (something I’ve often been culpable of):
Others are more simple graphing and chart ideas — simple lines or grids that we assume come with the territory, but whose removal creates clarity and improves focus and readability.
In each of the Bonsai metrics graphs, the labels are discrete and the same grey color. We don’t use gridlines. The palette is a limited set that each graph inherits. Overlays convey focused information only on hover. When we do use color, it’s because it has a significant meaning that operations-focused engineers inherently understand. We chose to display and label 5xx requests as red in the Response Codes graph because, at least in the western world where we have the majority of customers, red equates to danger or problems. In the first design pass-throughs, using an info-centric color like blue for 5xx requests looked like a chart lie.
<h4>Bonus: Perceptually Uniform Color Spaces</h4>
Whilst deciding on colors, our metrics engineer stumbled upon an article about perceptually uniform colors spaces.
Perceptually uniform color spaces “are human-friendly alternatives to color spaces such as sRGB, and they are incredibly helpful for designers working in code” (Madsen). When using a color space like sRGB, you cannot programmatically render colors that progress linearly. See this example of a 10 step transition from green to blue using sRGB:
The first half looks almost indistinguishable from each other, while the blues look like tints of blue instead of a linear transition from green.
<blockquote>
This happens because the default sRGB color space (and any color model built on it like HSV and HSL) is irregular, which means that even though the rectangles have evenly spaced hue values, the corresponding effect is not linear to the human eye.
(Madsen)
</blockquote>
If we instead use an even transition of hues, we get something like this:
In short, using color spaces that are perceptually uniform means that the graphs are easier to look at, account for color blindness, and ensure that any bars or boxes have <a href="https://usecontrast.com/guide" target="_blank" rel="nofollow">appropriate color contrast</a>. Our performance heatmap uses the <a href="https://github.com/bpostlethwaite/colormap" target="_blank" rel="nofollow">colormap npm package</a>, which is used to generate the palette using the viridis colormap (an open-source perceptually uniform colormap by Matlab, <a href="https://matplotlib.org/users/colormaps.html" target="_blank" rel="nofollow">https://matplotlib.org/users/colormaps.html</a>):
Perceptually uniform color is incredibly useful for any designer, and we really suggest <a href="https://programmingdesignsystems.com/color/perceptually-uniform-color-spaces/" target="_blank" rel="nofollow">reading more about it</a> and watching <a href="https://youtu.be/xAoljeRJ3lU" target="_blank" rel="nofollow">this lecture</a>. It makes all graphs better.
<h4>What’s Next</h4>
Metrics will continue to grow and become more powerful. Feature additions include additional timezone selection, date scrubbing, stateful links for collaboration, and request filters.
We’d love to hear any and all feedback, so let us know how you’re using your cluster metrics by dropping us a line at <a href="mailto:info@bonsai.io">info@bonsai.io</a>.
Happy searching, and happy holidays!
— ❤️ The Bonsai Product Team
<hr>
<h4>Bibliography</h4>
Madsen, Rune. “Programming Design Systems.” Programming Design Systems, programmingdesignsystems.com.
Tufte, Edward Rolfe. The visual display of quantitative information. Graphics Press, 2001.</div>

How We Built & Designed Operational Metrics

<div>Our operational uptime numbers from October to December of 2017.
*Virginia Incident in November:
<ul>
<li><a href="https://status.bonsai.io/incidents/rz5tf1ms1pgm" target="_blank" rel="nofollow">Performance issues for some users in US-East</a></li>
</ul>
See our entire uptime history here:
<ul>
<li><a href="https://status.bonsai.io" target="_blank" rel="nofollow">Bonsai Status</a></li>
</ul></div>

2017 Q4 Uptime Report

<div>*Sydney Incident in August:
<ul>
<li><a href="https://status.bonsai.io/incidents/vnnx8jz6628z" target="_blank" rel="nofollow">SSL Outage</a></li>
</ul>
See our entire uptime history here:
<ul>
<li><a href="https://status.bonsai.io" target="_blank" rel="nofollow">Bonsai Status</a></li>
</ul></div>

2017 Q3 Uptime Report

2017 Q2 Uptime Report

<div>2017 has started with a nearly perfect Q1. We&#8217;ve continually worked to stabilize our systems and report the numbers that display this improvement.
Our systems were largely unaffected by <a href="https://aws.amazon.com/message/41926/" target="_blank" rel="nofollow">Amazon&#8217;s February S3 Outage</a>. The only impact to Bonsai was a temporary suspension of automated backups. Clusters continued to serve requests normally, so no downtime was reported for February. This fact was referenced on <a href="https://status.bonsai.io/incidents/4n8ws6696bnm" target="_blank" rel="nofollow">our status page</a>.
There were two short-lived events in March affecting overall uptime. These were both proximately due to high load on hardware serving primarily free and staging clusters. Our response was to add capacity and migrate clusters to more evenly distribute the load.</div>

2017 Q1 Uptime Report

<div>This past week, a <a href="http://www.computerworld.com/article/3157766/linux/mongodb-ransomware-attacks-and-lessons-learned.html" target="_blank" rel="nofollow">widely-reported ransomware attack</a> was carried out against tens of thousands of MongoDB databases, and the attackers also targeted <a href="http://www.pcworld.com/article/3157417/security/after-mongodb-ransomware-groups-hit-exposed-elasticsearch-clusters.html" target="_blank" rel="nofollow">several hundred</a> Elasticsearch instances for good measure. Those affected were given a distressing ultimatum: pay Bitcoin to the attackers, or have their data deleted. To add further insult, many who paid the ransom did not even receive their data in return.
If you&#8217;re a Bonsai customer, rest assured: your clusters are perfectly safe.
This particular attack targeted databases that are open to the internet, and totally unsecured. Attackers would crawl public IP addresses en masse in order to find such services. By automating the collection, probing, and attack of MongoDB and Elasticsearch instances, hackers were able to ransom a significant amount of data in a very short amount of time.
As anyone with a public-facing SSH service on port 22 can attest, this kind of attack is not uncommon. With services that can automate the scanning of the entire IPv4 space, managing the security of your data is an ongoing concern for anyone hosting such services. And, unfortunately, <a href="http://thenextweb.com/insider/2017/01/08/mongodb-ransomware-exists-people-bad-security/" target="_blank" rel="nofollow">most people are not particularly great</a> at this kind of operational security.
At Bonsai, it&#8217;s our job to anticipate these kinds of problems, and make opinionated architectural design decisions to help prevent them. Here&#8217;s a small sample:
<ol>
<li>All Bonsai clusters are protected with HTTP authentication by default. These credentials are randomly generated, making them resistant to dictionary attacks or password reuse between services. All traffic to your cluster must have correct credentials, or it will be rejected with an HTTP 401 error. This protects against unauthorized access, including the kind of data exfiltration and modification used in this kind of attack.</li>
<li>Furthermore, we support secure TLS encryption by default for all clusters. This helps to further ensure the safety and security of your credentials by transmitting them over a secure channel. Granted, MITM and packet sniffing techniques weren&#8217;t used in this attack, but good security is designed in layers.</li>
<li>Bonsai cluster URLs have a randomly-generated hostname, and are routed through a specially designed layer 7 proxy, custom built for routing and load balancing Elasticsearch cluster traffic. This all but eliminates scanning by inflating the search space to an impractically large size. Your unique cluster URL is one of trillions of possibilities, and even if someone were to guess it perfectly, they can&#8217;t get any further without guessing the password.</li>
</ol>
We have also previously published a detailed blog entry covering a few other tidbits about <a href="https://bonsai.io/2015/10/26/bonsai-security" target="_blank" rel="nofollow">how we handle security for Elasticsearch</a>, if you&#8217;re interested in some light reading on the subject.
This most recent attack underscores that one of the most important thing our users can do to protect their data is to treat the cluster URL like a password. It&#8217;s sensitive information and access to it should follow the principal of least privilege. Also, we provide the option of <a href="https://docs.bonsai.io/docs/managing-your-cluster-url-and-credentials#rotating-your-url-credentials" target="_blank" rel="nofollow">rotating the randomly-generated credentials</a>, which users should do regularly (and especially if there is reason to suspect that their full URL has been leaked). For detailed information about your cluster URL, see our thorough documentation here:
<ul>
<li><a href="https://docs.bonsai.io/docs/managing-your-cluster-url-and-credentials" target="_blank" rel="nofollow">Managing your Cluster URL and Credentials</a></li>
</ul>
We take security very seriously and are constantly monitoring for and responding to potential threats. It would be much easier to design an architecture that pushed some of these decisions onto our customers. But we know that our customers appreciate our service because of the opinionated expertise that we integrate directly into our systems. We take every precaution and are opinionated about our infrastructure as a result.
Hopefully you&#8217;ve found this insightful! Both for a few specific design points and policies, as well as for the overall sense of responsibility and design philosophy with which we approach the design of our service.
If you ever have any concerns or questions about how we&#8217;re keeping your data safe, or what you can do to beef up your own security, please don&#8217;t hesitate to reach out!</div>

Bonsai Unaffected by Recent Ransomware Attack

<div>October was a nearly perfect month except for a widespread .io outage on Oct 28th. We wrote a detailed log of the incident, including what we did to prevent us from being affecting from such an outage, here:
<ul>
<li><a href="https://status.bonsai.io/incidents/9klg801sg1hd" target="_blank" rel="nofollow">Intermittent DNS resolution errors impacting .io TLD: Incident Report for Bonsai</a></li>
</ul>
Our ops team that day really went above and beyond to make sure impact was minimal, and we&#8217;re looking forward to a great year with improved performance to report even better numbers in 2017.</div>

2016 Q4 Uptime Report

<div>For every quarter we report on the uptime for production-grade clusters on our shared tier. Interested in how we calculate uptime? See our in-depth breakdown here:
<ul>
<li><a href="/2014/10/02/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider </a></li>
</ul></div>

2016 Q3 Uptime Report

<div>Getting started with Bonsai Elasticsearch and Ruby on Rails is incredibly quick and simple to do. [Sign Up]({{ site.bonsai_app_url }}/signup) for your free account we&#8217;ll take it from there!
<h3>Creating Your First Bonsai Cluster</h3>
Once you&#8217;ve signed up for your free account on Bonsai, you&#8217;ll be prompted to create a free developer cluster. Give your cluster a name and pick a region. The name helps you to distinguish between different applications and the region is how we manage our infrastructure on Amazon Web Services. Click on the &#8220;Create Cluster&#8221; button and voila, you&#8217;ve created your very first Bonsai cluster!
<h3>Our Example Rails App</h3>
Our example Rails app will provide you a good outline to follow along. You can access the Github repo here:
<ul>
<li><a href="https://github.com/omc/bonsai-rails-example" target="_blank" rel="nofollow">https://github.com/omc/bonsai-rails-example</a></li>
</ul>
In our example app, Bookfaces, we&#8217;ll have a list of books that are searchable via Book Title, Author, Publisher and Genre. You can see the app in action here:
<ul>
<li><a href="https://agile-anchorage-88938.herokuapp.com/" target="_blank" rel="nofollow">https://agile-anchorage-88938.herokuapp.com/</a></li>
</ul>
So now that we have our example code (or a Rails project of your own) let&#8217;s go ahead and get things ready for Bonsai.
###Connecting Your Rails App to Your Bonsai Cluster
Bonsai clusters are created with unique URLs designed for secure, authenticated access to the Elasticsearch API. To avoid security concerns we strongly recommend adding your Bonsai Elasticsearch URL to your environment as a variable, you can do so by running:
<pre><code> export BONSAI_URL="&lt;YOUR BONSAI URL&gt;"
</code></pre>
You can learn more about your Bonsai URL here:
<ul>
<li><a href="https://docs.bonsai.io/docs/managing-your-cluster-url-and-credentials" target="_blank" rel="nofollow">https://docs.bonsai.io/docs/managing-your-cluster-url-and-credentials</a></li>
</ul>
Now let&#8217;s add the following gems to your Gemfile:
<pre><code class="language-ruby"> gem 'elasticsearch-rails'
 gem 'elasticsearch-model'
 gem 'bonsai-elasticsearch-rails'
</code></pre>
Run <code>bundle install</code> and we&#8217;ll move on to the next step.
###Deploying to Heroku
As with all things related to the development process, deploy early and often. It’s a good idea to double check that you have completed all the steps above. For our example app we used Heroku to deploy. For step by step instructions as to how you can deploy your rails app with Heroku, please consult their excellent documentation:
<ul>
<li><a href="https://devcenter.heroku.com/articles/getting-started-with-rails4" target="_blank" rel="nofollow">https://devcenter.heroku.com/articles/getting-started-with-rails4</a></li>
</ul>
Once you have deployed to Heroku you can add the Bonsai add-on from the Overview tab.
###Inserting Data
Our next step will allow us to add data to the cluster as well as how to do a local database import. Let’s start with populating our local app first. For convenience sake, we will use the Faker gem to help populate db. Here’s a link to the Faker gem documentation:
<ul>
<li><a href="https://github.com/stympy/faker" target="_blank" rel="nofollow">https://github.com/stympy/faker</a></li>
</ul>
Let’s add the Faker gem to the gemfile:
<pre><code class="language-ruby"> gem 'faker'
</code></pre>
and run our <code>bundle install</code>. Now that we’ve got the Faker gem installed, we can start adding some sample data. Take a look at our <code>schema.rb</code> and <code>seeds.rb</code>. Files in our sample code if you need some guidance in how to structure those files.
Run the following commands to set up your local db, note that in Rails 5 we can now use the rails command instead of rake.
<pre><code class="language-ruby"> rails db:create
 rails db:migrate
 rails db:seed
</code></pre>
Now run <code>rails s</code> and check out all the new books for Bookfaces! Now that we know how to populate our local db, let’s proceed to getting our models ready for indexation.
###Get Your App Ready to Index Models You&#8217;d Like to be Searchable
We&#8217;ll want to prepare our models to be indexed and searchable by Bonsai. Take a look at <code>app/models/book.rb</code>, where we include our <code>Elasticsearch::Model</code>
<pre><code class="language-ruby"> class Book &lt; ApplicationRecord
 include Elasticsearch::Model

 belongs_to :author
 belongs_to :publisher
 belongs_to :genre

 ### View the entirety in the GitHub repo: https://github.com/omc/bonsai-rails-example/blob/master/app/models/book.rb
 end
</code></pre>
We&#8217;ll want to do this for all models that we want indexed. It&#8217;s also advisable that we create rake tasks to import documents, it will allow you to index searchable records via rake tasks. Plus, it’s important to to have reliable and easily executable indexing scripts should you need to need to duplicate this process or have a backup at hand. You can do this by creating a file <code>lib/tasks/elasticsearch.rake</code> and including:
<pre><code class="language-ruby"> require 'elasticsearch/rails/tasks/import'
</code></pre>
Keep in mind you will need to explicitly create your index before you can put documents there, since Bonsai does not support &#8220;lazy&#8221; index creation. The best way to create your index is using the rake task we set up in the prior step. The first time you index your content, you&#8217;ll want to set the FORCE parameter:
<pre><code class="language-ruby"> rake environment elasticsearch:import:all FORCE=y
</code></pre>
The FORCE parameter will explicitly delete and recreate indices, so you probably don&#8217;t want to use it frequently. Subsequent indexing tasks can drop this parameter.
The alternative way to do it would be to create your indices with curl. So in our Book model, use the following command to create the index:
<pre><code class="language-shell"> curl -XPUT http://username:password@redwood-12345.us-east-1.bonsai.io/books
</code></pre>
###Done!
You have created your Bonsai cluster, added your cluster URL to your App, indexed the models and now can deploy to the service of your choice! For our example we used Heroku, where you can install the <a href="https://elements.heroku.com/addons/bonsai" target="_blank" rel="nofollow">Bonsai Add-on</a>.
Please feel free to give us feedback or ask questions at <a href="mailto:info@omc.io">info@omc.io</a> or on Twitter [@{{ site.author.twitter }}](https://twitter.com/{{ site.author.twitter }}).</div>