Whether you are an Elasticsearch beginner or an expert, get helpful Bonsai.io news as well as Elasticsearch tutorials, tips, and updates from our team.

Updates from the Bonsai Elasticsearch team, from One More Cloud: the first and best cloud search platform, since 2009.


Search Snippets


Schedule a free consultation to see how we can create a customized plan to meet your search needs.


Find out how we can help you.

<div>We sometimes hear from users concerned about their shard counts. They ask how they can instruct Elasticsearch to add multiple indices to a single shard. This is a surprisingly difficult question to answer in full because the terminology can confuse the issue quite a bit.
In Elasticsearch, every index maps to one or more shards, but the reverse is not true for some pretty sound technical reasons. However, changes to data mapping do afford users the ability to logically separate data on a single shard, thus producing the effect of multiple indices per shard for many use cases.
If that explanation sounds confusing, don&#8217;t worry. In this post, we&#8217;re going to lay out the issue and solution as easily as possible.
<h2>The problem</h2>
<a href="https://en.wikipedia.org/wiki/Lucene" target="_blank" rel="nofollow">Lucene</a> is the Java-based information retrieval software that forms the backbone of Elasticsearch. Elasticsearch adds a few layers of abstraction on top of Lucene to provide features that are extremely useful to production applications: aggregations, analysis, distributed search, and more, all controlled through a well-documented and easy to use RESTful API. However, with these features comes a richer glossary of terms, with slightly more subtle definitions.
The biggest point of confusion for users is the word &#8220;index.&#8221; This word means different things, depending on the context. It can be used:
<ul>
<li>As a noun in the context of Lucene</li>
<li>As a noun in the context of Elasticsearch</li>
<li>As a verb in the context of either Lucene or Elasticsearch</li>
</ul>
Catch that? A Lucene index and an Elasticsearch index are very different things, even though Elasticsearch implements Lucene. In the context of Lucene, an index is basically a Lucene instance. In the context of Elasticsearch, an index is&#8230; well, read on.
I mentioned earlier that Elasticsearch offers a rich glossary, with terms like &#8220;shard,&#8221; &#8220;index,&#8221; and &#8220;type,&#8221; which are closely related but distinct concepts. So what distinguishes a shard from an index from a type in Elasticsearch? In the Elasticsearch lexicon:
<ul>
<li>A shard is a single Lucene instance</li>
<li>An index is a logical namespace that points to one or more shards</li>
<li>A type is a piece of metadata used to logically distinguish between documents of different types within an index</li>
<li>A document in Elasticsearch is essentially a Lucene document with some injected metadata</li>
</ul>
To distill this all down:
<blockquote>
An Elasticsearch shard is a Lucene index. An Elasticsearch index is a collection of one or more Lucene indices. An Elasticsearch type is a bit of metadata injected into the document which allows users to organize different types of documents within a single collection of Lucene indices.
</blockquote>
<h2>Pedantry! Just tell me what to do!!</h2>
Let&#8217;s say I have an application and I want to be able to search all of my users. My application has three kinds of users: administrators, moderators and customers. There are a couple of ways I could organize my data. I could create an Elasticsearch index for each type:
<pre><code class="language-bash">PUT localhost:9200/administrators/administrator/1 -d &lt;some fields&gt;
PUT localhost:9200/moderators/moderator/1/ -d &lt;some fields&gt;
PUT localhost:9200/customers/customer/1 -d &lt;some fields&gt;

# Search for customers:
GET localhost:9200/customers/_search
</code></pre>
In this configuration, I have three indices, one for each type of user. Elasticsearch therefore requires a minimum of 3 shards, one for each index. This is kind of a waste, because each index only contains documents of a single type, and Elasticsearch offers the ability have multiple types on an index.
If I want my index to use fewer shards, I could instead create a single index and use types to distinguish between different users. Maybe my index could be called <code>accounts</code>, and my types would be the same as my user types:
<pre><code class="language-bash">PUT localhost:9200/accounts/administrator/1 -d &lt;some fields&gt;
PUT localhost:9200/accounts/moderator/1 -d &lt;some fields&gt;
PUT localhost:9200/accounts/customer/1 -d &lt;some fields&gt;

# Search for customers:
GET localhost:9200/accounts/customer/_search
</code></pre>
In the second configuration, I have one index containing all three types of users. Elasticsearch therefore only requires a minimum of one shard, and uses said types to scope search results to a particular class of user. My goal of reducing the shard count has been completed.
<h2>In closing</h2>
In this post, we focused more on the basic problem and solution to a particular concern without really addressing the underlying question of why you can&#8217;t have multiple indices on a shard. The short answer is that Elasticsearch is magic, and you should just accept this as a reality.
A slightly longer, more technical answer is that it spins up fresh Lucene instances (shards) when an index is created, and handles the coordination of requests between those shards. Elasticsearch takes this approach to make sure that data is spread out for both performance and integrity reasons. That&#8217;s why it doesn&#8217;t make sense to have multiple indices per shard, because that is essentially asking why you can&#8217;t have multiple Lucene instances per Lucene instance, when the Lucene instance is the lowest possible worker unit. That&#8217;s also why types exist. Types are used to achieve the functionality of multiple indices per shard, but the terminology can confuse the issue by masking what&#8217;s going on internally.
Hopefully this has all helped to clarify the issue. If you have questions, concerns, hate mail or songs of praise, feel free to <a href="mailto:support@bonsai.io">gives us a shout</a>!</div>

Can I have multiple indices on a shard?

<div>We often get asked about security at Bonsai. How do you ensure only I can see my data? What kind of encryption do you offer? What if my credentials are compromised?
<h2>1. Unique, Randomized URL</h2>
We offer several levels of security to all customers by default, and it starts
with your cluster&#8217;s hostname. Each cluster is assigned a unique, randomized
subdomain known only to the customer. This is a fairly standard practice for
cloud hosting providers (consider, for example, AWS Elastic Load Balancer or
CloudFront URLs).
The randomized hostname is security by obscurity: a good start, but not the full
story. A useful global identifier against which we can attach more tools.
Clusters are also provided with access credentials, which are required for all
requests. The extra layer of credentials makes it impossible for an attacker to
randomly guess the right combination to your cluster, and also provide a means
of periodically replacing access credentials.
Altogether, this cluster URL with credentials should be treated as sensitively
as any other database password, and only accessed from an application you
control and trust.
<h2>2. TLS/SSL</h2>
We support HTTPS on all of our clusters by default, providing encryption of all
traffic in transit via recent TLS encryption algorithms.
Encryption in transit works alongside your randomized URL and credentials,
helping to keep both private from any prying eyes. Not to mention the contents
of your data and search queries as your application interacts with the cluster.
<h2>3. Firewall</h2>
Like any good network administrators, we operate under the principle of least
privilege. Our servers are all tightly firewalled away from communication with
the wider internet, for obvious security reasons.
Each cluster&#8217;s private URL — with proper authorization over an encrypted
protocol — makes for a tightly controlled access path through our networks
and over to your cluster&#8217;s resources.
This access to Elasticsearch is controlled and mediated by our custom-built,
high-performance layer 7 routing proxy. This proxy serves as a central choke
point where we can authorize and audit all of the traffic incoming to your
traffic.
In addition to logging and access control, this proxy layer applies intelligent
connection management policies to help ensure best access patterns and
interacting with Elasticsearch itself. This lets us protect your cluster against
unexpected traffic surges, but that&#8217;s a whole different kind of security that
deserves its own article!
<h2>4. IP and Network CIDR Whitelisting</h2>
An additional benefit of our proxy layer is that it gives us tools to support
more advanced networking functionality that are not present in a stock
Elasticsearch deployment.
Currently in beta with select customers, we have the ability to also enforce IP
and network CIDR whitelisting. This provides an extra level of comfort (or
security compliance!) for customers whose deployments all route their outgoing
traffic through a predictable network gateway.
<h2>5. Encrypted, Offsite Backups</h2>
Bonsai also ensures regular backups are encrypted and stored in a completely
separate AWS account than the one that runs our servers. We use the bare minimum
in credentials for access to backups to ensure that your data is safe from
scrutiny, and always available for disaster recovery should the worst occur.
<h2>6. Encryption at Rest</h2>
Finally, we offer encryption at rest as a feature of single tenant clusters.
Generally speaking, we also have experience in designing and deploying clusters
built to customer specifications in order to support sensitive data and comply
with a range of regulatory policies, like HIPAA and Safe Harbor. <a href="mailto:support@bonsai.io">Contact
us</a> with any specific questions you might have on
security and compliance.</div>

Six Important Facts about Bonsai Security

<div>At <a href="https://bonsai.io" target="_blank" rel="nofollow">Bonsai</a>, we frequently answer questions about the many
interesting ways to use Elasticsearch.
So to help spread the knowledge, we&#8217;re hosting our first live Q&amp;A next Friday.
This is a great opportunity to talk directly with our founder, Nick Zadrozny,
who has managed some of the world’s biggest deployments of Elasticsearch.
The format is informal. It will take place on Google Hangouts and in a public
Slack channel. All sessions will be recorded and published on our blog at a
later date.
Here are the details:
<ul>
<li>WHAT: Bonsai’s Ask Me Anything</li>
<li>WHO: Bonsai Founder Nick Zadrozny (<a href="https://twitter.com/nz_" target="_blank" rel="nofollow">@nz_</a>)</li>
<li>WHEN: October 30, 2015 at 10-11am PDT / 1-2pm EDT / 1700-1800 UTC</li>
<li>WHERE: Google Hangouts <a href="https://bonsaiioama.wufoo.com/forms/bonsaiio-ama-with-nick-zadrozny/" target="_blank" rel="nofollow">Reserve your spot</a> to get the Hangout link.</li>
</ul>
We hope to see you there!</div>

Do you have questions about Elasticsearch? We have answers!

<div>Earlier today, new versions of Elasticsearch were released. These updates add new features as well as address two security vulnerabilities: CVE-2015-5377 and CVE-2015-5531. Initial assessments show that our systems are not vulnerable to either.
The first is a remote code execution vulnerability in Groovy, which targets Elasticsearch&#8217;s internal binary protocol on port 9300. Bonsai does not support or expose this protocol of port to end-users, it is only made available on a secure private network with access limited to nodes within a given Elasticsearch cluster.
Next is a directory traversal attack in the Elasticsearch Snapshot and Restore API. While we do currently take advantage of the Snapshot and Restore API for our periodic backups of customer data, we do not currently expose this API to end-users directly.
In both cases, our policy of only exposing features and functionality on a whitelist basis has helped to avoid exposing our customers to unexpected vulnerabilities.
Regardless, we&#8217;re accelerating our plans to roll out upgrades to newer versions of Elasticsearch. In addition to security fixes for the above vulnerability, recent releases of Elasticsearch all include a number of useful improvements that will directly help improve performance and reliability. Expect an announcement of our upgrade schedule shortly.
You can read more about the new 1.6.1 and 1.7.0 releases, and their security fixes, on <a href="https://www.elastic.co/blog/elasticsearch-1-7-0-and-1-6-1-released" target="_blank" rel="nofollow">Elastic&#8217;s release announcement</a>.</div>

Status Update on Recent Elasticsearch Security Vulnerabilities

<div>Elasticsearch is much more than just a search engine: it&#8217;s also a powerful analytics tool. One of the awesome things that Elasticsearch provides out of the box is the ability to calculate the distance between geographic points, and order the results by proximity. A common use case for this is an application where a user wants to see search results that are near a given point.
Take an app like OpenTable, for example, which uses Elasticsearch to power its mobile application. Someone using the iPhone app might want to see all of the best nearby restaurants. The application uses the iPhone&#8217;s GPS to get a latitude and longitude for the user, then uses Elasticsearch to quickly return all nearby results within a region, sorted by the closest ones.
However, some users trying to build similar capability find that their search is lightning fast in development, but slows to a crawl in production. We see this at Bonsai from time to time, when users open support tickets asking what gives.
The answer has to do with how search is being implemented. Consider the following snippet, and see if you can spot the problem:
<pre><code class="language-ruby">Restaurant.search(
 "*",
 sort: {
 _geo_distance: {
 location: "#{lat},#{lon}",
 order: "asc",
 unit: "km"
 }
 },
 size: 20
)
</code></pre>
Seems pretty straightforward, right? The code block returns the 20 nearest restaurants to <code>lat,lon</code>, ordered by nearest to furthest. But there is a subtle problem.
See how we&#8217;re searching for all documents (<code>*</code>)? The code is passing along a lat/lon point to Elasticsearch, and a geodistance is being calculated between that point and every other point in _all the documents in the index. If there are a large number of documents, then the time to compute geo_distances on all of them becomes non-trivial.
This is a direct consequence of how search engines sort results. In order to get the top 20 results, the engine has to calculate a score for every point in the set of results. Once that&#8217;s done, it needs to pass the scored results through a sorting algorithm before the top results can be returned.
This process is generally pretty well optimized in Elasticsearch. Calculating geo_distances is more computationally complex than basic TF-IDF relevancy scoring, but the default <code>sloppy_arc</code> algorithm is blazing fast, and over 99% accurate. The sorting algorithms are smart too. For example, it will stop sorting after the first 20 results (or whatever is specified), since we don&#8217;t need to know where a particular document ranks if it&#8217;s not going to be returned to the user anyway.
This explains why sorting by geo_distance is often blazing fast in development, when there is a low document count, but sluggish in production. You&#8217;re wasting too much CPU (and you may be memory-bound too).
<h2>What can I do about it?</h2>
The short answer is to reduce the number of documents you&#8217;re sorting over. Your particular use case will drive exactly how you want to do this, but it typically involves dropping the wildcard character and/or using filters.
Going back to the OpenTable example, if a user in San Francisco wants to see nearby Chinese restaurants, there&#8217;s no reason to calculate a geo_distance for a bistro in Paris or a steakhouse in Texas. The search should be limited to San Francisco, and specifically Chinese joints. Consider the following adaptation of the code above:
<pre><code class="language-ruby">Restaurant.search(
 "*",
 filter: {
	 must: [
 { term: { category: "chinese" }},
 { term: { city: "san francisco" }}
 ]
 },
 sort: {
 _geo_distance: {
 location: "#{lat},#{lon}",
 order: "asc",
 unit: "mi"
 }
 },
 size: 20
)
</code></pre>
This would limit the scope of the search to documents with a category of &#8220;chinese&#8221; and a city of &#8220;san francisco,&#8221; which is all the user wants to see anyway. That would require computing geo_distances and sorting on perhaps a few dozen entries in a specific area rather than millions around the world.
Another solution is to use the <a href="http://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-bounding-box-filter.html" target="_blank" rel="nofollow">geo_bounding_box filter</a>. This is a filter specifically designed for isolating geo_points in a result set. It allows you to define the lat/lon points of a rectangle, and filters by results within that area. This method is a little more elegant, but comes with a cost.
There&#8217;s some discussion over the efficiency of this approach, since it seems to have mixed results (performance-wise) in production. Using a bounding box still requires geographic coordinates to be computed for every document in the result set when the filter is applied. In theory, there is less overhead in determining whether a point resides within an area than computing the distance between two points, but it&#8217;s still overhead. So applying this filter to a wildcard query and nothing else will not be much more efficient than the original code.
<h2>Questions? Comments?</h2>
Using geo_distances in your app? We&#8217;d love to hear how you&#8217;re using it and what kind of performance you&#8217;re seeing. If you have any questions, comments or concerns, <a href="mailto:support@bonsai.io">shoot us an email</a> and we&#8217;ll see how we can help.</div>

Efficient sorting of geo distances in Elasticsearch

<div>We sometimes get support tickets from users asking about this error message. They report that some queries, like <code>hotdog</code>, work just fine while others, like <code>hamburger</code>, return an error like <code>IllegalArgumentException[docID must be &gt;= 0 and &lt; maxDoc=... (got docID=2147483647)]</code>. They&#8217;re confused because it seems to happen randomly and without an underlying cause. They ask: if the index is broken, why do some queries work? If the index is operating normally, why do some queries fail?
The tl;dr answer here is (usually) that the index is operating normally, but that it&#8217;s populated with bad data. This issue is not a bug per se, has been known for <a href="https://github.com/elastic/elasticsearch/issues/2426" target="_blank" rel="nofollow">a while</a> now, and should hopefully be fixed in Elasticsearch 2.0+.
To elaborate a bit, users normally see this error when they&#8217;re using custom scoring scripts to order results. For example, if you have an index of restaurants and want to score results based on the number of reviews each restaurant has, you might have something like this:
<pre><code>curl localhost:9200/restaurants/_search -d '{
 "query": {
 "function_score": {
 "query": {
 "match_all": {}
 },
 "script_score": {
 "script":"1.0 * log(doc[\"review_count\"].value + 2.718281828)"
 }
 }
 }
}
'
</code></pre>
This query uses a script to boost the score of matching documents, based on their number of reviews.
Can you spot the problem? Given our error message which asks for a value that’s greater than or equal to zero, can we find something in here that would be unhappy with a negative number? Here&#8217;s a hint: think back to your high school algebra classes.
Logarithms are only defined over the domain of positive, non-zero real numbers. Using a zero or negative value for the input is undefined, and results in a <code>NaN</code> response from the computer. The problem with the script above is there&#8217;s no logic to deal with negative values for <code>review_count</code>.
What happens if there is an edge case in your application where negative values are assigned for the number of reviews a restaurant has received? Elasticsearch will happily update the <code>review_count</code> field with a negative number. Then, during query time, if the document with the negative value in this field is matched by the query, Elasticsearch will try to calculate a score for it using that custom script. It ends up with an undefined number and complains.
<h2>But why can&#8217;t the error message just say that, instead of babbling incoherently? And what&#8217;s with the whole &#8220;got docID=2147483647&#8221; thing? I don&#8217;t even have that many documents!</h2>
Elasticsearch doesn&#8217;t really have a say in the matter, as the error is actually raised by Lucene (in <a href="http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/index/BaseCompositeReader.java" target="_blank" rel="nofollow">BaseCompositeReader.java</a>). The short explanation here is summarized by a comment deep in the Lucene <a href="http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/search/Scorer.java" target="_blank" rel="nofollow">source code</a>:
<blockquote>
NOTE: The values Float.Nan,
Float.NEGATIVE_INFINITY and Float.POSITIVE_INFINITY are
not valid scores. Certain collectors (eg
TopScoreDocCollector) will not properly collect hits
with these scores.
</blockquote>
When non-numbers and infinities are used as scores, Lucene doesn&#8217;t have a way to deal with it. Instead, there&#8217;s a regression that sets the docID of the problem document to 2147483647 (Lucene instantiates docID as an <code>int</code>, which has a maximum value of 2147483647 in Java), which in turn causes the strange error message to be raised. Interestingly, it seems that Lucene imposes some checks to raise an error complaining about invalid scores, but these checks are not present in the Lucene expressions module, which is why the error is raised when using expressions as custom scoring scripts.
To work around this error, at least one of the following has to happen: 1) Elasticsearch needs to include validity checks with custom scoring scripts; 2) Lucene needs to test expressions for invalid scores and raise a proper warning; or 3) users need to be more cognizant of what values they&#8217;re indexing. Guess which one of these solutions is available today :-)
<h2>What can I do about it?</h2>
Elasticsearch doesn&#8217;t currently support unsigned integers (to be fair, neither does Java), so there isn&#8217;t a direct way to prevent negative numbers from being indexed. Ideally you would want your application to handle the logic of indexing sane values to Elasticsearch. If you see this error coming up in your logs, the first place to do is ensure you&#8217;re using good values.
Try running a <code>curl</code> command against the cluster and see what happens:
<pre><code>curl "localhost:9200/&lt;index&gt;/_search?pretty" -d '
{
 "filter":{
 "range":{
 "review_count":{
 "lt":0
 }
 }
 }
}'
</code></pre>
This search will return any and all documents in your index that have a negative value. This will help identify problem documents, and aid in troubleshooting any regressions on your end.
If you can’t otherwise purge those documents or fix their values, you can always just filter the original search itself to ensure only documents with valid field values are scored.
<pre><code>curl localhost:9200/restaurants/_search -d '{
 "query": {
 "function_score": {
 "query": {
 "match_all": {}
 },
 "script_score": {
 "script":"1.0 * log(doc[\"review_count\"].value + 2.718281828)"
 },
 "filter":{
 "range":{
 "review_count":{
 "gte":0
 }
 }
 }
 }
 }
}
'
</code></pre>
Got a tricky query? <a href="mailto:support@onemorecloud.com">Shoot us an email</a>! We&#8217;ll help you dig into it and identify a fix. We get questions about funky queries and responses all the time, so maybe we’ve already found an answer to your question.</div>

Elasticsearch and the IllegalArgumentException (docID must be >= 0)

<div>This month, some Bonsai customers were impacted by networking problems in AWS&#8217; Oregon region. Our services were not impacted internally, however customers in our Oregon region had issues resolving hostnames and connecting to their clusters. There was also a period of scheduled maintenance in upgrading all customers from Elasticsearch 1.3 to 1.4, which went smoothly for the most part. There have been some unexpected growing pains with 1.4, but we have more or less worked out the kinks.
For these reasons, we&#8217;re reporting our March 2015 availability as 99.991%. We expect that this should improve over the next month as system loads are rebalanced.
For further information on how we think about and report our availability, see our article on <a href="/blog/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider</a>.</div>

March 2015 Uptime: 99.991%

<div>There was a period of scheduled downtime in February, due to a <a href="https://www.elastic.co/blog/elasticsearch-1-4-3-and-1-3-8-released" target="_blank" rel="nofollow">serious vulnerability</a> in Elasticsearch, as well as some scheduled maintenance in rebalancing load across the system. Otherwise, it was a smooth month, and Bonsai is reporting 100% uptime.
The pending upgrade to Elasticsearch 1.4 is expected to mitigate some of the sporadic 503 errors and memory leaks that have been causing problems in 1.3, so we anticipate more monthly reports of high availability to come post-rollout.
For further information on how we think about and report our availability, see our article on <a href="/blog/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider</a>.</div>

February 2015 Uptime: 100.000%

<div>We kicked the year off with an uptime in January of more than 99.999%! Over the past month, we have seen a handful of 503 errors, which we are expecting to dissipate when we roll out Elasticsearch 1.4 to all regions. Stay tuned!
For further information on how we think about and report our availability, see our article on <a href="/blog/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider</a>.</div>

January 2015 Uptime: 99.999%

<div>Our uptime in December was greater than 99.999% after accounting for a very slight increase of 503 errors for the month on a very small number of tenants. We expect an upcoming rollout of Elasticsearch 1.4 will help us eliminate these errors, and until then we are capping our uptime at five nines for the month.
During December we did launch our first Elasticsearch 1.4 clusters into our AWS US East region. All new clusters in that region have been provisioned on Elasticsearch 1.4 to overall positive reception.
Development for Elasticsearch 1.4 had focused specifically on resiliency, and so brings a number of performance and availability improvements. We&#8217;ll be announcing plans for broader upgrades to Elasticsearch 1.4 within the next few weeks.
For further information on how we think about and report our availability, see our article on <a href="/blog/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider</a>.</div>

December 2014 Uptime: 99.999%

<div>Regional cluster upgrades, maintenance and rebalancing saw some brief windows of slightly elevated error rates in November. While our external polling reports 100% uptime, we calculate closer to 99.998% for the month.
For further reading, see our thoughts on <a href="/blog/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider</a> for more information on how we think about and measure total uptime.</div>

November 2014 Uptime: 99.998%

<div>Starting Wednesday, November 12, 2014, existing clusters will be automatically upgraded to version 1.3.5. In addition, new clusters will be provisioned with version 1.4. Customers looking to upgrade their existing clusters to version 1.4 will be able to upgrade starting the following week (November 17th).
1.3.5 Highlights
<pre><code> - Bug fixes for 1.3.x
</code></pre>
1.4 Highlights
<pre><code> - Major theme of 1.4.0 is resiliency: making Elasticsearch more stable and reliable than ever before, with better memory management, improved discovery algorithms, and better detection of corrupted data.
</code></pre>
Both 1.3.5 and 1.4 help us provide customers with a more resilient, faster, and more stable service.
Any customers with questions may email us at <a href="mailto:support@bonsai.io">support@bonsai.io</a>.</div>

Elasticsearch 1.3.5 and 1.4.0 Rollout Schedule

<div>October was a quiet month, with no major maintenance or notable problems. We were one minor incident shy of a perfect month, but rather than chalk it up to a rounding error, we&#8217;re capping ourselves at 99.999% for the month.
In case you missed it, our recent thoughts on <a href="/blog/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider</a> is an interesting read on how we measure, think about and report our uptime.</div>

October 2014 Availability: 99.999%

<div>Our <a href="/blog/september-2014-uptime">uptime in September</a>, as reported by external polling, was 100%. But sometimes we like to take a stricter view for the sake of self-improvement, and for September, that number might more accurately be 99.98%. Here&#8217;s a story of how we think about availability in general.
We leverage a number of tools to measure our total system uptime, one of which is an external polling system. This mechanism exercises all of the layers in our technology stack, and most realistically emulates real customer traffic.
Over time, our system has become increasingly insulated and isolated on a per-customer basis. This is a good thing, because it ensures that the problems of one tenant don&#8217;t become the problems of many, a core concern of managing a multi-tenant cloud hosting service.
However, this starts to become something of a problem when the primary system of measuring uptime is external polling. External polling suffers from the fact that it is coarse-grained, and so will only detect outages that affect broad segments of customers for longer than its polling interval.
Recognizing this, we have built much more fine-grained measurement and analytics tools for our own internal usage, as any good server operator must. Our systems can track response times and error rates for all traffic on a global, regional, and per-customer basis.
We refer to these analytics constantly to ensure that all of our customers&#8217; clusters are healthy, and to help identify proactive maintenance before it becomes a problem.
The trick for uptime reporting, though, is simplifying these data into an answer for a deceptively simple question, &#8220;Is the system available?&#8221;
The coarseness of a failed external health check makes for a fairly strong signal of &#8220;no.&#8221; And so we definitely take advantage of these measurements, treating them as an upper bound on our availability for a given month.
As your measurements become more and more fine-grained, the judgement call of what constitutes &#8220;downtime&#8221; becomes a little more subjective. It starts with the underlying root cause. Is a &#8220;404 Not Found&#8221; response due to legitimate customer usage, or is a routing proxy working with an outdated routing table? Is a &#8220;504 Gateway Timeout&#8221; the result of performance problems, or an unusually large document hitting a timeout? Does temporary slowness count a downtime? Is the problem affecting a single tenant, a certain segment of tenants, or some percentage of all traffic in a region?
One way that we simplify this analysis is to be strict. Our current policy internally is that if a tenant is experiencing degraded performance or increased server-side errors for a period of time, then we count that tenant as down for that period of time. These are the kinds of incidents that get reported in our <a href="http://status.bonsai.io/history" target="_blank" rel="nofollow">Status History</a>, each of which receives an internal post-mortem for the sake of this kind of analysis.
For a hypothetical example, say that some regression causes service degradation for 10% of our customers for 15 minutes. That&#8217;s a very serious issue for a hosting provider such as ourselves, which we would want to accurately represent in our uptime reports. For a thirty-day month (720 hours), we would measure that as 10% × 0.25h / 720h = 0.003%, or a lower bound of 99.996% availability for the month.
It may not seem like much, but it adds up, and it helps us stay focused on providing the best hosted Elasticsearch service available.
So while our external polling might report 100% uptime for September, and while strictly speaking everything was available during the month, we have somewhat higher standards, and count this month as 99.98%.</div>

Measuring Availability as an Elasticsearch Hosting Provider

Our uptime in September, as reported by external polling, was 100%. But sometimes we like to take a stricter view for the sake of self-improvement, and for September, that number might more accurately be 99.98%. If you&#8217;re interested in how we come up with that figure, see our article on <a href="/blog/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider</a>.

September 2014 Uptime: 99.98%

<div>Many of our customers have asked for <a href="http://www.elasticsearch.org/overview/kibana/" target="_blank" rel="nofollow">Kibana</a>, the popular analytics package which is now officially part of Elasticsearch&#8217;s suite of tools. We&#8217;ve answered the call and are proud to announce support for Kibana. As of September 22nd, 2014 all Bonsai Hosted Elasticsearch clusters (Heroku and direct signups) can access Kibana from their respective dashboards.
For Heroku customers, access to Kibana can be found here:
<a href="https://app.bonsai.io/" target="_blank" rel="nofollow"></a>
For direct signup customers, access to Kibana can be found here:
<a href="https://heroku.com/" target="_blank" rel="nofollow"></a>
Kibana is a very powerful analytics package that provides direct search and graphing for your own data. It&#8217;s eminently extensible allowing for the creation of custom dashboards. There are some default dashboards that cover many of the most common needs as well.
For more information on Kibana, please visit: <a href="http://www.elasticsearch.org/overview/kibana/" target="_blank" rel="nofollow">http://www.elasticsearch.org/overview/kibana/</a>.</div>

Announcing Kibana!

During August 2014 we performed widespread upgrades to Elasticsearch 1.3 while maintaining 100% uptime in all regions.

August 2014 Availability: 100%

July 2014 Availability: 99.98%

<div>Starting next week, we will be upgrading our customers to Elasticsearch 1.3. This upgrade brings a slew of improvements to Elasticsearch, a new scripting engine, and some important improvements to our own underlying architecture.
<ul>
<li>Monday, 11 August 2014 — Starter and Staging clusters.</li>
<li>Monday, 18 August 2014 — Production clusters and up.</li>
</ul>
We waited to upgrade to Elasticsearch 1.3 directly, skipping 1.1 and 1.2, in part due to the changes made to dynamic scripting support. Notably the switch to using Groovy as the default scripting engine. The Elasticsearch blog has more details about the changes and improvements to scripting in their recent post, <a href="http://www.elasticsearch.org/blog/scripting/" target="_blank" rel="nofollow">All About Scripting</a>. Customers making use of dynamic scripting should take particular care to verify application compatibility when Starter and Staging clusters are updated next week.
While the upgrade to 1.3 should be backward-compatible for the majority of use cases, it&#8217;s always a good idea to test your applications. We recommend that our customers update Elasticsearch in their development environments, as well as test against a Starter or Staging cluster next week to verify their application&#8217;s compatibility. Particularly if your app uses dynamic scripting.
Any customers with concerns about compatibility may email us at <a href="mailto:support@bonsai.io">support@bonsai.io</a> for an extension on their migration. We also have an option to host clusters on a single-tenant architecture for anyone who needs long-term version support.
In addition to the many new features and improvements to performance and resiliency in Elasticsearch itself, this upcoming migration include some major improvements in our own underlying architecture. We&#8217;ll write about these improvements in more detail starting next week as they start becoming accessible.</div>

Scheduled Upgrade to Elasticsearch 1.3

June 2014 Availability

<div>For every quarter we report on the uptime for production-grade clusters on our shared tier. Interested in how we calculate uptime? See our in-depth breakdown here:
<ul>
<li><a href="/2014/10/02/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider </a></li>
</ul></div>

2016 Q3 Uptime Report

<div>Getting started with Bonsai Elasticsearch and Ruby on Rails is incredibly quick and simple to do. [Sign Up]({{ site.bonsai_app_url }}/signup) for your free account we&#8217;ll take it from there!
<h3>Creating Your First Bonsai Cluster</h3>
Once you&#8217;ve signed up for your free account on Bonsai, you&#8217;ll be prompted to create a free developer cluster. Give your cluster a name and pick a region. The name helps you to distinguish between different applications and the region is how we manage our infrastructure on Amazon Web Services. Click on the &#8220;Create Cluster&#8221; button and voila, you&#8217;ve created your very first Bonsai cluster!
<h3>Our Example Rails App</h3>
Our example Rails app will provide you a good outline to follow along. You can access the Github repo here:
<ul>
<li><a href="https://github.com/omc/bonsai-rails-example" target="_blank" rel="nofollow">https://github.com/omc/bonsai-rails-example</a></li>
</ul>
In our example app, Bookfaces, we&#8217;ll have a list of books that are searchable via Book Title, Author, Publisher and Genre. You can see the app in action here:
<ul>
<li><a href="https://agile-anchorage-88938.herokuapp.com/" target="_blank" rel="nofollow">https://agile-anchorage-88938.herokuapp.com/</a></li>
</ul>
So now that we have our example code (or a Rails project of your own) let&#8217;s go ahead and get things ready for Bonsai.
###Connecting Your Rails App to Your Bonsai Cluster
Bonsai clusters are created with unique URLs designed for secure, authenticated access to the Elasticsearch API. To avoid security concerns we strongly recommend adding your Bonsai Elasticsearch URL to your environment as a variable, you can do so by running:
<pre><code> export BONSAI_URL="&lt;YOUR BONSAI URL&gt;"
</code></pre>
You can learn more about your Bonsai URL here:
<ul>
<li><a href="https://docs.bonsai.io/docs/managing-your-cluster-url-and-credentials" target="_blank" rel="nofollow">https://docs.bonsai.io/docs/managing-your-cluster-url-and-credentials</a></li>
</ul>
Now let&#8217;s add the following gems to your Gemfile:
<pre><code class="language-ruby"> gem 'elasticsearch-rails'
 gem 'elasticsearch-model'
 gem 'bonsai-elasticsearch-rails'
</code></pre>
Run <code>bundle install</code> and we&#8217;ll move on to the next step.
###Deploying to Heroku
As with all things related to the development process, deploy early and often. It’s a good idea to double check that you have completed all the steps above. For our example app we used Heroku to deploy. For step by step instructions as to how you can deploy your rails app with Heroku, please consult their excellent documentation:
<ul>
<li><a href="https://devcenter.heroku.com/articles/getting-started-with-rails4" target="_blank" rel="nofollow">https://devcenter.heroku.com/articles/getting-started-with-rails4</a></li>
</ul>
Once you have deployed to Heroku you can add the Bonsai add-on from the Overview tab.
###Inserting Data
Our next step will allow us to add data to the cluster as well as how to do a local database import. Let’s start with populating our local app first. For convenience sake, we will use the Faker gem to help populate db. Here’s a link to the Faker gem documentation:
<ul>
<li><a href="https://github.com/stympy/faker" target="_blank" rel="nofollow">https://github.com/stympy/faker</a></li>
</ul>
Let’s add the Faker gem to the gemfile:
<pre><code class="language-ruby"> gem 'faker'
</code></pre>
and run our <code>bundle install</code>. Now that we’ve got the Faker gem installed, we can start adding some sample data. Take a look at our <code>schema.rb</code> and <code>seeds.rb</code>. Files in our sample code if you need some guidance in how to structure those files.
Run the following commands to set up your local db, note that in Rails 5 we can now use the rails command instead of rake.
<pre><code class="language-ruby"> rails db:create
 rails db:migrate
 rails db:seed
</code></pre>
Now run <code>rails s</code> and check out all the new books for Bookfaces! Now that we know how to populate our local db, let’s proceed to getting our models ready for indexation.
###Get Your App Ready to Index Models You&#8217;d Like to be Searchable
We&#8217;ll want to prepare our models to be indexed and searchable by Bonsai. Take a look at <code>app/models/book.rb</code>, where we include our <code>Elasticsearch::Model</code>
<pre><code class="language-ruby"> class Book &lt; ApplicationRecord
 include Elasticsearch::Model

 belongs_to :author
 belongs_to :publisher
 belongs_to :genre

 ### View the entirety in the GitHub repo: https://github.com/omc/bonsai-rails-example/blob/master/app/models/book.rb
 end
</code></pre>
We&#8217;ll want to do this for all models that we want indexed. It&#8217;s also advisable that we create rake tasks to import documents, it will allow you to index searchable records via rake tasks. Plus, it’s important to to have reliable and easily executable indexing scripts should you need to need to duplicate this process or have a backup at hand. You can do this by creating a file <code>lib/tasks/elasticsearch.rake</code> and including:
<pre><code class="language-ruby"> require 'elasticsearch/rails/tasks/import'
</code></pre>
Keep in mind you will need to explicitly create your index before you can put documents there, since Bonsai does not support &#8220;lazy&#8221; index creation. The best way to create your index is using the rake task we set up in the prior step. The first time you index your content, you&#8217;ll want to set the FORCE parameter:
<pre><code class="language-ruby"> rake environment elasticsearch:import:all FORCE=y
</code></pre>
The FORCE parameter will explicitly delete and recreate indices, so you probably don&#8217;t want to use it frequently. Subsequent indexing tasks can drop this parameter.
The alternative way to do it would be to create your indices with curl. So in our Book model, use the following command to create the index:
<pre><code class="language-shell"> curl -XPUT http://username:password@redwood-12345.us-east-1.bonsai.io/books
</code></pre>
###Done!
You have created your Bonsai cluster, added your cluster URL to your App, indexed the models and now can deploy to the service of your choice! For our example we used Heroku, where you can install the <a href="https://elements.heroku.com/addons/bonsai" target="_blank" rel="nofollow">Bonsai Add-on</a>.
Please feel free to give us feedback or ask questions at <a href="mailto:info@omc.io">info@omc.io</a> or on Twitter [@{{ site.author.twitter }}](https://twitter.com/{{ site.author.twitter }}).</div>

Rails 5 with Bonsai

As your operations team, we at Bonsai take uptime—and reporting it—seriously. We&#8217;ve had another great quarter, and look forward to reporting even better numbers throughout the rest of 2016.

2016 Q2 Uptime Report

<div>We’re excited to announce that we now support ES 2.2.2 in addition to 1.7.5 and 1.5. We’ve spent quite a bit of time improving our infrastructure to ensure 2.x will be a stable, blazing-fast build. Two weeks ago we shipped a button that allows anyone to upgrade their cluster to 2.2.2:
Because of backwards incompatibility with 2.x, clusters on 1.7 won’t be able to upgrade if they have data on them. If you have active shards, you will see a warning message and your button will be disabled:
So, make sure to clear your cluster! You can do this by running <code>curl -XDELETE "BONSAI_URL"</code>
Elasticsearch v5 should be ready by the end of the year, and we don&#8217;t expect any updates to 1.x after this release. In early January 2017 we&#8217;ll phase out support for 1.x on non-dedicated plans, so we recommend all of our users to start their process of moving off of 1.x and into 2.x.</div>

Now Supporting 2.x!

<div>As your operations team, we at Bonsai take uptime (and reporting it) seriously. We&#8217;ve had a fantastic first quarter and look forward to reporting even better numbers in the coming quarters.
If you&#8217;re interested, you can read more about how we measure uptime here: <a href="/2014/10/02/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as a Service Provider</a>. We&#8217;ve added even more high-precision monitoring tools to our stack since that post was written, which we plan to write about in upcoming posts.</div>

2016 Q1 Uptime Report

<div>###Access
You can find the logs for your cluster under the Logs tab in your cluster dashboard. Once you click on the tab, logs will populate in real time as requests are made to your cluster.
Each row in the table is one request made to your cluster. You will see the timestamp of the request, the type of request [GET, POST, PUT, UPDATE, DELETE], the URL of the request, the time in milliseconds it took for the request to complete, and the status.
####Color Indicators
The time in milliseconds and status have color indicators that let you know how long it took for the request to complete, and if it was successful.

The top 20 slow requests are shown on a separate tab within the Logs menu, and shows requests that take longer than 200ms. Any yellow or red requests will appear on this tab.
This feature is still in beta, so we would appreciate any feedback you have. Hit us up at <a href="mailto:support@bonsai.io">support@bonsai.io</a> or you can ping us on <a href="https://twitter.com/bonsaisearch" target="_blank" rel="nofollow">Twitter @bonsaisearch</a>.</div>

Introducing: Live Streaming Logs

<div>Whether you&#8217;re a small team of two or part of a large organization, there&#8217;s a good chance you will need to delegate responsibilities of managing your cluster at some point. Even if you just want to include someone else on email communication, or if you want to stop our emails going out to everyone on the group email you signed up with (gasp!), we can help.
##Heroku Contacts
For our Heroku add-on users, we built Heroku Contacts. Being a contact means that you will receive notifications from us about your cluster usage, and overage warnings if you&#8217;re going over your plan limits. You can manage your contacts in the Bonsai Dashboard, which you can access through Heroku, or by typing this in your terminal:
<pre><code>heroku addons:open bonsai -a &lt;your-app-name-here&gt;
</code></pre>
###Contacts page in the Bonsai Dashboard:
Your Heroku app owner email is automatically set up as a contact, as well as the email address of anyone who SSO&#8217;s into the dashboard. If your Heroku owner email is a group email, you will likely want to remove it as a contact and replace it with another email address so we don&#8217;t spam your whole company!
##Cluster Collaborators
For our direct sign-up users, we built Cluster Collaborators. A collaborator is able to login to the Bonsai Dashboard with their own user account, which means no more sharing passwords! To add a collaborator, click on the Collaborators tab from your Bonsai Dashboard.
###Collaborators page in the Bonsai Dashboard:
You have the option of adding either a regular collaborator or an administrator. A collaborator will receive notifications from us about the cluster, be able to view API credentials, and view other collaborators. An administrator has the additional ability to regenerate credentials, add and remove other collaborators, update the cluster plan, and deprovision the cluster.
Once you invite a collaborator, they will be emailed a link to sign up as a user for Bonsai, where they can choose to accept or decline your invitation.
After clicking &#8216;Accept&#8217;, they will be able to see the cluster on their cluster dashboard. A cluster that you are a collaborator on is shown with a collaborator symbol in the upper right hand corner.
##Go Forth, and Collaborate!
We loved building these tools, and we hope you enjoy using them! If you have any suggestions for improvements or you have a business need for something else, hit us up at support@bonsai.io or you can ping us on <a href="https://twitter.com/bonsaisearch" target="_blank" rel="nofollow">Twitter @bonsaisearch</a>.</div>

Teamwork Makes the Dream Work

<div>Sometimes we’re asked if we support Logstash. The relationships between Elasticsearch, Logstash and Kibana (often referred to as the ELK stack) can sometimes foster a confusing mental model, and a simple &#8220;yes&#8221; from our support staff probably isn’t enough to help users get up and running quickly. In this post, we’d like to unveil some of the mystery behind why the ELK stack is so popular.
We whipped up a quick example showing how to set up and use Logstash with Bonsai Elasticsearch and a Kibana 4 instance. This example takes a real world use case and walks the reader through the process of setting up Logstash to pass a particular type of event to Bonsai for indexing, where it can then be analyzed in Kibana. The intent is to help new users get up to speed quickly, without needing to do a lot of Googling for answers.
###So what is Logstash?
Logstash is an awesome tool for log parsing. Logstash can accept inputs from a number of different sources, apply a series of filters to the incoming data, then pass the results to a data store. In combination with Elasticsearch (for data storage, aggregation and retrieval) and Kibana (for analysis and visualization), users can get a very nice high-level view of a data set. It can do many other things, but that is its primary function.
##A real world example
In this example, we&#8217;ll look at the SSH daemon running on an Ubuntu server. SSH logs, particularly failed login attempts, can be extremely useful in investigating break-in attempts, identifying unusual traffic patterns, and performing regular security audits. It&#8217;s part of a proactive security policy, and critical if you&#8217;re a sysadmin in charge of dozens or hundreds of servers.
We could use other input contexts, but since the ‘net is rife with automated attack traffic, we think security is particularly interesting. If you have a machine connected to the Internet, you will eventually be hit by someone. Hackers use bots to scan the IPv4 space looking for particular ports &#8212; 21 for FTP, 22 for SSH, 23 for Telnet, and so on — and when a server responds to a request on a given port, the bot will try everything it can to log in to the system.
Generally, the bot doesn’t know anything about the target, so it just tries a bunch of random usernames and passwords. By studying this behavior, we can perform some counterintelligence gathering on the hacker community and use the ELK stack to aid in our reconnaissance. For another fun read, see how the developer of Elastichoney uses a <a href="http://jordan-wright.com/blog/2015/05/11/60-days-of-watching-hackers-attack-elasticsearch/" target="_blank" rel="nofollow">similar approach</a> to learn how hackers try to exploit Groovy scripting vulnerabilities in Elasticsearch.
###The Game Plan
We&#8217;ll pass the system logs into Logstash, run some filters to make sure we&#8217;re only seeing what we want, send the results along to Bonsai, and finally, we&#8217;ll examine the data using Kibana.
For the purposes of this example, we&#8217;ll make a few assumptions:
<ol>
<li>
SSH activity is being logged to <code>/var/log/auth.log</code>. This is the default setting for Amazon&#8217;s Ubuntu images, which have not (yet) switched over to systemd. Although you can use systemd with Logstash if you like.
</li>
<li>
You have the ability to install Logstash on the server(s), and have already done so. There are ways to do it with a package manager, and Elastic offers Logstash as a .deb on their downloads page, so it can be pretty easily installed. Read their documentation if you need more help getting it installed wherever you&#8217;re trying to use it.
In preparing this example, I downloaded the .deb to an Ubuntu machine, so the instructions are based on that. If you use a package manager or another OS, some files might end up in other places and some commands may need minor tweaks.
</li>
<li>
You already have a Bonsai Elasticsearch cluster provisioned. In this example, the URL of that cluster will be <code>http://user123:pass456@logstash-12345.us-east-1.bonsai.io</code>
</li>
<li>
You have a Kibana 4 instance set up somewhere. Need more info on this? <a href="http://docs.bonsai.io/docs/using-kibana-with-bonsai" target="_blank" rel="nofollow">We&#8217;ve got you covered</a>.
</li>
</ol>
###Laying the groundwork
Logstash can accept streams from a large number of inputs. They also have an ever-expanding list of input plugins to support more streams and file types. One requirement that all inputs have in common is permissions; that is, Logstash must have permission to access the stream in order to read from it. This probably sounds trivial, but it is a pretty common &#8220;gotcha,&#8221; especially when dealing with daemon log files.
When Logstash is installed in Debian, the OS creates a user called <code>logstash</code>. This is the user that will need to be able to access the log file. We can see the permissions of the auth.log file with <code>ls -lAh /var/log/auth.log</code>:
<pre><code>-rw-r----- 1 syslog adm 6.4M Jan 20 17:24 /var/log/auth.log
</code></pre>
We don&#8217;t want to change these permissions, because that&#8217;s not a best practice and could be dangerous. But what we can do is add the <code>logstash</code> user to the <code>adm</code> group. The <code>adm</code> group typically grants members access to read log files and nothing else. Joining this group would give the Logstash process the ability to read the file, but not modify it in any way. This is the minimum necessary access to do the job, and it doesn&#8217;t expose security details to other programs/users on the server, so it&#8217;s the route we&#8217;ll take:
<pre><code>$ sudo usermod -G adm logstash
</code></pre>
The last thing we need to do is create an index on our Bonsai cluster for Logstash to send data. We&#8217;ll call this index <code>logstash</code>:
<pre><code>$ curl -XPUT http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash
</code></pre>
###Configuring Logstash
Now that we know Logstash will be able to read the input stream, we&#8217;ll need to configure it. Go ahead and edit <code>/etc/logstash/conf.d/logstash.conf</code>:
<pre><code>$ sudo nano /etc/logstash/conf.d/logstash.conf
</code></pre>
Copy/paste the settings below (making sure to substitute the correct values for your cluster URL):
<pre><code>input {
 file {
 path =&gt; "/var/log/auth.log"
 start_position =&gt; beginning
 sincedb_path =&gt; "/dev/null"
 }
}

output {
 elasticsearch {
 hosts =&gt; ["logstash-12345.us-east-1.bonsai.io:443"]
 user =&gt; "user123"
 password =&gt; "pass456"
 ssl =&gt; true
 index =&gt; "logstash"
 }
}
</code></pre>
Finally, make sure that Logstash has permission to read its own config file:
<pre><code>$ sudo chown logstash /etc/logstash/conf.d/logstash.conf
</code></pre>
###Test it out!
We&#8217;ve just completed the bare minimum necessary to get Logstash up and running. We&#8217;ve given Logstash access to a log file, defined it as an input stream, created a place in Elasticsearch for the stream to go, and defined that index as an output. Now all we need to do is run it and make sure it works. When I installed the .deb for Logstash, the one thing it didn&#8217;t do was symlink the binary, so I&#8217;ll just use the full path. YMMV:
<pre><code>$ /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/logstash.conf
Logstash startup completed
</code></pre>
Now, if everything worked as expected, I should be able to see the data in my cluster:
<pre><code>$ curl "http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash/_search?pretty&amp;size=1"
{
 "took" : 1,
 "timed_out" : false,
 "_shards" : {
 "total" : 1,
 "successful" : 1,
 "failed" : 0
 },
 "hits" : {
 "total" : 1975,
 "max_score" : 1.0,
 "hits" : [ {
 "_index" : "logstash",
 "_type" : "logs",
 "_id" : "AVJdzwbLfbt0kfwq_Tzs",
 "_score" : 1.0,
 "_source":{"message":"Jan 17 04:30:52 ip-XXX-XXX-XXX-XXX sshd[12170]: Invalid user cms from XXX.XXX.XXX.XXX","@version":"1","@timestamp":"2016-01-17T11:30:52.000Z","host":"localhost","path":"/var/log/auth.log"}
 } ]
 }
}
</code></pre>
Neat! Let&#8217;s look at this in Kibana:
So, a partial success. We can see data in the index, and it is coming out of the auth.log file as expected. But the timestamp associated to the event is not when it occurred in real time, but when it was indexed. Not very helpful.
This is a pretty common gotcha as well. I included it here to underscore it as a pitfall (and maybe save everyone a few support tickets). It&#8217;s easily addressed, we just need to do a bit more work.
##A more complex setup
Let&#8217;s go ahead and delete the logstash index on our cluster:
<pre><code>$ curl -XDELETE http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash
</code></pre>
With that out of the way, we can focus on refining our Logstash configuration. We&#8217;ll use some of the more popular filters to extract the most important details, and add those to fields that we can later query and display with Kibana.
Before we do that, let&#8217;s think about what our goal is here. We want to look for failed login attempts over SSH; we&#8217;re not interested in <code>cron</code> tasks or <code>sudo</code> sessions, or anything else that commonly gets logged to auth.log. Let&#8217;s take a look at a sample line:
<pre><code>Jan 17 04:30:52 ip-XXX-XXX-XXX-XXX sshd[12170]: Invalid user cms from XXX.XXX.XXX.XXX
</code></pre>
What kind of data do we have there? I see a date (in a non-standard format, boo!), a private IP address (useful for identifying what server is being hit), as well as a probe at a user name (&#8220;cms,&#8221; really?) and an IP address from where the attempt originated.
So we want to disregard lines that don&#8217;t match this pattern, and extract some useful information from lines that do match the pattern. We&#8217;ll also have to do something about that date &#8212; it will need to be in a standard format to be useful in Kibana. It would also be cool if we could perform some further analysis on the originating IP address, to see if there are any interesting trends. Who&#8217;s hitting us, and where are they coming from, would be good to know.
Fortunately, Logstash has filters that allow us to do all of this:
<ul>
<li>
Grok. The <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html" target="_blank" rel="nofollow">grok filter</a> allows you to use regular expressions to parse log entries. It has a number of built-in patterns, and allows you to arbitrarily define your own. This is how we&#8217;ll extract the lines of interest.
</li>
<li>
Date. The <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html" target="_blank" rel="nofollow">date filter</a> allows you to extract timestamps from your logs, then using those as the Logstash timestamp for the event. This will fix that &#8220;single bar&#8221; problem in Kibana.
</li>
<li>
GeoIP. The <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html" target="_blank" rel="nofollow">geoip filter</a> allows you to take an IP address, run it through the GeoLiteCity database, and save geographic information to a new field. We can use this info in Kibana to make a heat map of where the nefarious traffic is originating.
</li>
</ul>
Update the Logstash configuration file to this, again changing the Elasticsearch output to match your Bonsai URL :
<pre><code>input {
 file {
 path =&gt; "/var/log/auth.log"
 start_position =&gt; beginning
 }
}

filter {
 grok {
 match =&gt; { "message" =&gt; "(?&lt;logdate&gt;%{MONTH} %{HOUR}.%{MINUTE}.%{SECOND}) (?&lt;private_ip&gt;ip-[0-9]{1,3}-[0-9]{1,3}-[0-9]{1,3}-[0-9]{1,3}) sshd\[[0-9]{1,}\]: (?&lt;sshd_message&gt;Invalid user (?&lt;invalid_user&gt;[a-z0-9]{1,}) from (?&lt;invalid_ip&gt;%{IP}))"}
 }
 if "_grokparsefailure" in [tags] {
 drop { }
 }
 date {
 match =&gt; [ "logdate", "MMM dd HH:mm:ss" ]
 }
 geoip {
 source =&gt; "invalid_ip"
 target =&gt; "geoip"
 database =&gt; "/etc/logstash/conf.d/GeoLiteCity.dat"
 }
}

output {
 elasticsearch {
 hosts =&gt; ["logstash-12345.us-east-1.bonsai.io:443"]
 user =&gt; "user123"
 password =&gt; "pass456"
 ssl =&gt; true
 index =&gt; "logstash"
 }
}
</code></pre>
We&#8217;ll need to make sure that the GeoLiteCity.dat file is in its proper place:
<pre><code>$ cd /etc/logstash/conf.d/
$ sudo curl -O "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz"
$ sudo gunzip GeoLiteCity.dat.gz
$ sudo chown logstash GeoLiteCity.dat
</code></pre>
Finally, we need to make one little tweak to our index before running Logstash. As of this writing, there is a bug where Elasticsearch will dynamically map the <code>geoip.location</code> as a <code>float</code> instead of a <code>geo_point</code>, which makes it impossible for Kibana to create a tile map with the information. We just need to explicitly set this field up first. We&#8217;ll do that when we recreate the index:
<pre><code>$ curl -XPUT http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash -d '{"mappings":{"logs":{"properties":{"geoip":{"properties":{"location":{"type":"geo_point"}}}}}}}'
</code></pre>
Let&#8217;s spin up Logstash again and see what happens:
<pre><code>$ logstash agent -f /etc/logstash.conf
Settings: Default filter workers: 2
Logstash startup completed
</code></pre>
If we check on the data again, we see:
<pre><code>curl "http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash/_search?pretty&amp;size=1"
{
 "took" : 1,
 "timed_out" : false,
 "_shards" : {
 "total" : 1,
 "successful" : 1,
 "failed" : 0
 },
 "hits" : {
 "total" : 144,
 "max_score" : 1.0,
 "hits" : [ {
 "_index" : "logstash",
 "_type" : "logs",
 "_id" : "AVJggJNEfbt0kfwq_U0P",
 "_score" : 1.0,
 "_source":{"message":"Jan 17 04:30:52 ip-XXX-XXX-XXX-XXX sshd[12170]: Invalid user cms from XXX.XXX.XXX.XXX","@version":"1","@timestamp":"2016-01-17T11:30:52.000Z","host":"localhost","path":"/var/log/auth.log","logdate":"Jan 17 04:30:52","private_ip":"ip-XXX-XXX-XXX-XXX","sshd_message":"Invalid user cms from XXX.XXX.XXX.XXX","invalid_user":"cms","invalid_ip":"XXX.XXX.XXX.XXX","geoip":{"ip":"XXX.XXX.XXX.XXX","country_code2":"CN","country_code3":"CHN","country_name":"China","continent_code":"AS","region_name":"12","city_name":"Wuhan","latitude":30.580099999999987,"longitude":114.27339999999998,"timezone":"Asia/Shanghai","real_region_name":"Hubei","location":[114.27339999999998,30.580099999999987]}}
 } ]
 }
}
</code></pre>
Nice! So our filters worked. We&#8217;ve dropped from almost 2K lines to just 144. That&#8217;s a big reduction in noise. Let&#8217;s look in Kibana:
Great, so the timestamps are correct as well. We can see a big spike in activity at one point. From a security standpoint, that would be something to investigate. Ideally, it would also be something we could wire up to an alarm and page a sysadmin to let them know there might be an intrusion attempt.
But let&#8217;s take it further. We can create a whole dashboard around this. We can create a Tile Map using the <code>geoip.location</code> field and see where these requests are originating. We can also look at stats like the user account that was probed; this might help us to determine if these attacks are some random bot or script kiddie, or if there is a more sophisticated adversary targeting specific systems.
I&#8217;m sure you could think of even more ways to use this data. For example, imagine that the source data is coming from multiple servers in your network. You could see which ones are being hit hardest and take corrective action. You could set up an alert that notifies you if someone logs in from an IP address overseas &#8212; perhaps they&#8217;ve been compromised. There is a wealth of intelligence that can be gleaned from looking at a setup like this.
###Wrapping up
People use Logstash for all sorts of things. It really is an amazing tool, and we wish it had a larger following. If you&#8217;re using Logstash and want to host your Elasticsearch cluster with Bonsai, we&#8217;d love to work with you. We fully support both Logstash and Kibana, so&#8230;
##Give us a shout!
Got a cool use case you&#8217;d like to share? Maybe an anecdote or useful tip? Did you find this article tedious and boring? We&#8217;d like to hear about that too. Hit us up at support@bonsai.io and share. You can also ping us at <a href="https://twitter.com/bonsaisearch" target="_blank" rel="nofollow">@bonsaisearch</a> on Twitter if that&#8217;s something you&#8217;re in to.</div>

Logstash and Bonsai and bots, oh my!

<div>A t the core of Elasticsearch’s popularity is the ease and simplicity of setting up a cluster. And with hosted search options like <a href="https://bonsai.io" target="_blank" rel="nofollow">Bonsai.io</a>, powerful, multi-node clusters can be created instantly.
In a lot of ways, ease of use is both a blessing and a curse. When the learning curve isn’t a barrier to entry, it’s easy to start on a path that causes problems later. Many users find that decisions they made early in the product cycle regarding Elasticsearch don’t scale well in production. Some early decisions can become extremely painful or difficult to change later.
At Bonsai, we’ve had the opportunity to work directly with thousands of users around the world. We’ve had the unique experience of getting a peek into projects integrating Elasticsearch at scale, and helping people get the absolute most out of search. This experience has been instrumental in crafting some guidelines for users just starting out.
This is the guide a lot of users wished they’d had in the beginning.
<h3>Why Focus on the Index? Why Not the Cluster?</h3>
The short answer is that you should let your data decide the cluster specifications, not vice-versa. The index will determine how the data is stored and retrieved. If you haven’t thought through what your indices will look like and how they will be configured, you may end up underprovisioning the cluster (hurting performance) or overprovisioning the cluster (hurting your wallet).
This guide will help you understand and reason about the factors that will impact the quality of your search experience over the long term.
<h3>Elasticsearch Has So Many Use Cases, How Relevant is this Advice to Me?</h3>
People use Elasticsearch in so many ways. And each use case is going to benefit from a different set of settings and low level tweaks. However, these are typically changes that can be made at any time, in contrast to immutable settings like primary sharding. In other words, the fine tuning can happen a little later.
Instead, we’re going to focus on a couple of high level characteristics shared by all well-configured indices, and leave the more esoteric stuff for a later series.
<h2>That said, the Ideal Elasticsearch Index is…</h2>
<ol>
<li>Backed by a database</li>
<li>Sharded intelligently</li>
<li>Replicated, but not overreplicated</li>
<li>Has well thought out data structures</li>
</ol>
Let’s look into those in a bit more detail.
<h3>Backing Elasticsearch with a Database</h3>
Most developers have at least a casual understanding of what a database is: some software that stores data in a structured format, and allows applications to interact with that data using a structured query language. And Elasticsearch is basically the same thing, right?
While that is an accurate description of what Elasticsearch does, the devil is in the details. Database software aims for something called ACID compliance. ACID (Atomicity, Consistency, Isolation, Durability) is a set of properties that guarantee validity of data in the event of unexpected service interruptions.
An “ACID-compliant data store” is a fancy way of saying that the technology is highly resistant to corruption and data loss. Some examples of ACID-compliant data stores you may be familiar with: Postgres, SQLite, Oracle, MySQL (with InnoDB), and MongoDB (4.0+). Not listed: Elasticsearch.
Elasticsearch uses a search engine technology called <a href="https://lucene.apache.org/" target="_blank" rel="nofollow">Lucene</a>. Lucene is an information retrieval technology built for speed, not redundancy. It has a radically different architecture that gives it blazing fast performance, at the expense of being more susceptible to data loss.
Data loss can happen in a number of ways, you need to be able to recreate the data if needed. True, Elasticsearch has a snapshot/restore feature, but this process will only ever partially recover you in the event of data loss. Updates made between the most recent snapshot and outage will be lost unless you have another system in place to queue them. Snapshot/restore will also not help in the event of split brain, because there’s no mechanism for reconciling updates to each partition. Updates will just be lost.
This is why every Elasticsearch index you create should be backed by an ACID-compliant data store. The database should be the ultimate source of truth, from which the index is populated. If you ever lose the index — via node loss, outage, or simple fat-finger — you’ll be able to completely recover it.
<h3>Intelligent Sharding</h3>
Are you brand new to Elasticsearch? Start by reading <a href="https://docs.bonsai.io/docs/what-are-shards-and-replicas" target="_blank" rel="nofollow">this article on Elasticsearch shards and replicas</a>.
Every Elasticsearch index is composed of one or more shards. There are two types of shards: primary, and replica. Primary shards are responsible for managing updates to the index, while replica shards are simply copies on the primary and live on other nodes. The idea is that if a primary shard is taken offline, the replica will be able to fill the role and keep search from going down.
Shards are really just abstractions for Lucene indices. When an Elasticsearch index has several primary shards, it can be thought of having the data spread out over several different search engines. This is useful, for example, when you have millions of documents, because it allows queries to be served in parallel, which can save time and IO.
“Oh,” I hear you say, “well then I’ll just create my index with 100 primary shards and be done with it!” That certainly is one approach, but it’s a terrible one. Shards come with overhead; even empty shards require memory and disk. And <a href="https://docs.bonsai.io/docs/what-are-shards-and-replicas" target="_blank" rel="nofollow">tf-idf</a> relevance is calculated on a per-shard basis. Oh, and if you have more shards than CPU cores, they will end up fighting each other for system resources on every request.
In other words, this approach would give you the worst of all worlds unless you already have a lot of data and a lot of nodes.
Another thing to understand is that Elasticsearch uses a naive hashing algorithm to route documents to a given primary shard. This design choice allows documents to be randomly distributed in a reproducible way. This avoids “hot spots” that affect performance and overallocation. However, it has one major downside, which is that the number of primary shards can not be changed after an index has been created. Replicas can be added and removed at will, but the number of primary shards is basically written in stone.
This means that we need to have some sense of our scale before even creating the index. We need a shard scheme that makes sense for our present needs, but with enough headroom to scale up.
<h3>Benchmarking is key</h3>
Benchmarking a subset of your data before you decide on your production settings will give you a sense of how much data will be used per document, both on-disk and in memory. From there, you can use some heuristics to figure out a good launch point.
Something else to think about is the number of nodes you will be using, and what are their capabilities. A production cluster should not have fewer than three nodes, and should almost always have an odd number. With that in mind, here are some heuristics:
If you don’t expect data to grow significantly, then:
<ul>
<li>One primary shard is fine if you have less than 100K documents</li>
<li>One primary shard per node is good if you have over 100K documents</li>
<li>One primary shard per CPU core is good if you have a couple million documents</li>
</ul>
If you do anticipate significant growth, then you’ll want to balance your present needs with your expected needs. At a minimum, you’ll want a number of primary cores equal to the number of nodes you would scale to when horizontal scaling is required.
In other words, if you’re starting out on a three node cluster, create the index with at least five primary shards. If you have a large enough collection of documents, you could also go with the product of your current and future number of nodes (in this case, 3 nodes today x 5 nodes at scale-up=15 primary shards). That assumes that the nodes in your three node cluster have at least five CPU cores.
See why this is an interesting problem?
Thinking about your needs and planning a little ahead will go a long way in creating an Elasticsearch index that performs well over the long term.
<h3>Replicating shards, but not Overreplicating them</h3>
Replica shards are copies of primary shards that live on any node other than the primary. Like primary shards, replicas can serve search requests if needed. Unlike primary shards, replicas do not handle update operations; they simply receive updates from the primary at regular intervals. A nice benefit of this design is that replicas are also not subject to immutability like primary shards. You can add/remove replicas at any time.
Replicas reduce stress on primary shards, and provide protection against data loss, node loss, network partitions, etc. However, too many replicas lead to wasted resources, because shards aren’t free. The cost-benefit ratio of replication gets worse with each new replica shard.
The ideal Elasticsearch index has a replication factor of at least 1. This means for every primary shard — however many there may be — there is at least one replica. And the maximum number of replicas never exceeds (n-1), where n is the number of nodes in the cluster. (This is a common newbie mistake: creating 10 replicas on a three node cluster and then wondering why the cluster state is perpetually “yellow.” Where are all of those extra replicas supposed to go?)
How do you know how many replicas is “enough”? With a minimum of 1 and maximum of (n-1), what is the “sweet spot?” Incremental benchmarking is a good way to narrow in on that. Add replicas incrementally and measure performance improvements with each new replica. If improvements are not noticeable, replication is probably too high.
<h3>Thinking through data structures</h3>
What are “data structures,” and how are they pertinent to Elasticsearch? Elasticsearch has a concept known as <a href="https://www.tutorialspoint.com/elasticsearch/elasticsearch_mapping.htm" target="_blank" rel="nofollow">“mappings.”</a> Mappings to specify how data will be ingested and queried. Mappings define analyzer chains composed of tokenizers and filters; these allow data (and queries) to be tokenized and mutated before being used.
While a well considered mapping can improve your search results, it’s wise to start simply, benchmark, and iterate later. Tweaking mappings and then promoting them to production is easier than you think. That means you don’t have don’t index everything in your database. Don’t try to do All The Things™. Think about what features your application actually needs right now, and worry about new features later.
It’s also carefully considered: don’t use mappings that you don’t fully understand. Don’t just copy/paste from the Internet. Read the documentation, know what each component does and how it will affect performance and system resources. If you don’t know what your mappings look like, take an hour to look at them and comprehend how they will work with your data.
The Ideal Elasticsearch Index isn’t necessarily just implementing default data structures, but has mappings that were honed in small scale testing. A subset of production data can be used to benchmark the performance and resource demands of a mapping. Figure these things out before taking it to scale.
<h4>Understanding how a mapping can affect performance</h4>
If you’re familiar with the <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/pipes-and-filters" target="_blank" rel="nofollow">Pipe and Filter Pattern</a>, then this will make a lot of sense. But just in case, let’s look at the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-standard-tokenizer.html" target="_blank" rel="nofollow">standard tokenizer</a> for a moment. This component takes a text input and breaks it up using the Unicode Text Segmentation algorithm. The short explanation is that this splits the input up on whitespace. So an input like:
<pre><code>It was the best of times, it was the BLURST of times
</code></pre>
is broken up into an array like so:
<pre><code>[It, was, the, best, of, times, it, was, the, BLURST, of, times]
</code></pre>
The elements of this array are called tokens, and they can be filters out and mutated before reaching Lucene. We can use stopwords filters to get rid of tokens that aren’t likely to be relevant to a search:
<pre><code>[best, times, BLURST, times]
</code></pre>
Then we can pass these tokens through a lowercase filter to get rid of the all-caps, and then through a synonyms filter, which maps <code>blurst =&gt; worst</code>. That will leave us with:
<pre><code>[best, times, worst, times]
</code></pre>
These tokens are now stored in the inverted index along with the corresponding document ID, ensuring queries will be much more relevant. This pattern gives developers a lot of flexibility, however some seemingly innocuous choices will have enormous consequences at scale.
Why? Well, mappings are easy to understand conceptually, but can take time to master in practice. Here’s a classic example: <a href="https://en.wikipedia.org/wiki/N-gram" target="_blank" rel="nofollow">n-grams</a>. This is a filter which breaks up tokens into lots of sub-tokens, with set minimum and maximum sizes. Developers often like to use n-grams for substring matching at search time, and subsequently give the tokens a size from 1 to 100 (or something obnoxiously large).
To see why this is a problem, look at what a token like <code>“terrific”</code> turns into:
<pre><code>["t", "e", "r", "r", "i", "f", "i", "c", "te", "er", "rr", "ri", "if", "fi", "ic", "ter", "err", "rri", "rif", "ifi", "fic", "terr", "erri", "rrif", "rifi", "ific", "terri", "errif", "rrifi", "rific", "terrif", "errifi", "rrific", "terrifi", "errific", "terrific"]
</code></pre>
If you work it out, you find that there is a parabolic relationship between the token length and the number of n-grams that it generates:
Worse, this pattern is generated per token. A standard 500-token document will create 500 times more grams, much of them garbage. How many words can you think of with substrings that are also, unrelated word? “Scatter” contains the totally unrelated word “cat.” If a user searches for “cat,” a document with the term “scatter” could plausibly be a match in this scenario.
And if your gram size is 1, then you’re basically indexing every single character in the alphabet to every single document. Pro-tip: this is not a good way to get relevant results or use system resources intelligently. But it’s a great way to consume lots of disk and memory for no reason.
How about another example? Synonym matching is another feature that is easy to implement, and even easier to screw up. Synonyms allow developers to create rules that group or translate related tokens. You might have a rule that replaces “ram” with “memory,” or a rule that takes the token “cat” and expand it out to the tokens: “cat”, “kitty” and “kitten.”
Elasticsearch allows developers to perform synonym expansion at index time, query time, or both. Expanding at index time means that the tokens are being expanded before being written out to disk. This does lead to larger indices, and has the downside of needing to reindex every time you want to change them.
That said, many developers opt instead for query time expansion. The user searches for a term that the synonym filter matches and expands, and those new tokens become part of the Lucene query. But then in production, the users report lots of unexpected and irrelevant results.
Why? Because multi-word synonyms break phrase queries, and the tf-idf scores for rare terms lead to score boosting in ways that are not immediately obvious. The low level details of why this happens is well beyond the scope of this post, but needless to say, you could spend weeks wrangling with relevancy tuning because of an ill-considered choice when working out the mappings.
These are just a few examples; broader coverage of the kinds of subtle mistakes and inefficiencies that are introduced in early stage data structures would be a whole other series of articles. But it’s just a taste of what can happen when choices are made without understanding the consequences and trade offs.
<h2>Conclusion</h2>
Elasticsearch and search engines are incredibly broad topics, and mastery of these subjects can be incredibly difficult. Fortunately, the fundamentals are easy to grasp and implement. The Ideal Elasticsearch Index is a set of design principles common to most use cases.</div>

The Ideal Elasticsearch Index

<div>W
e’ve had Elasticsearch 1.7 out and running in production for a few months now, and, as expected, it’s operating like a champ. This is a minor update that doesn’t change user functionality, so we don’t expect any impact. Next week we’ll begin the process of upgrading all remaining 1.5 clusters onto ES 1.7, from Tuesday through Thursday. We will be posting updates on <a href="https://twitter.com/bonsaisearch" target="_blank" rel="nofollow">twitter</a>.
###Gain Improved Versions of Features Available Through Elasticsearch 1.6
Most user-facing changes to Elasticsearch in 1.6 and 1.7 have been minor or subtle additions to existing functionality, so we’re not expecting any notable changes for clusters being upgraded.
One such example from our friends over at <a href="http://www.wework.com" target="_blank" rel="nofollow">WeWork</a>: the <code>field_value_factor</code> function now lets you specify a default value for a field that is missing on a document. A handy improvement if you’re using <code>function_score_query</code> with inconsistent document values!
<ul>
<li><a href="https://github.com/elastic/elasticsearch/pull/10845" target="_blank" rel="nofollow">elastic/elasticsearch#10845</a></li>
</ul>
Other changes to ES 1.6 and 1.7 have been focused on security and operational improvements. While these kinds of improvements are always nice to have, they weren’t critical for us. The security fixes, in particular, do not affect our systems due to our architectural design choices and other preventative measures.
So if ES 1.5 is working out fine, why risk an upgrade? Good question. These newer versions definitely are better, even if subtly. We think it’s a core part of our value as a service provider to provide continuous managed upgrades to our customers, so that you can enjoy the full benefits of open source community built software.
Imagine, if the cost:benefit ratio had us putting this upgrade on the backburner, how much longer might a busy team focused on their app wait to manage their own upgrade!
###A Smoother Transition for Future Releases: 2.0
More importantly here, Elasticsearch 2.0 is around the corner, and that upgrade will not be so trivial. We’re laying the groundwork now to help prepare for that upgrade some time in the new year, and ES 1.6 and 1.7 also help to provide interim tooling toward that end. You will receive at least a month of notification for this update so you can test your app against this particular upgrade.
###How to opt-out of 1.7
If, out of an abundance of caution, you’d like to opt-out of the upgrade, that’s fine too! We can provide long-term version support for any Elasticsearch version on one of our dedicated cluster plans. Just <a href="mailto:support@bonsai.io">contact support</a> and let us know.
<h2>Recap:</h2>
<ul>
<li>We’re upgrading everyone from 1.5 to 1.7</li>
<li>This will happen next week, Dec 22nd — Dec 24th</li>
<li>1.7 improves security and operations, so we don’t expect any user-facing impact</li>
<li>Upgrades to 2.0 are coming in the new year</li>
</ul></div>