Whether you are an Elasticsearch beginner or an expert, get helpful Bonsai.io news as well as Elasticsearch tutorials, tips, and updates from our team.

Updates from the Bonsai Elasticsearch team, from One More Cloud: the first and best cloud search platform, since 2009.


Search Snippets


Schedule a free consultation to see how we can create a customized plan to meet your search needs.


Find out how we can help you.

<div><p>March was a hard month for our availability and uptime. We suffered from two major incidents in March, between them causing multiple hours of degraded performance, and four full cluster restarts, contributing nearly 20 minutes of hard downtime each.</p>
<p>Our <a href="http://status.bonsai.io/657919/2013/03" target="_blank" rel="nofollow">March uptime</a>, as reported by Pingdom health checks, was 99.73%. That’s the second-lowest in our history, and the worst since launching into production in January.</p>
<p>We are humbled, both by the scope of responsibility of managing infrastructure, as well as the trust and support shown by our customers. Thank you for bearing with us as we redouble our efforts to build the best possible hosted Elasticsearch service for you.</p>
<p>For more information about the outage, read our previous <a href="/blog/2013/03/05/outage-post-mortem">post-mortem blog post</a> detailing our March 4th outage. We suffered a similar incident again on March 9th, but have since managed to mitigate most of the underlying causes.</p></div>

March 2013 Availability

<div><pre><code class="language-json">{
  "ok": true,
  "status": 200,
  "name": "Slick",
  "version": {
    "number": "0.20.6",
    "snapshot_build": false
  },
  "tagline": "You Know, for Search"
}
</code></pre>
<p>Yesterday, Elasticsearch announced the release of <a href="http://www.elasticsearch.org/2013/03/25/0-20-6-released/" target="_blank" rel="nofollow">version 0.20.6</a>, which includes several critical bug fixes related to Elasticsearch and Lucene. Today we have updated our clusters to this version.</p>
<p>One of these bugs in particular (<a href="https://issues.apache.org/jira/browse/LUCENE-4870" target="_blank" rel="nofollow">LUCENE-4870</a>) has the potential to cause index data loss during certain circumstances, an issue which we have been affected by in the past, early on in our beta. So we are rather gratified to see it fixed for good in Lucene.</p>
<h3>On the Road to 1.0</h3>
<p>More updates are on the way! Elasticsearch itself draws closer and closer to its 1.0, and a critical next step toward that is its upcoming release of version 0.90. This version upgrades Lucene from the 3.x series up to the latest Lucene 4.2 series, bringing substantial performance and stability improvements to Elasticsearch.</p>
<p>Upgrading to version 0.90 will require a full cluster stop and start. We plan to upgrade to 0.90 as soon as it is available, with a notice to our customers here, via Twitter and over email with a time frame for the scheduled downtime.</p>
<p>All in all, we&#8217;re excited to see Elasticsearch and Lucene moving forward with steady gains in performance, reliability, and new features. Open source at work.</p></div>

Cluster Upgraded to Elasticsearch 0.20.6

<div><p>On Monday, March 5, 2013 we experienced a substantial outage of our production
Elasticsearch cluster. We are profoundly sorry for the outage and its affect on
our customers and the users and customers of their services.</p>
<p>At approximately 12:05pm PST (UTC-0800), we began receiving alerts of high load
on one of our Elasticsearcn nodes. We noted a high level of CPU usage by
Elasticsearch on this node, coupled with a higher than normal usage of system
memory. Our logs are still unclear on the precise cause of the excess CPU usage,
but we were able to rule out other common system causes of high load, such as
high disk IO latency.</p>
<p>At this time we removed the node from our front end load balancers,
redistributed its shards throughout the cluster, and shut down the node in an
attempt to redistribute the load throughout the rest of the cluster. However,
the node was not able to successfully reconnect to the cluster, and logs on
other servers indicated that other nodes were having trouble connecting to the
cluster as well.</p>
<p>At this time, around 12:40pm PST, we decided to perform a restart of the
Elasticsearch cluster to temporarily suspend index traffic and allow nodes to
reconnect to each other. After the restart, cluster health checks were reporting
all but the first node to be online and connected. As indexes gradually reloaded
into the cluster, we noticed some errors persisting, particularly with index provisioning.</p>
<p>Around 1:00pm PST we identified that the previous cluster restart had not been
entirely successful. Rather, the cluster had partitioned itself into two
clusters. One with half of the nodes present, and another with all but one node.
The last node was correctly failing to join either. This is a so-called
&#8220;split-brain&#8221; state, wherein a cluster partitioning event may incorrectly lead
one subset of the cluster to elect a new master.</p>
<p>Much of the normal index traffic would have been successful at this time, having
been likely to route to the larger cluster subset. However, index deletion and
creation was still failing at this time, in addition to the single node refusing
to join the cluster, prompting our continued investigation.</p>
<p>By about 2:21pm PST, we had identified the split brain state of the cluster, as
well as the precise discrepancies in the cluster state among all of our nodes.
At this time, we decided to perform another full restart of the cluster. This
time we waited briefly after stopping all nodes before starting them again, to
allow cluster discovery to proceed without any potential issues.</p>
<p>This restart was successful in reuniting a correct and complete cluster. All
nodes connected and successfully elected a master node. Indexes were gradually
recovered throughout the cluster. By 3:00pm PST, the cluster had recovered
98–99% of all indexes.</p>
<p>While troubleshooting the missing indexes, and cross-referencing them with
accounts and user activity, we were continuing to receive reports from customers
of failed attempts to recreate their indexes. These customers had generally
attempted to self-troubleshoot by deleting and recreating their indexes, only to
find that index deletion requests would fail with a 502 error, only to
apparently succeed a few minutes later, while index creation requests failed
entirely.</p>
<p>Based on the logs, we saw that servers within the Elasticsearch cluster were
rapidly and repeatedly trying to recover the 1% of unassigned indexes. These
attempts, and subsequent failures, were generating a storm of cluster state
updates, which effectively poisoned the cluster&#8217;s attempt to service normal
traffic which changes the cluster state, such as index creation and deletion.</p>
<p>Ultimately, we were able to verify that the missing indexes had all been
previously deprovisioned by customer activity, and were awaiting their final
deletion. Removing these indexes on the filesystem prevented Elasticsearch from
attempting to recover the indexes, quieting internal cluster state traffic, and
restoring the ability for customers to delete and recreate indexes.</p>
<p>During this outage, we communicated in some detail with Shay from Elasticsearch,
who graciously provided assistance troubleshooting and recovering our cluster.
Shay was able to confirm that a few of the issues we faced in our outage have
since been fixed in the upcoming 0.90 release of Elasticsearch, as well as made
some recommendations for mitigating similar events in the future. We also
provided excerpts of our logs for his more thorough analysis.</p>
<p>By 6:50pm PST our cluster returned to normal operation, reporting a green state,
with normal load levels, CPU and memory usage across all nodes.</p>
<p>Total duration of this outage was approximately 20 minutes of hard downtime for
an average index, due to cluster restarts. In addition to that, approximately
six hours of degraded service, primarily in the form of failures to create and
destroy indexes, as well as high latency and various intermittent 404, 500, 502
and 503 errors.</p>
<h2>Lessons learned &amp; fixes to implement</h2>
<p>This outage identified a number of lessons for us to apply, from simple growing
pains, to outright misconfigurations of our cluster. Following yesterday&#8217;s
outage we have a number of changes to implement to harden our cluster against
similar future outages.</p>
<h3>Isolate master-only routing nodes</h3>
<p>It is a recommended practice for high-scale Elasticsearch clusters to use a set
of dedicated nodes which do not store data, and whose only responsibilities are
routing and master eligibility. This helps avoid the problem of splitting a
cluster when a node is subjected to excessive load. It has long been our plan to
evolve to this kind of cluster design, and now we see that our cluster has
reached a size where this is now necessary.</p>
<h3>Enforce more precise connection throttling</h3>
<p>We believe the original root cause of high system load was due to an excessively
high number of expensive requests being allowed through to the cluster. We are
in the process of making our connection throttling a bit more sophisticated to
better match our desired traffic levels.</p>
<p>In our experience, scaling effectively in the cloud means building up and
planning cluster capacity from known quantities, including the number of
connections allowed per customer and resource. We have a design which should
help protect fair and reasonable usage within the cluster, which should go
mostly unnoticed by the vast majority of normal usage.</p>
<h3>Set minimum master nodes to a correct value</h3>
<p>Our minimum master nodes setting was incorrectly set to half of our
master-eligible nodes, rather than the recommended half plus one. This was
likely a significant contributing factor to our cluster partitioning. We have
already corrected this setting to prevent the possibility of future split brain
scenarios.</p>
<h3>More frequent purging of old index data to help improve cluster restart</h3>
<p>When a customer deletes an index, we issue a &#8220;soft&#8221; delete which leaves the
index in place within the cluster, until we periodically run a manually verified
process to purge old data. This lets us recover large indexes from accidental
deletion, which does happen from time to time. However, all of the extra index
data simply served to slow down cluster restarts as old indexes were loaded
alongside the new.</p>
<h3>Sunsetting our beta indexes</h3>
<p>Similar to more frequently purging deleted index data, the time has come for us
where we will soon announce the sunsetting of our public beta indexes and ask
customers to upgrade to one of our production accounts. Moving beta accounts to
our newer production plans will allow us to remove some deprecated internal
architecture, as well as more accurately plan our cluster capacity and provide
for faster recoveries during server and cluster restarts.</p>
<h2>Summary</h2>
<p>We are continually grateful for all of our customers&#8217; patience and
understanding. Thank you for your support! We take these outages very seriously.
We have much higher standards for ourselves and our business, and we are
committed to improving ourselves and serving you better in the future.</p></div>

March 4th Cluster Outage and Post Mortem

<div><p>Our <a href="http://status.bonsai.io/657919/2013/02" target="_blank" rel="nofollow">uptime in February 2013</a> was a
whopping 100%. We&#8217;re pretty pleased with this figure, as it included a number of
improvements deployed to most of the components in our system, including an
update to Elasticsearch itself.</p>
<p>There were two minor hiccups in the service not accounted for in our uptime
reports.</p>
<p>On the 21st, we had a few hours during which <a href="/blog/2013/02/21/index-status-creation-post-mortem">index creation requests were
failing</a> due to a regression
deployed with our <a href="/blog/2013/02/19/multi-index-search">multi-index syntax support</a>.</p>
<p>Another was on Monday the 25th, when an Elasticsearch node was restarted
abruptly to recover from unusual load and memory usage, without having been
removed from our load balancers. During that time, a fraction of requests routed
to that node would have failed until our load balancer noticed and removed it
from the rotation.</p></div>

February 2013 Availability & Uptime

<div><p>We recently found and fixed a small regression in our routing layer, which caused some complications with index creation over the past 24 hours. We felt the issue deserved a post-mortem writeup with more details.</p>
<p>On the evening of February 19th, we deployed an update to our routing layer to support basic <a href="/blog/2013/02/19/multi-index-search">multi-index routing syntax</a>. However, there was a subtle bug in this update which affected index deletion, in turn causing complications with subsequent index creation.</p>
<p>First, some background.</p>
<p>Our architecture has an additional routing proxy layer that runs in front of Elasticsearch. Among other things, it is this layer&#8217;s responsibility to handle load balancing, log traffic metrics, whitelist and pre-process certain APIs, enforce account limits, and authorize requests.</p>
<p>Toward this end, one major function of the proxy is to map the &#8220;logical&#8221; or &#8220;public&#8221; URL which identifies a specific index belonging to a specific account. This URL is transformed into a &#8220;private&#8221; URL for an index within the flat, global namespace of our per-region shared cluster. This mapping functions as a kind of whitelisting for request authorization, and also allows us to run a single more efficient cluster and pass on some beneficial economies of scale to our customers.</p></div>

Post-mortem - Fixed Handling of Deleted Indexing

<div><h2>The Case for Multiple Indexes</h2>
<p>One of Elasticsearch&#8217;s biggest strengths is the ease of creating many indexes. It&#8217;s just a simple HTTP POST, and you&#8217;re ready to start indexing documents!</p>
<h5>Example: One index per environment</h5>
<pre><code class="language-shell">curl -X POST http://index.bonsai.io/blog-development
curl -X POST http://index.bonsai.io/blog-staging
curl -X POST http://index.bonsai.io/blog-production
</code></pre>
<p>Using many indexes for your application is very useful, particularly for scaling. When you have a lot of data, you can use many kinds of natural partitioning points in order to flexibly create many small indexes, which can be easier to scale than a single monolithic index.</p>
<p>Some examples of why an app may need many indexes:</p>
<ul>
<li><strong>Multiple environment</strong> support for keeping your development, staging, production and experimental data isolated.</li>
<li><strong>Multi-tenancy,</strong> in which you create an index per customer in multi-tenant applications.</li>
<li><strong>Temporal sharding,</strong> for applications which process a high volume of updates, particularly when recent documents are more valuable than older documents.</li>
<li><strong>Logical partitioning</strong> between models, to scale one model separately from another. Or within models, to keep related activity closely grouped.</li>
<li><strong>Hot reindexing,</strong> when you need to update your index analysis and mappings settings. Just create a new index and reindex into that in parallel without altering your existing index.</li>
</ul>
<h5>Example: Basic three day rolling-window temporal sharding</h5>
<pre><code class="language-shell">curl -X POST http://index.bonsai.io/events-day0
curl -X POST http://index.bonsai.io/events-day1
curl -X POST http://index.bonsai.io/events-day2
</code></pre>
<h2>Searching Across Indexes</h2>
<p>Storing your data into multiple indexes is only as useful as getting your documents back <em>out</em> of those indexes. Toward that end, Elasticsearch has excellent support for multi-index querying. Present since its earliest versions, a multi-index search provides multiple index names in the search request, separated by commas:</p>
<h5>Multiple-index search with curl</h5>
<pre><code class="language-shell">curl http://index.bonsai.io/authors,articles,comments/_search
</code></pre>
<p>This is a useful feature for, e.g., the Tire Ruby client, which partitions separate models into their own indexes by default:</p>
<h5>Multiple index search with Tire</h5>
<pre><code class="language-ruby">Tire.index('authors,articles,comments').search do
  # ...
end
</code></pre>
<h2>Now Supported on Bonsai</h2>
<p>Because of the design of our systems, we don&#8217;t always support 100% of Elasticsearch&#8217;s API (see <a href="/faq">our new FAQ</a> for more reasons why). However, as of today, you can now query your indexes with basic multi-index search syntax.</p>
<p>Reserved for future implementation work is the more advanced wildcard and operator syntax, <a href="http://www.elasticsearch.org/blog/2012/07/02/0.19.8-released.html" target="_blank" rel="nofollow">introduced in Elasticsearch 0.19.8</a>. Both are being worked on with a careful eye on ensuring proper security and privacy of the indexes in our clusters.</p>
<p>If you need this, or any other Elasticsearch feature, <a href="/contact">let us know</a>! We are a small, focused development team and prioritize our plans based on input from our customers.</p></div>

The Case for Multi-Index Search

<div><p>We upgraded our cluster today to the latest version of Elasticsearch, 0.20.4.</p>
<p>As we were previously running on 0.20.1, this deploy includes a handful of new features and bug fixes from <a href="http://www.elasticsearch.org/download/2012/12/27/0.20.2.html" target="_blank" rel="nofollow">0.20.2</a>, <a href="http://www.elasticsearch.org/download/2013/01/25/0.20.3.html" target="_blank" rel="nofollow">0.20.3</a>, and <a href="http://www.elasticsearch.org/download/2013/01/29/0.20.4.html" target="_blank" rel="nofollow">0.20.4</a>. Some of these recent versions of Elasticsearch include bug fixes for issues such as we saw earlier in January.</p>
<p>Maintaining an up to date system free from outdated dependencies and bitrot is one of our main priorities as a service provider. As such, we try to stay as up to date as possible with the latest version of Elasticsearch, and have designed our systems in order to minimize the cost of deploying the latest version of Elasticsearch.</p>
<h3>“My Code&#8217;s Deploying!”</h3>
<p>Deploying new versions of our internal services, Elasticsearch included, is a very deliberate process for us. Getting the code and configurations onto the servers tends to be straightforward. It&#8217;s the restarts that can be nerve-wracking.</p>
<p>Our rolling restarts go through a very thorough checklist which look something like this:</p>
<ol>
<li>Gracefully remove a server from our load balancers to stop it from receiving new requests.</li>
<li>Wait for all current requests to finish processing.</li>
<li>Restart the service with the new code and configurations.</li>
<li>Poll the service health checks. For Elasticsearch, we wait until the cluster status is green and everything has completely initialized and rebalanced itself.</li>
<li>Add the server back to our load balancers.</li>
<li>Wait a few seconds until requests start processing.</li>
</ol>
<p>This is an unhurried process, but after a few years of experiencing managing production hosting services, we wouldn&#8217;t have it any other way.</p>
<p>The best part? Deploying new services becomes the perfect legitimate excuse for slacking off.</p></div>

Cluster Upgrade to Elasticsearch 0.20.4

<div><p>According to our status page, our cluster availability was approximately 99.99% in January 2013, due to a five minute outage. This was a partial outage with limited impact, so that five minute figure is slightly less, and slightly more, serious than it looks.</p>
<p>Let me start from the beginning.</p>
<h3>Early alerts</h3>
<p>Our first indication of a problem was when an index failed to provision successfully, causing the index status to go red. This is what triggered an outage with Pingdom, and starts the five minute outage in our uptime report for January.</p>
<p>We were paged right away, and within a few minutes had identified the affected index as the cause of the red cluster status. Recovering the cluster status was a matter of re-creating the index. We also contacted the affected customer to let them know of the issue that we discovered, and to ask for any feedback they could provide on their end.</p>
<h3>Initial recovery, and continued investigation</h3>
<p>Fixing the single affected index let us restore the cluster state back to green. However, our tests, and some tests from our helpful customer, showed that some index creation attempts were still failing randomly.</p>
<p>The random index creation failures seemed proportional to a single node in our index. Based on this, and some output from our logs, we identified a node that we thought was having issues connecting to the rest of the cluster.</p>
<p>After restarting this node, however, it was not able to join the cluster successfully. We promptly removed this node from our load balancing to further troubleshoot the problem. Around this time, a subset of some requests routed to this particular node would have returned with a 404 error.</p>
<h3>Bringing in the big guns</h3>
<p>With one node offline, and index creation still failing intermittently, we enlisted the help of Elasticsearch creator Shay Banon to troubleshoot the situation.</p>
<p>Shay was able to help direct our attention to the relevant log output, as well as interpret its output, to ultimately identify an anomaly during cluster discovery. One of the nodes in the cluster was returning outdated information for the other nodes, causing attempts for a new node to join the cluster to fail.</p>
<p>After cross referencing each node&#8217;s picture of the rest of the cluster, we identified the node that was more likely to be the source of our cluster connection problems. Restarting this node went smoothly, and allowed us to bring our other node back online smoothly as well.</p>
<p>After a few minutes monitoring the cluster&#8217;s recovery and rebalancing, we added all nodes back to load balancing.</p>
<h3>The total impact</h3>
<p>During this incident, index creation and deletion would have been intermittently unsuccessful for approximately 90 minutes. We were able to recover affected indexes once the cluster had been recovered.</p>
<p>Furthermore, some requests would have failed with a 404 error or similar when inadvertently routed to a node with an incomplete view of the rest of the cluster. These happened in one- or two-minute intervals for a total of less than ten minutes throughout our debugging.</p>
<p>Those issues aside, the rest of our cluster traffic should have been un-impacted. Production indexes were replicated across multiple nodes, and recovered seamlessly from the loss of a single node. Staging and beta indexes were similarly running at an extra level of replication in preparation for a later cluster expansion, and were likewise unaffected.</p>
<h3>Looking toward the future</h3>
<p>Having recently soft launched our <a href="http://addons.heroku.com/bonsai" target="_blank" rel="nofollow">Heroku Elasticsearch Add-on</a> to production this month, we&#8217;re counting this as our first production outage. As such, we decided it would be worthwhile to write up a more comprehensive post-mortem on the event.</p>
<p>Going forward, we would like to maintain a habit of posting a monthly uptime summary, including the details of any similar outages, if they have not already been published.</p>
<p>We take uptime and availability very seriously. We recognize that our service is being used in production by hundreds of applications, who are quite literally trusting their business on the quality of our service. We&#8217;re humbled and grateful for that trust, and will keep working hard to earn it.</p>
<h3>We&#8217;re hiring!</h3>
<p>Are you enamored with Elasticsearch? Do you enjoy some ops? Would you like to work more closely with customers? We&#8217;re looking for another engineer to join our team. If you&#8217;re a junior- to mid-level developer who wants to jump into the deep end of running an exciting infrastructure service, introduce yourself to Nick Zadrozny at <a href="mailto:nick@bonsai.io">nick@bonsai.io</a>.</p></div>

January Availability & Operations

<div><p>Our original plan called for us to upgrade our 0.90 clusters one month after the initial release. We bumped that back a few times in order to allow everyone plenty of time to test their applications on 1.0. This upgrade does introduce a small number backwards incompatible API changes for some use cases. But overall the benefits of moving to Elasticsearch 1.0 far outweigh the costs of upgrading.</p>
<p>During that waiting period, we&#8217;ve been far from idle on our end. Over the last few weeks, we have run dozens of practice migrations. Even now, we&#8217;re running a full copy of all our 0.90 clusters on Elasticsearch 1.0. This means that we have the capability to perform &#8220;hot&#8221; upgrades to 1.0, one customer at a time.</p>
<p>Starting next week, we will be carefully and individually migrating each customer still on 0.90. Each cluster will experience a short period of read-only access to help synchronize the data between clusters. Any updates made to your index will be rejected with a 503 error, and should be retried.</p>
<p>Migrations will start on the afternoon (PST) of <strong>Monday, April 7th</strong>, and run throughout the week.</p>
<h3>Q&amp;A</h3>
<h4>What if I haven&#8217;t had time to try Elasticsearch 1.0? Can I opt-out?</h4>
<p>If you haven&#8217;t yet tested your apps locally on Elasticsearch 1.0, we strongly encourage you to do so as soon as possible. If you need to opt-out of next week&#8217;s upgrades and schedule a specific migration window, we understand, just send us an email to support@bonsai.io and we&#8217;ll schedule something with you.</p>
<h4>Why upgrade to 1.0 at all?</h4>
<p>We recognize that long-term support for major versions of older software is valuable to some. We are considering that for future post-1.0 major upgrades of Elasticsearch.</p>
<p>Elasticsearch 1.0 is something of a special case. It introduces a bunch of improvements, both for normal users, as well as cluster operators such as ourselves. One important new operational feature is an official API for performing backups. Finally, in our opinion, the API changes are relatively minor for most use cases.</p>
<p>We consider it our responsibility to take an informed position on these kinds of questions. Our customers choose our service because we can stay up to date more effectively, providing you with the benefits of updated software, rather than risk stagnating on older versions.</p>
<p>Overall, we think the benefits of migrating to 1.0 far exceed the costs of adapting to its changes.</p>
<h4>I use Tire, is it compatible with Elasticsearch 1.0?</h4>
<p>Tire is compatible with Elasticsearch 1.0 for many use cases, but as of this writing there are some isolated issues in current versions of the gem (0.6.2). The best way to find out if your app is compatible is to run it against 1.0 to test, which you can do easily on Heroku with our free Starter plan.</p>
<p>Because Tire has been deprecated in favor of the official <a href="https://github.com/elasticsearch/elasticsearch-rails" target="_blank" rel="nofollow">elasticsearch-rails</a> gem, we strongly recommend that you migrate your application&#8217;s code to replace tire with elasticsearch-rails as soon as possible.</p>
<p>If migrating to elasticsearch-rails is not an option, we recommend using Tire from the git master branch in order to benefit from unreleased fixes for 1.0</p>
<h4>I have some other question, where can I get it answered?</h4>
<p>Drop us a line at <a href="mailto:support@bonsai.io">support@bonsai.io</a> and we&#8217;ll be happy to help.</p></div>

Update on Elasticsearch 0.90 to 1.0 Migration

March 2014 Uptime: 99.8%

February Uptime: 99.94%

<div><p>Big congratulations to the Elasticsearch team, which shipped its version 1.0 yesterday! This is a major milestone that represents thousands of hours of effort from a team with a ton of momentum.</p>
<p>We&#8217;re also pleased to announce that you can now use Elasticsearch 1.0 on Bonsai! As of yesterday, all new Heroku Add-on accounts in US East are being provisioned on Elasticsearch 1.0.</p>
<pre><code class="language-json">{
  "status": 200,
  "name": "Crimson Dynamo",
  "version": {
    "number": "1.0.0"
  },
  "tagline": "You Know, for Search"
}
</code></pre>
<h3>Upgrade Plans for Existing Customers</h3>
<p>Because Elasticsearch 1.0 is a major release, it also introduces a number of backwards-incompatible changes to its APIs. That means we need to be careful to allow plenty of time for our customers to test their applications against the newer APIs and fix any changes that need to be made.</p>
<p>Our first step toward 1.0 will be to upgrade our clusters to the very latest version of the Elasticsearch 0.90 series (currently 0.90.11). We&#8217;ll perform that upgrade next week, on Wednesday, February 19th. This upgrade will include an upgrade to Lucene, and a number of other changes and fixes that should help ease the transition to 1.0 on the back end.</p>
<p>We&#8217;ll keep a close eye on our new 1.0 cluster, as well as on Elasticsearch development activities, in order to spot any last-minute bug fixes that may necessitate a bug-fix 1.0.1 release. Assuming all goes smoothly, we will perform the final upgrade to Elasticsearch 1.0 on Wednesday, March 12th. That is in one month from today.</p>
<h4>Timeline</h4>
<ul>
<li><strong>Wednesday, February 12th</strong>
<ul>
<li>Elasticsearch 1.0 release</li>
<li>All new Bonsai clusters in US East are created on Elasticsearch 1.0</li>
</ul>
</li>
<li><strong>Wednesday, February 19th</strong>
<ul>
<li>Bonsai 0.90 clusters are upgraded to the latest 0.90 series</li>
</ul>
</li>
<li><strong>Wednesday, March 12</strong>
<ul>
<li>All Bonsai 0.90 clusters are upgraded to Elasticsearch 1.0</li>
</ul>
</li>
</ul>
<h3>Preparing For Elasticsearch 1.0</h3>
<p>First off, you should <a href="http://www.elasticsearch.org/overview/elkdownloads/" target="_blank" rel="nofollow">download and install Elasticsearch 1.0</a>. Set it up in your local development environment, and test your application accordingly.</p>
<p>Next, Heroku users can create a new Bonsai Add-on account to be provisioned on our 1.0 cluster. Using <a href="https://blog.heroku.com/archives/2013/6/27/heroku-fork" target="_blank" rel="nofollow">heroku fork</a> makes it trivial to create a full copy of your production environment, at which point you can reindex and use the app to verify that all is well in production.</p>
<p>If you&#8217;re willing to reindex, and would like to move over sooner, you can email us at <a href="mailto:support@bonsai.io">support@bonsai.io</a> and we can migrate you right away. Otherwise, stay tuned for the in-place upgrade in March.</p></div>

Elasticsearch 1.0 has launched!

<div><h3>January 2013</h3>
<p>January brought a fresh new year, with new projects, renewed ambitions, and 100% availability from our services. Our Heroku add-on experienced a few hours of unavailability for adding and updating accounts, but that was easily corrected, and search traffic was otherwise unaffected.</p>
<h3>Previous months</h3>
<h4>December 2013</h4>
<blockquote>
<p>&#8216;Twas the month before Christmasand all through the cloudnot a service was stirringour clusters ran proud.</p>
</blockquote>
<p>December was a fairly quiet, hands-off month. Technically we did experience a small blip: about five seconds of partial 503 responses during a minor system maintenance deploy. However, the issue was automatically corrected, and otherwise our service ran uninterrupted for the month.</p>
<h4>November 2013</h4>
<p>While our Pingdom checks report a bit differently due to a config change in our health-check endpoint, our uptime for November 2013 was 100%. During November, we performed some notable upgrades to our routing proxy layer, as well as minor release upgrades to Elasticsearch in all our clusters and regions.</p></div>

January Uptime: 100%

<div><blockquote>
<p>&#8216;Twas the month before Christmas / and all through the cloud / not a service was stirring / our clusters ran proud.</p>
</blockquote>
<p>December was a fairly quiet, hands-off month. Technically we did experience a small blip: about five seconds of partial 503 responses during a minor system maintenance deploy. However, the issue was automatically corrected, and otherwise our service ran uninterrupted for the month.</p></div>

December Uptime: 99.999%

<p>While our Pingdom checks report a bit differently due to a config change in our health-check endpoint, our uptime for November 2013 was 100%. During November, we performed some notable upgrades to our routing proxy layer, as well as minor release upgrades to Elasticsearch in all our clusters and regions.</p>

November Uptime: 100%

<div><h2>Summary</h2>
<p>On Tuesday, October 29, we experienced an outage on our primary Elasticsearch cluster in our US East region. Up to 20% of our customers in this cluster experienced a total downtime for 24 hours. The remaining 80% experienced several windows of downtime during which we performed emergency maintenance to the cluster. All customers have since been migrated to one of several replacement clusters.</p>
<p>This outage was the most severe in our four-year history, which we profoundly regret. The following post-mortem provides a summary of events, our analysis of them, and the actions we have taken and continue to take to prevent this from ever happening again.</p>
<h2>Background</h2>
<p>At the time of the outage, our main US East Elasticsearch shared cluster had been hosting over 10,000 shards for many thousands of indices. We fully expected this size to become a scalability challenge, and already had taken steps and released some back-end tools to help deploy and manage a set of smaller, isolated Elasticsearch clusters.</p>
<p>As part of this effort, we have relatively recently developed a new generation of our operational provisioning and configuration management scripts. This latest revision of these tools represent an accumulation of our last few years of experience managing production data services in AWS, and they have proven to be much leaner, more maintainable, and more efficient to operate.</p>
<p>These new tools had already seen fairly extensive usage and testing over the last few months. They are used for all of our staging and development clusters, as well as in production for all new customer dedicated clusters. We have also since migrated over our younger and smaller EU shared cluster without incident.</p>
<p>Our plan for the older US East cluster was to migrate it to use these new scripts, and then use the improved toolset to help partition and migrate its data to a set of replacements.</p>
<h2>Monday, 28 Oct 2013</h2>
<p>On Monday evening we had seen some signs of cluster load. The symptoms we observed were an increase in the average maximum response times of requests, as well as occasional timeouts when performing cluster state operations.</p>
<p>This triggered an alert internally to investigate. The engineer on call interpreted the symptoms as a result of the cluster being under-provisioned, and decided to add new servers. This is similar to previous incidents, where similar symptoms had been alleviated by adding new servers to the cluster.</p>
<p>Our newer servers joined the cluster successfully, and symptoms seemed to subside.</p>
<p>We had deployed the replacement servers using our newer generation operations tools, with the reasoning that it was sufficiently tested, and we could begin the migration by gradually replacing servers over time. So in this case, we spent some extra time monitoring the logs as a team following the deploy.</p>
<p>In this case, we began to see an unfamiliar error message in the logs. This message included a cause of <code>TransportSerializationException[Failed to deserialize exception response from stream]</code>. We investigated these messages for a few hours, testing and ruling out a handful of early hypotheses of its cause.</p>
<p>With only a few of these errors, and with no clear indication of the source, and with no other signs of a negative impact to the cluster, we decided to defer more in-depth analysis until the next morning.</p>
<h2>Tuesday, 29 Oct 2013</h2>
<p>Around 8:30am EST / 5:30am on Tuesday morning, we were alerted of continued degradation of cluster performance. The morning showed similar signs of errors that we had seen the previous evening.</p>
<p>Our engineer on call interpreted these symptoms as continued cluster load, and decided to add another large batch of extra servers to our cluster as temporary extra capacity. With the new servers coming online, we only saw cluster state operations sharply degrade, along with a corresponding increase in the <code>RecoveryFailedException</code> and <code>TransportSerializationException</code> errors in the logs.</p>
<p>At this time we realized that there was a more serious error happening than simple cluster load from customer traffic, and promptly paged the rest of the team to have all hands available to respond.</p>
<p>Our first priority was to immediately get an accurate assessment of the state of our cluster, so that we could logically search our way back toward identifying contributing factors.</p>
<p>The largest and most obvious source of recent change was the set of new servers, which had been provisioned using a newer version of our cluster provisioning and configuration management tools. Our first step in troubleshooting was to simply shut down Elasticsearch on all of the recently provisioned servers, in order to minimize any negative impact they may have been contributing.</p>
<p>Shutting down these servers caused a subset of indexes which had already been successfully relocated to the new servers to go offline. However, it did buy us some room to further troubleshoot the new servers without worrying about propagating a more serious error.</p>
<p>On a closer look, we immediately noticed one error. A handful of the new servers experienced a bug in their provisioning scripts which caused them to prepare their data volumes incorrectly. This left the Elasticsearch data directory to be set up on the wrong device, generating disk full errors on those nodes, causing corruption on a small number of shards that had attempted and failed to be relocated to those nodes. These nodes had their data volumes repaired in place, and the affected shards were later recovered from a replica or a recent data snapshot.</p>
<p>Continuing to troubleshoot, we returned to our earlier <code>TransportSerializationException</code> message.</p>
<p>At this time, most of our servers in the older cluster were running Java <code>1.7.0_17</code>. Due to an oversight in our configuration management scripts, we hadn&#8217;t &#8220;pinned&#8221; the patch version of Java. The newer servers had been built using a more recent minor version of Java, <code>1.7.0_45</code>. Some more focused research eventually indicated this version mismatch as a problem.</p>
<p>The full description of that error is described in this GitHub issue for Elasticsearch: <a href="https://github.com/elasticsearch/elasticsearch/issues/3145" target="_blank" rel="nofollow">https://github.com/elasticsearch/elasticsearch/issues/3145</a></p>
<p>Once we learned about this problem, we updated our provisioning scripts to rollback and replace the more recent versions of Java to the already widely deployed version. Once completed, we would be able to bring these servers back online one by one to recover the shards that had previously been taken offline in their shutdown.</p>
<p>While starting these servers, we observed that the cluster was taking a much longer time to recover the missing shards than we would expect. We spent some time iterating on Elasticsearch recovery throttling settings, however the master node would occasionally become unresponsive, forcing us to perform an emergency full cluster restart for new settings to take effect.</p>
<p>Over the course of the afternoon, we continued to observe the cluster recovery process gradually slow down and hang as the cluster approached 80% online. We continued to develop, research and test a number of hypotheses to overcome this problem and recover the cluster to a green state.</p>
<p>Some of our hypotheses included tuning cluster recovery throttling settings, allocation rack-awareness settings, and possibly-corrupt transaction logs. Each of these attempts generally required a full cluster stop and restart, which was very time consuming, continuing through the afternoon and well into the night.</p>
<p>Around 2:30am PST / 5:30am EST on Wednesday, we still had roughly 80% indices online and no clear ETA on full recovery of the Elasticsearch cluster. At this point we decided to call for a short break to sleep and regroup.</p>
<h2>Wednesday, 30 Oct 2013</h2>
<p>At 6:30am PST / 9:30am EST we decide that it is time to deprecate the old cluster and abandon our attempts to recover it in place.</p>
<p>Within the next hour, we provisioned the first of a new set of replacement clusters, and deployed a handful of systems we had developed to help manage a set of partitioned, independent clusters. After a quick round of testing, we began allocating all new accounts on this replacement cluster, and updated all our open support cases about the availability of a quick fix to migrate their data.</p>
<p>Over the next few hours, we helped a large number of our customers still impacted by the outage migrate over to the newer cluster. By 7:30am PST we were &#8220;through the worst of it,&#8221; with over 95% of indexes online. Throughout the rest of the morning, we provisioned another replacement cluster where we were able to manually migrate the data of anyone who was still offline and unable to reindex.</p>
<h2>Thursday &amp; Friday</h2>
<p>Throughout the rest of the day on Wednesday, and into the rest of the week, we continued to migrate our customers off the older cluster and onto our newer architecture. We worked painstakingly in batches of decreasing apparent impact. Over the course of Wednesday and Thursday we developed our capability to mass-migrate customers in large batches, and quickly verify the success of those migrations.</p>
<p>Ultimately, we completed the last of our migrations on Friday afternoon, bringing our service back to 100% operational status for production accounts.</p>
<h2>Today</h2>
<p>All of our clusters are running well. We have sufficiently partitioned our original monolithic cluster to avoid the scalability issues we recently encountered. This is the culmination of a longer-term effort which we think will greatly serve our long-term scalability.</p>
<p>Our older cluster is still running, however at this point it is only serving our much older beta accounts. These accounts were left in place due to their dependence on legacy system compontents that were not migrated. Anyone still running a beta index has since been contacted directly with our plans to end-of-life support for beta accounts, and instructions for a smooth upgrade.</p>
<h2>Elasticsearch Lessons Learned</h2>
<p>We did a fair amount of digging into the source code for Elasticsearch itself while trying to recover our monolithic cluster in place, and learned a few things about how the ES master works.</p>
<p>The ES master runs in a single thread. In addition to performing recovery, this thread also makes adjustments to the resources alotted to running indices. This has some consequences for recovery and failover:</p>
<p>With a low value of <code>node_concurrent_recoveries</code>, recovery runs in <code>N^2</code> time. Each loop iteration recovers a constant number of indices, but that loop takes time proportional to the number of running indices. With a high value of <code>node_concurrent_recoveries</code>, the garbage collector can have a hard time keeping up.</p>
<p>Also, at certain points in the loop, updates to cluster settings are not accepted immediately, and may timeout entirely. Updating the cluster settings via API is often impossible, and you should be prepared to patch config files and/or the binary itself to affect the cluster settings you need. Expect some failovers to require a full cluster restart.</p>
<p>For example, we were unable to set <code>index.recovery.initial_shards</code> via the api, so we were forced to deploy a patched build of Elasticsearch.</p>
<p>Handling of cluster state traffic has had a number of improvements in the 0.90 series, and we expect those improvements to continue in future versions of Elasticsearch. We&#8217;re also looking forward to the cleaner backup and restore APIs being implemented in 1.0.</p>
<p>Until some of these improvements happen, we don&#8217;t currently recommend running individual clusters with much more than 1,000 shards. Use cases which require a large number of shards should run many smaller clusters where possible. Clusters with many shards should run dedicated master-only nodes on servers with <code>node_concurrent_recoveries</code> set to at least 10% of the total number of shards. A large amount of RAM on the master will help relieve some pressure from the garbage collector.</p>
<h2>Lessons Learned for Humans</h2>
<p>Call for help when making changes to production under duress. Just because it looks like something you&#8217;ve seen before doesn&#8217;t mean it is, and it&#8217;s much easier to misinterpret the circumstances or commit errors in judgement when you&#8217;re operating alone. Develop a culture of asking for help, because by definition it&#8217;s impossible to tell in advance if you&#8217;re about to make things worse. (Otherwise, you wouldn&#8217;t be doing it, right?)</p>
<p>Businesses grow, and change happens. Embrace it, but always minimize the changes you make to a running production system. Even code that is well tested when running in isolation can cause problems if it introduces too many changes at once in a running system.</p>
<p>Just because you have a horizontally scalable system doesn&#8217;t mean that you should run an arbitrarily large cluster. In many cases, it&#8217;s beneficial to partition after an empirically or even arbitrarily determined size, both to isolate failure and to minimize time-to-recovery. This is something we have historically done well with our sister search hosting service, Websolr, and was the basis for what we&#8217;ve now implemented with Bonsai.</p>
<h2>Conclusion</h2>
<p>This outage was the most severe in our company&#8217;s four years, and we are humbled by the experience. We are also humbled by the gracious response of our customers. We can only hope to make up to all of you with our continued efforts to learn and adapt improve our service on your behalf.</p>
<p>The silver lining from last month&#8217;s outage is that we have since completed our long-intended migration forward onto a much more robust architecture. This has been working very well, and will serve as the foundation for many improvements to come.</p></div>

October 2013 Post-Mortem

<div><p>After a smooth upgrade, all of our customers are now running on the latest and greatest Elasticsearch 0.90.0!</p>
<pre><code class="language-json">{
  "ok": true,
  "status": 200,
  "version": {
    "number": "0.90.0",
    "snapshot_build": false
  },
  "tagline": "You Know, for Search"
}
</code></pre>
<p>Elasticsearch 0.90 was released just last week, and is planned to be the last major version before an upcoming 1.0 release. Version 0.90 includes a major upgrade of the underlying Lucene full-text indexing and search library, moving from its version 3.6 to 4.2.</p>
<p>While this upgrade did require a full cluster restart, we were able to limit our hard downtime to about three minutes. All index primary shards were back online in less than ten minutes, with full recovery of all shards and replicas within 30.</p>
<p>We are already pleased to see improved system usage metrics across the board, and we trust that you are all likewise seeing the benefits of the upgrade in your applications.</p></div>

Say 'Hello' to 'oh dot nine oh'

<div><p>Good news, everyone! Elasticsearch 0.90.0 was released today. Version 0.90 brings some excellent improvements for Elasticsearch users everywhere, and represents a large step forward toward its eventual 1.0 release.</p>
<p>We are getting ready to upgrade to 0.90.0 very soon. More on our plans in a bit. First, I’d like to give a sneak peek of some of 0.90.0’s biggest improvements, as well as a quick overview on the state of Bonsai itself.</p>
<h2>What&#8217;s new with Bonsai the business?</h2>
<p>It’s been a quiet month on the Bonsai blog. So what have we been up to?</p>
<p>First of all, we’ve been keeping up with you. The last few months have humbled us with very solid growth. We are rapidly approaching 1,000 active accounts, having long since passed 1,000 total (hello, freemium pricing plans!).</p>
<p>This is not our <a href="https://websolr.com/" target="_blank" rel="nofollow">first time</a> building a hosted search service, but it’s still encouraging to see a service bootstrap itself to profitability in the span of a few short months! With all signs pointing up, both for us and for Elasticsearch proper, you can bet that we’ll be around for a while.</p>
<p>The last few weeks have seen us quietly releasing a lot of quiet bug fixes and performance improvements to our systems, as well as support for a couple of Elasticsearch APIs that we initially launched without. Our thanks to customers who helped us identify odd edge case bugs, particularly in places where our account management and multitenancy intersects with the Elasticsearch API!</p>
<h2>What&#8217;s new in Elasticsearch 0.90?</h2>
<p>Lucene 4.</p>
<p>Lucene is the underlying full-text indexing library used by Elasticsearch, and is itself the industry standard for open source full-text search. Last fall, Lucene released a major upgrade to its version 4.0, bringing with it a huge number of new features and performance improvements.</p>
<p>Lucene 4 offers <a href="http://blog.mikemccandless.com/2012/07/lucene-400-alpha-at-long-last.html" target="_blank" rel="nofollow">a number of notable improvements</a> in performance and efficiency, along with a slew of faster and better abstractions for search engine developers to build upon. The bottom line: you get faster search results in a more robust cluster.</p>
<p>And Elasticsearch 0.90 is shipping with the latest Lucene 4.2.1. That’s over six months of new features and improvements building on top of Lucene 4.0, putting all of those powerful new abstractions to work.</p>
<blockquote>
<p>Probably the biggest improvement that our users will notice is much better memory usage when loading fielddata for faceting or sorting on a field. Fielddata uses less memory and makes it easier for the garbage collector to do its work, resulting in more stable clusters. This change alone makes it worth upgrading.</p>
</blockquote>
<p>Indeed, we see that the most common causes of query slowdowns are due to memory usage in queries with heavy faceting and sorting, as well as their negative impact to garbage collection behavior. We’re particularly happy to see these kinds of improvements coming to Lucene and Elasticsearch.</p>
<p>Here are a few other examples, among many, of the improvements in Lucene 4, via Lucene contributor <a href="http://blog.mikemccandless.com/" target="_blank" rel="nofollow">Mike McCandless</a>:</p>
<h4><a href="http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html" target="_blank" rel="nofollow">265% indexing speedup with Lucene&#8217;s concurrent flushing</a></h4>
<blockquote>
<p>Before concurrent flushing, whenever IndexWriter needed to flush a new segment, it would stop all indexing threads and hijack one thread to perform the rather compute intensive flush… But with concurrent flushing, each thread freely flushes its own segment even while other threads continue indexing. No more bottleneck!</p>
</blockquote>
<h4><a href="http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html" target="_blank" rel="nofollow">FuzzyQuery is 100 times faster in 4.0</a></h4>
<p>Fuzzy queries are great for flexibly matching terms which may have spelling mistakes or variations. This is particularly helpful for matching proper nouns — something I can personally appreciate given my last name, <em>“Zadrozny.”</em></p>
<p>Rather than brute-forcing its way through every term in the index, like it used to, fuzzy queries in Lucene 4 use a pre-generated <a href="http://en.wikipedia.org/wiki/Levenshtein_automaton" target="_blank" rel="nofollow">finite-state automaton</a> to quickly and efficiently calculate Levenshtein distances between words. Be sure to read Mike’s post in its entirety to see one of the many herculean feats of engineering that went into Lucene 4.</p>
<p>Efficiency improvements here also show up in suggesting similarly-spelled terms for spelling correction, and for autocomplete term suggestion.</p>
<h2>Elasticsearch 0.90 webinar</h2>
<p>There is a lot to unpack in Elasticsearch 0.90, and this Thursday, at 9:00am PST (18:00 CET), Elasticsearch developer Clinton Gormley will be holding a <a href="http://www.elasticsearch.org/webinar-whats-new-in-0-90/" target="_blank" rel="nofollow">webinar</a> to talk more about them. We’ll be there, and if you use Elasticsearch, you should be too!</p>
<p><a href="http://info.elasticsearch.com/Webinar_Version_0.90_.html" target="_blank" rel="nofollow">Register here for the webinar.</a> (It&#8217;s free!)</p>
<h2>Scheduled maintenance window</h2>
<p>Upgrading from Elasticsearch 0.20 to 0.90 will unfortunately require a full cluster hard shut down and restart. While we are doing everything in our power to minimize the actual impact and effect of this restart, you should expect a brief window of downtime.</p>
<p>We are scheduling our upgrade for next <strong>Monday, May 6th,</strong> at <strong>16:00 PST / 23:00 UTC.</strong></p>
<ul>
<li></li>
</ul>
<h3>Will you need to reindex your data?</h3>
<p>No, you do not need to reindex.</p>
<p>Fortunately, Lucene is backwards compatible within one major version. This means that Lucene 4 can read the Lucene 3 index format used by Elasticsearch 0.20, with a negligible performance decrease. Upgrading to Lucene 4 happens in the background, over time, as older Lucene 3 segments of your index are merged into the newer-format Lucene 4 segments.</p>
<h3>Other changes you need to make to your applications</h3>
<p>You should read the [Elasticsearch 0.90 download and changelog page] for full details on bug fixes, new features and one breaking change in 0.90.</p>
<p>We encourage all of our customers to take some time this week to try Elasticsearch 0.90 in their local development environment to ensure that everything is working as expected.</p>
<h2>Any other questions? Let us know!</h2>
<p>We&#8217;re here to help. If you have questions, just let us know with a quick email to <a href="mailto:support@bonsai.io">support@bonsai.io</a>.</p></div>