Whether you are an Elasticsearch beginner or an expert, get helpful Bonsai.io news as well as Elasticsearch tutorials, tips, and updates from our team.

Updates from the Bonsai Elasticsearch team, from One More Cloud: the first and best cloud search platform, since 2009.


Search Snippets


Schedule a free consultation to see how we can create a customized plan to meet your search needs.


Find out how we can help you.

<div><p>Along with many of your favorite service providers, Bonsai will be taking a firm stance of support for the new <a href="https://www.eugdpr.org/" target="_blank" rel="nofollow">GDPR</a> (General Data Protection Regulation) legislation. We at Bonsai believe this is an important piece of legislation that is designed to strengthen and unify data protection laws for all individuals within the European Union. In essence, the GDPR requires internet companies to expose how they store and share data belonging to any EU citizen. The regulation will become effective and enforceable on May 25, 2018.</p>
<p>While this regulation specifically protects EU citizens, Bonsai has always been committed to the data privacy of its customers. As Internet users and developers ourselves, we appreciate the goals and ideals of this legislation, and find no reason to not apply the features we build for GDPR to everyone.</p>
<h2>What is Bonsai doing about the GDPR?</h2>
<p>From its beginning, Bonsai has been designed to store as little personal information as possible. We generally store things required to communicate with you without sounding like complete robots. Nonetheless, we did a thorough review of our data stores to make sure that nothing was hiding in our database tables.</p>
<p>We personally reviewed every table in our database. For those of you that are using PostgreSQL for your database, this was the SQL I used to manually inspect every column.</p>
<pre><code>SELECT table_name, column_name, *
FROM information_schema.columns
WHERE table_schema = 'public'
ORDER BY table_name;
</code></pre>
<p>Additionally, has collaborated with our lawyers to ensure that we are able to fulfill every aspect of this regulation. Here are the measures we’ve taken in advance of our attorneys to meet the GDPR deadline:</p>
<ol>
<li>We have reviewed all of the data that we track about our customers, and identified what needs to be kept and what can be thrown away. This list will be incorporated into our application soon for your review.</li>
<li>We have internally documented our data flows of this data, and verified that all usage is needed for business operation. We then removed any data that was deemed unnecessary.</li>
<li>We have removed any vendor that is either not planning to support GDPR or has no public stance on GDPR.</li>
<li>We are improving security around the management and display of resource credentials, and are improving administrative workflows around system access for operational maintenance.</li>
<li>We have identified all third party cookies and removed those we no longer need or use. As an example, after review, we have decided to remove Facebook Pixel tracking.</li>
<li>We reviewed our Privacy policy, and are working to further clarify or update it where needed to satisfy new GDPR compliance requirements.</li>
</ol>
<h2>What do Bonsai Customers need to do?</h2>
<p>GDPR identifies two roles for handling data. The role of the Data Processor, a legal entity that processes personal data on behalf of the Data Controller and the role of Data Controller, a legal entity that determines the purposes and means of the processing. As a customer of Bonsai, you will be in the role of Data Controller for the data loaded into Bonsai&#8217;s services, and Bonsai wil be in the role of Data Processor. Its important when you are reviewing GDPR to keep this in mind.</p>
<p>Among many other things, there are two things that you might need to do depending on your situation and jurisdiction. Below are the only impactful changes that we can foresee that might affect you as a result of using Bonsai:</p>
<ol>
<li>Make sure your Terms of Service or Privacy Policy properly communicate to your users how you are using Bonsai (and any other similar services) on your website or app. We recommend you ensure your policies are up to date and clear to your readers.</li>
<li>If you are in the European Union, you’ll likely want to sign a Data Processing Agreement (DPA) with Bonsai. We’re happy to do so. Working with outside counsel, we are building a standard DPA that will be available to all customers.</li>
</ol>
<p>If you have any questions, please don&#8217;t hesitate to contact us at <a href="mailto:support@bonsai.io">support@bonsai.io</a>.</p></div>

GDPR and Bonsai: An Overview of Your Data Privacy

<div><p>The newest feature to join the dashboard family is the one we’ve been most excited about: announcing Bonsai’s Elasticsearch operations metrics, available now on your cluster dashboard! All accounts have access to cluster metrics, including those that provision our add-ons through Heroku or Manifold.</p>
<p>These performance metrics are fast, live-updating, and retain up to a month of request data. From platform to UI, they are a technical feat built entirely in-house with great care and attention to detail. You can toggle between local and UTC time for debugging with app logs, explore time ranges, and click and drag sections of time to focus your time window.</p>
<h3>Metrics are built for problem-solving</h3>
<p>At its core, the metrics page shows visibility and transparency into what is coming in and out of your cluster. It reveals not just how users are interacting with your cluster, but how your app is.</p>
<p>The graphs in this page give you a visual, and what they say depends on the question you’re asking. A question like “Is it fast?” is answered by the performance heat map and percentiles. “When are we under the most load during the week?” — zoom out the timeline and look at request count patterns.</p>
<p>The most valuable puzzle these graphs unravel is the question of causation. Each graph retains the same timeline so that trends can be linked across multiple graphs. For example, an increase in request counts can reveal load handling problems, which may mean that you need more resources. An increase in queue time might be linked to higher write traffic, perhaps from an ill-timed indexing task during peak read times.</p>
<h3>Worth the wait</h3>
<p>We’ve long wanted to empower Bonsai customers with visibility into their cluster, but it’s been difficult to find a solution that would scale affordably and remain performant. For many years our operations and support teams have been using things like Grafana and Kibana to manage, debug, and optimize clusters. But our databases that house the huge amount of request data we handle are under limited load: only a small set of engineers have access to them, and during analysis they have the patience to wait for complicated aggregation requests to complete (sometimes up to 30s, depending on the time window length). There was no way we could scale these tools for our thousands of clusters we manage without spending a ton of money on hardware to keep up with our client base. Moreover, there was doubt that those existing platforms would support a good UI experience (namely, a fast one). In fact, our first prototypes worked well if we only showed requests from the last hour. But every time we zoomed out, things slowed to a glacial pace. That was unacceptable.</p>
<p>In many ways, building metrics was a waiting game. Tooling is much better now than when we first launched Bonsai back in 2012. Our current infrastructure supports the high standards we had when we set out to ship this project.</p>
<h3>The technical stuff (or, Make It Fast)</h3>
<p>From the onset of this project, our team had three goals to measure success:</p>
<h4>1. The metrics database must be operationally dependable, affordable, and fast.</h4>
<p>The platform has huge implications on cost, because we need to store
requests for 10k+ clusters and read metrics off of them quickly. That’s around 1B request insertions per day. The first metrics prototypes used Elasticsearch, but we ran into performance issues once we started ran larger aggregations for bigger time slices. Our metrics engineer ultimately replaced Elasticsearch with Cassandra, which is extremely fast, consistent, and scales for long periods of time.</p>
<h4>2. The UI must be fast and responsive, and integrate easily our current javascript environment.</h4>
<p>Web users have high expectations for interface usability. All graphs need to load on the DOM quickly and respond to interaction smoothly. A lot of our UI is React-based, so we first tried building our graphs with open-source React libraries. During prototyping, we weren’t able to customize and simplify the axes and labels. A whole new React-based charting library was developed in-house to give us all the tools we needed to keep our charts Tuftian. Keep on the lookout for more news from our team, as we’ll be open-sourcing this library soon.</p>
<h4>3. The graphical design must be easy to read and relay information accurately.</h4>
<p>Charts must be easy to read — the design should avoid ‘getting in the way’ of answering any questions people have when they come to the Metrics page. In order to build something we’d be proud to look at, we took some lessons from Tufte to keep our designs straightforward and readable.</p>
<h3>Tufte as a baseline</h3>
<p>If you’ve studied data visualization, you know the work of statistician <a href="https://www.edwardtufte.com/tufte/" target="_blank" rel="nofollow">Edward Tufte</a>. Our charts’ design takes two directives from Tufte very seriously. First, <em>never lie</em>. Second, <em>don’t add chart junk</em>.</p>
<h4>1. Never Lie</h4>
<p>Every graph has a measurable lie factor. The biggest contributor is when vertical space leads to incorrect comparisons within the data set. If you don’t start with base zero, the graphs can look drastically different based on the smallest value or zero.</p>
<p>Lie factor can easily fly under the radar to an unsuspecting viewer. While most people are trained from a young age to read charts and graphs, they aren’t taught about designing them or how to choose the appropriate graph type given a data set. It’s easy to give a lot of absolute weight to graphs, because we’re taught that they’re a picture of some truth. But what truth are they telling? Some lies are easy to spot, like this price of crude oil graph:</p>
<p>Others are harder to determine, usually because the display is unconventional.</p>
<p>This is an example of our request count graph, with a y-axis minimum of 0:</p>
<p>Now, let’s take the same graph and push the minimum closer to the minimum of the data set (700):</p>
<p>Forcing a non-zero minimum not only diminishes the ratio of total requests running per minute, but also lies about the ratio of write to read requests.</p>
<p>We mitigate our lie factor by always starting at zero. This means that the ratio between two data points is neither (a) given undue significance or (b) visually downplayed.</p>
<h4>2. Don’t Add Chart Junk</h4>
<p>The ‘stuff’ around charts and graphs can become eye-catching distractions to the thing that matters the most: the data. Sometimes, it’s a function of a overeager designer looking to add more visual interest (something I’ve often been culpable of):</p>
<p>Others are more simple graphing and chart ideas — simple lines or grids that we assume come with the territory, but whose removal creates clarity and improves focus and readability.</p>
<p>In each of the Bonsai metrics graphs, the labels are discrete and the same grey color. We don’t use gridlines. The palette is a limited set that each graph inherits. Overlays convey focused information only on hover. When we do use color, it’s because it has a significant meaning that operations-focused engineers inherently understand. We chose to display and label 5xx requests as red in the Response Codes graph because, at least in the western world where we have the majority of customers, red equates to danger or problems. In the first design pass-throughs, using an info-centric color like blue for 5xx requests looked like a chart lie.</p>
<h4>Bonus: Perceptually Uniform Color Spaces</h4>
<p>Whilst deciding on colors, our metrics engineer stumbled upon an article about perceptually uniform colors spaces.</p>
<p>Perceptually uniform color spaces “are human-friendly alternatives to color spaces such as sRGB, and they are incredibly helpful for designers working in code” (Madsen). When using a color space like sRGB, you cannot programmatically render colors that progress linearly. See this example of a 10 step transition from green to blue using sRGB:</p>
<p>The first half looks almost indistinguishable from each other, while the blues look like tints of blue instead of a linear transition from green.</p>
<blockquote>
<p>This happens because the default sRGB color space (and any color model built on it like HSV and HSL) is irregular, which means that even though the rectangles have evenly spaced hue values, the corresponding effect is not linear to the human eye.</p>
<p>(Madsen)</p>
</blockquote>
<p>If we instead use an even transition of hues, we get something like this:</p>
<p>In short, using color spaces that are perceptually uniform means that the graphs are easier to look at, account for color blindness, and ensure that any bars or boxes have <a href="https://usecontrast.com/guide" target="_blank" rel="nofollow">appropriate color contrast</a>. Our performance heatmap uses the <a href="https://github.com/bpostlethwaite/colormap" target="_blank" rel="nofollow">colormap npm package</a>, which is used to generate the palette using the viridis colormap (an open-source perceptually uniform colormap by Matlab, <a href="https://matplotlib.org/users/colormaps.html" target="_blank" rel="nofollow">https://matplotlib.org/users/colormaps.html</a>):</p>
<p>Perceptually uniform color is incredibly useful for any designer, and we really suggest <a href="https://programmingdesignsystems.com/color/perceptually-uniform-color-spaces/" target="_blank" rel="nofollow">reading more about it</a> and watching <a href="https://youtu.be/xAoljeRJ3lU" target="_blank" rel="nofollow">this lecture</a>. It makes all graphs better.</p>
<h4>What’s Next</h4>
<p>Metrics will continue to grow and become more powerful. Feature additions include additional timezone selection, date scrubbing, stateful links for collaboration, and request filters.</p>
<p>We’d love to hear any and all feedback, so let us know how you’re using your cluster metrics by dropping us a line at <a href="mailto:info@bonsai.io">info@bonsai.io</a>.</p>
<p>Happy searching, and happy holidays!</p>
<p>— ❤️ The Bonsai Product Team</p>
<hr>
<h4>Bibliography</h4>
<p>Madsen, Rune. “Programming Design Systems.” Programming Design Systems, programmingdesignsystems.com.</p>
<p>Tufte, Edward Rolfe. The visual display of quantitative information. Graphics Press, 2001.</p></div>

How We Built & Designed Operational Metrics

<div><p>Our operational uptime numbers from October to December of 2017.</p>
<p>*Virginia Incident in November:</p>
<ul>
<li><a href="https://status.bonsai.io/incidents/rz5tf1ms1pgm" target="_blank" rel="nofollow">Performance issues for some users in US-East</a></li>
</ul>
<p>See our entire uptime history here:</p>
<ul>
<li><a href="https://status.bonsai.io" target="_blank" rel="nofollow">Bonsai Status</a></li>
</ul></div>

2017 Q4 Uptime Report

<div><p>*Sydney Incident in August:</p>
<ul>
<li><a href="https://status.bonsai.io/incidents/vnnx8jz6628z" target="_blank" rel="nofollow">SSL Outage</a></li>
</ul>
<p>See our entire uptime history here:</p>
<ul>
<li><a href="https://status.bonsai.io" target="_blank" rel="nofollow">Bonsai Status</a></li>
</ul></div>

2017 Q3 Uptime Report

2017 Q2 Uptime Report

<div><p>2017 has started with a nearly perfect Q1. We&#8217;ve continually worked to stabilize our systems and report the numbers that display this improvement.</p>
<p>Our systems were largely unaffected by <a href="https://aws.amazon.com/message/41926/" target="_blank" rel="nofollow">Amazon&#8217;s February S3 Outage</a>. The only impact to Bonsai was a temporary suspension of automated backups. Clusters continued to serve requests normally, so no downtime was reported for February. This fact was referenced on <a href="https://status.bonsai.io/incidents/4n8ws6696bnm" target="_blank" rel="nofollow">our status page</a>.</p>
<p>There were two short-lived events in March affecting overall uptime. These were both proximately due to high load on hardware serving primarily free and staging clusters. Our response was to add capacity and migrate clusters to more evenly distribute the load.</p></div>

2017 Q1 Uptime Report

<div><p>This past week, a <a href="http://www.computerworld.com/article/3157766/linux/mongodb-ransomware-attacks-and-lessons-learned.html" target="_blank" rel="nofollow">widely-reported ransomware attack</a> was carried out against tens of thousands of MongoDB databases, and the attackers also targeted <a href="http://www.pcworld.com/article/3157417/security/after-mongodb-ransomware-groups-hit-exposed-elasticsearch-clusters.html" target="_blank" rel="nofollow">several hundred</a> Elasticsearch instances for good measure. Those affected were given a distressing ultimatum: pay Bitcoin to the attackers, or have their data deleted. To add further insult, many who paid the ransom did not even receive their data in return.</p>
<p>If you&#8217;re a Bonsai customer, rest assured: your clusters are perfectly safe.</p>
<p>This particular attack targeted databases that are open to the internet, and totally unsecured. Attackers would crawl public IP addresses en masse in order to find such services. By automating the collection, probing, and attack of MongoDB and Elasticsearch instances, hackers were able to ransom a significant amount of data in a very short amount of time.</p>
<p>As anyone with a public-facing SSH service on port 22 can attest, this kind of attack is not uncommon. With services that can automate the scanning of the entire IPv4 space, managing the security of your data is an ongoing concern for anyone hosting such services. And, unfortunately, <a href="http://thenextweb.com/insider/2017/01/08/mongodb-ransomware-exists-people-bad-security/" target="_blank" rel="nofollow">most people are not particularly great</a> at this kind of operational security.</p>
<p>At Bonsai, it&#8217;s our job to anticipate these kinds of problems, and make opinionated architectural design decisions to help prevent them. Here&#8217;s a small sample:</p>
<ol>
<li>All Bonsai clusters are protected with HTTP authentication by default. These credentials are randomly generated, making them resistant to dictionary attacks or password reuse between services. All traffic to your cluster must have correct credentials, or it will be rejected with an HTTP 401 error. This protects against unauthorized access, including the kind of data exfiltration and modification used in this kind of attack.</li>
<li>Furthermore, we support secure TLS encryption by default for all clusters. This helps to further ensure the safety and security of your credentials by transmitting them over a secure channel. Granted, MITM and packet sniffing techniques weren&#8217;t used in this attack, but good security is designed in layers.</li>
<li>Bonsai cluster URLs have a randomly-generated hostname, and are routed through a specially designed layer 7 proxy, custom built for routing and load balancing Elasticsearch cluster traffic. This all but eliminates scanning by inflating the search space to an impractically large size. Your unique cluster URL is one of trillions of possibilities, and even if someone were to guess it perfectly, they can&#8217;t get any further without guessing the password.</li>
</ol>
<p>We have also previously published a detailed blog entry covering a few other tidbits about <a href="https://bonsai.io/2015/10/26/bonsai-security" target="_blank" rel="nofollow">how we handle security for Elasticsearch</a>, if you&#8217;re interested in some light reading on the subject.</p>
<p>This most recent attack underscores that one of the most important thing our users can do to protect their data is to treat the cluster URL like a password. It&#8217;s sensitive information and access to it should follow the principal of least privilege. Also, we provide the option of <a href="https://docs.bonsai.io/docs/managing-your-cluster-url-and-credentials#rotating-your-url-credentials" target="_blank" rel="nofollow">rotating the randomly-generated credentials</a>, which users should do regularly (and especially if there is reason to suspect that their full URL has been leaked). For detailed information about your cluster URL, see our thorough documentation here:</p>
<ul>
<li><a href="https://docs.bonsai.io/docs/managing-your-cluster-url-and-credentials" target="_blank" rel="nofollow">Managing your Cluster URL and Credentials</a></li>
</ul>
<p>We take security very seriously and are constantly monitoring for and responding to potential threats. It would be much easier to design an architecture that pushed some of these decisions onto our customers. But we know that our customers appreciate our service because of the opinionated expertise that we integrate directly into our systems. We take every precaution and are opinionated about our infrastructure as a result.</p>
<p>Hopefully you&#8217;ve found this insightful! Both for a few specific design points and policies, as well as for the overall sense of responsibility and design philosophy with which we approach the design of our service.</p>
<p>If you ever have any concerns or questions about how we&#8217;re keeping your data safe, or what you can do to beef up your own security, please don&#8217;t hesitate to reach out!</p></div>

Bonsai Unaffected by Recent Ransomware Attack

<div><p>October was a nearly perfect month except for a widespread .io outage on Oct 28th. We wrote a detailed log of the incident, including what we did to prevent us from being affecting from such an outage, here:</p>
<ul>
<li><a href="https://status.bonsai.io/incidents/9klg801sg1hd" target="_blank" rel="nofollow">Intermittent DNS resolution errors impacting .io TLD: Incident Report for Bonsai</a></li>
</ul>
<p>Our ops team that day really went above and beyond to make sure impact was minimal, and we&#8217;re looking forward to a great year with improved performance to report even better numbers in 2017.</p></div>

2016 Q4 Uptime Report

<div><p>For every quarter we report on the uptime for production-grade clusters on our shared tier. Interested in how we calculate uptime? See our in-depth breakdown here:</p>
<ul>
<li><a href="/2014/10/02/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as an Elasticsearch Hosting Provider </a></li>
</ul></div>

2016 Q3 Uptime Report

<div><p>Getting started with Bonsai Elasticsearch and Ruby on Rails is incredibly quick and simple to do. [Sign Up]({{ site.bonsai_app_url }}/signup) for your free account we&#8217;ll take it from there!</p>
<h3>Creating Your First Bonsai Cluster</h3>
<p>Once you&#8217;ve signed up for your free account on Bonsai, you&#8217;ll be prompted to create a free developer cluster. Give your cluster a name and pick a region. The name helps you to distinguish between different applications and the region is how we manage our infrastructure on Amazon Web Services. Click on the &#8220;Create Cluster&#8221; button and voila, you&#8217;ve created your very first Bonsai cluster!</p>
<h3>Our Example Rails App</h3>
<p>Our example Rails app will provide you a good outline to follow along. You can access the Github repo here:</p>
<ul>
<li><a href="https://github.com/omc/bonsai-rails-example" target="_blank" rel="nofollow">https://github.com/omc/bonsai-rails-example</a></li>
</ul>
<p>In our example app, Bookfaces, we&#8217;ll have a list of books that are searchable via Book Title, Author, Publisher and Genre. You can see the app in action here:</p>
<ul>
<li><a href="https://agile-anchorage-88938.herokuapp.com/" target="_blank" rel="nofollow">https://agile-anchorage-88938.herokuapp.com/</a></li>
</ul>
<p>So now that we have our example code (or a Rails project of your own) let&#8217;s go ahead and get things ready for Bonsai.</p>
<p>###Connecting Your Rails App to Your Bonsai Cluster</p>
<p>Bonsai clusters are created with unique URLs designed for secure, authenticated access to the Elasticsearch API. To avoid security concerns we strongly recommend adding your Bonsai Elasticsearch URL to your environment as a variable, you can do so by running:</p>
<pre><code>  export BONSAI_URL="&lt;YOUR BONSAI URL&gt;"
</code></pre>
<p>You can learn more about your Bonsai URL here:</p>
<ul>
<li><a href="https://docs.bonsai.io/docs/managing-your-cluster-url-and-credentials" target="_blank" rel="nofollow">https://docs.bonsai.io/docs/managing-your-cluster-url-and-credentials</a></li>
</ul>
<p>Now let&#8217;s add the following gems to your Gemfile:</p>
<pre><code class="language-ruby">  gem 'elasticsearch-rails'
  gem 'elasticsearch-model'
  gem 'bonsai-elasticsearch-rails'
</code></pre>
<p>Run <code>bundle install</code> and we&#8217;ll move on to the next step.</p>
<p>###Deploying to Heroku</p>
<p>As with all things related to the development process, deploy early and often. It’s a good idea to double check that you have completed all the steps above. For our example app we used Heroku to deploy. For step by step instructions as to how you can deploy your rails app with Heroku, please consult their excellent documentation:</p>
<ul>
<li><a href="https://devcenter.heroku.com/articles/getting-started-with-rails4" target="_blank" rel="nofollow">https://devcenter.heroku.com/articles/getting-started-with-rails4</a></li>
</ul>
<p>Once you have deployed to Heroku you can add the Bonsai add-on from the Overview tab.</p>
<p>###Inserting Data</p>
<p>Our next step will allow us to add data to the cluster as well as how to do a local database import. Let’s start with populating our local app first. For convenience sake, we will use the Faker gem to help populate db. Here’s a link to the Faker gem documentation:</p>
<ul>
<li><a href="https://github.com/stympy/faker" target="_blank" rel="nofollow">https://github.com/stympy/faker</a></li>
</ul>
<p>Let’s add the Faker gem to the gemfile:</p>
<pre><code class="language-ruby">  gem 'faker'
</code></pre>
<p>and run our <code>bundle install</code>. Now that we’ve got the Faker gem installed, we can start adding some sample data. Take a look at our <code>schema.rb</code> and <code>seeds.rb</code>. Files in our sample code if you need some guidance in how to structure those files.</p>
<p>Run the following commands to set up your local db, note that in Rails 5 we can now use the rails command instead of rake.</p>
<pre><code class="language-ruby">  rails db:create
  rails db:migrate
  rails db:seed
</code></pre>
<p>Now run <code>rails s</code> and check out all the new books for Bookfaces! Now that we know how to populate our local db, let’s proceed to getting our models ready for indexation.</p>
<p>###Get Your App Ready to Index Models You&#8217;d Like to be Searchable</p>
<p>We&#8217;ll want to prepare our models to be indexed and searchable by Bonsai. Take a look at <code>app/models/book.rb</code>, where we include our <code>Elasticsearch::Model</code></p>
<pre><code class="language-ruby">  class Book &lt; ApplicationRecord
    include Elasticsearch::Model

    belongs_to :author
    belongs_to :publisher
    belongs_to :genre

    ### View the entirety in the GitHub repo: https://github.com/omc/bonsai-rails-example/blob/master/app/models/book.rb
  end
</code></pre>
<p>We&#8217;ll want to do this for all models that we want indexed. It&#8217;s also advisable that we create rake tasks to import documents, it will allow you to index searchable records via rake tasks. Plus, it’s important to to have reliable and easily executable indexing scripts should you need to need to duplicate this process or have a backup at hand. You can do this by creating a file <code>lib/tasks/elasticsearch.rake</code> and including:</p>
<pre><code class="language-ruby">  require 'elasticsearch/rails/tasks/import'
</code></pre>
<p>Keep in mind you will need to explicitly create your index before you can put documents there, since Bonsai does not support &#8220;lazy&#8221; index creation. The best way to create your index is using the rake task we set up in the prior step. The first time you index your content, you&#8217;ll want to set the FORCE parameter:</p>
<pre><code class="language-ruby">  rake environment elasticsearch:import:all FORCE=y
</code></pre>
<p>The FORCE parameter will explicitly delete and recreate indices, so you probably don&#8217;t want to use it frequently. Subsequent indexing tasks can drop this parameter.</p>
<p>The alternative way to do it would be to create your indices with curl. So in our Book model, use the following command to create the index:</p>
<pre><code class="language-shell">  curl -XPUT http://username:password@redwood-12345.us-east-1.bonsai.io/books
</code></pre>
<p>###Done!</p>
<p>You have created your Bonsai cluster, added your cluster URL to your App, indexed the models and now can deploy to the service of your choice! For our example we used Heroku, where you can install the <a href="https://elements.heroku.com/addons/bonsai" target="_blank" rel="nofollow">Bonsai Add-on</a>.</p>
<p>Please feel free to give us feedback or ask questions at <a href="mailto:info@omc.io">info@omc.io</a> or on Twitter [@{{ site.author.twitter }}](https://twitter.com/{{ site.author.twitter }}).</p></div>

Rails 5 with Bonsai

<p>As your operations team, we at Bonsai take uptime—and reporting it—seriously. We&#8217;ve had another great quarter, and look forward to reporting even better numbers throughout the rest of 2016.</p>

2016 Q2 Uptime Report

<div><p>We’re excited to announce that we now support ES 2.2.2 in addition to 1.7.5 and 1.5. We’ve spent quite a bit of time improving our infrastructure to ensure 2.x will be a stable, blazing-fast build. Two weeks ago we shipped a button that allows anyone to upgrade their cluster to 2.2.2:</p>
<p>Because of backwards incompatibility with 2.x, clusters on 1.7 won’t be able to upgrade if they have data on them. If you have active shards, you will see a warning message and your button will be disabled:</p>
<p>So, make sure to clear your cluster! You can do this by running <code>curl -XDELETE "BONSAI_URL"</code></p>
<p>Elasticsearch v5 should be ready by the end of the year, and we don&#8217;t expect any updates to 1.x after this release. In early January 2017 we&#8217;ll phase out support for 1.x on non-dedicated plans, so we recommend all of our users to start their process of moving off of 1.x and into 2.x.</p></div>

Now Supporting 2.x!

<div><p>As your operations team, we at Bonsai take uptime (and reporting it) seriously. We&#8217;ve had a fantastic first quarter and look forward to reporting even better numbers in the coming quarters.</p>
<p>If you&#8217;re interested, you can read more about how we measure uptime here: <a href="/2014/10/02/measuring-availability-as-an-elasticsearch-hosting-provider">Measuring Availability as a Service Provider</a>. We&#8217;ve added even more high-precision monitoring tools to our stack since that post was written, which we plan to write about in upcoming posts.</p></div>

2016 Q1 Uptime Report

<div><p>###Access
You can find the logs for your cluster under the Logs tab in your cluster dashboard. Once you click on the tab, logs will populate in real time as requests are made to your cluster.</p>
<p>Each row in the table is one request made to your cluster. You will see the timestamp of the request, the type of request [GET, POST, PUT, UPDATE, DELETE], the URL of the request, the time in milliseconds it took for the request to complete, and the status.</p>
<p>####Color Indicators
The time in milliseconds and status have color indicators that let you know how long it took for the request to complete, and if it was successful.</p>
<p></p>
<p>The top 20 slow requests are shown on a separate tab within the Logs menu, and shows requests that take longer than 200ms. Any yellow or red requests will appear on this tab.</p>
<p>This feature is still in beta, so we would appreciate any feedback you have. Hit us up at <a href="mailto:support@bonsai.io">support@bonsai.io</a> or you can ping us on <a href="https://twitter.com/bonsaisearch" target="_blank" rel="nofollow">Twitter @bonsaisearch</a>.</p></div>

Introducing: Live Streaming Logs

<div><p>Whether you&#8217;re a small team of two or part of a large organization, there&#8217;s a good chance you will need to delegate responsibilities of managing your cluster at some point. Even if you just want to include someone else on email communication, or if you want to stop our emails going out to everyone on the group email you signed up with (gasp!), we can help.</p>
<p>##Heroku Contacts
For our Heroku add-on users, we built Heroku Contacts. Being a contact means that you will receive notifications from us about your cluster usage, and overage warnings if you&#8217;re going over your plan limits. You can manage your contacts in the Bonsai Dashboard, which you can access through Heroku, or by typing this in your terminal:</p>
<pre><code>heroku addons:open bonsai -a &lt;your-app-name-here&gt;
</code></pre>
<p>###Contacts page in the Bonsai Dashboard:</p>
<p>Your Heroku app owner email is automatically set up as a contact, as well as the email address of anyone who SSO&#8217;s into the dashboard. If your Heroku owner email is a group email, you will likely want to remove it as a contact and replace it with another email address so we don&#8217;t spam your whole company!</p>
<p>##Cluster Collaborators
For our direct sign-up users, we built Cluster Collaborators. A collaborator is able to login to the Bonsai Dashboard with their own user account, which means no more sharing passwords! To add a collaborator, click on the Collaborators tab from your Bonsai Dashboard.</p>
<p>###Collaborators page in the Bonsai Dashboard:</p>
<p>You have the option of adding either a regular collaborator or an administrator. A collaborator will receive notifications from us about the cluster, be able to view API credentials, and view other collaborators. An administrator has the additional ability to regenerate credentials, add and remove other collaborators, update the cluster plan, and deprovision the cluster.</p>
<p>Once you invite a collaborator, they will be emailed a link to sign up as a user for Bonsai, where they can choose to accept or decline your invitation.</p>
<p>After clicking &#8216;Accept&#8217;, they will be able to see the cluster on their cluster dashboard. A cluster that you are a collaborator on is shown with a collaborator symbol in the upper right hand corner.</p>
<p>##Go Forth, and Collaborate!
We loved building these tools, and we hope you enjoy using them! If you have any suggestions for improvements or you have a business need for something else, hit us up at support@bonsai.io or you can ping us on <a href="https://twitter.com/bonsaisearch" target="_blank" rel="nofollow">Twitter @bonsaisearch</a>.</p></div>

Teamwork Makes the Dream Work

<div><p>Sometimes we’re asked if we support Logstash. The relationships between Elasticsearch, Logstash and Kibana (often referred to as the ELK stack) can sometimes foster a confusing mental model, and a simple &#8220;yes&#8221; from our support staff probably isn’t enough to help users get up and running quickly. In this post, we’d like to unveil some of the mystery behind why the ELK stack is so popular.</p>
<p>We whipped up a quick example showing how to set up and use Logstash with Bonsai Elasticsearch and a Kibana 4 instance. This example takes a real world use case and walks the reader through the process of setting up Logstash to pass a particular type of event to Bonsai for indexing, where it can then be analyzed in Kibana. The intent is to help new users get up to speed quickly, without needing to do a lot of Googling for answers.</p>
<p>###So what <em>is</em> Logstash?</p>
<p>Logstash is an awesome tool for log parsing. Logstash can accept inputs from a number of different sources, apply a series of filters to the incoming data, then pass the results to a data store. In combination with Elasticsearch (for data storage, aggregation and retrieval) and Kibana (for analysis and visualization), users can get a very nice high-level view of a data set. It can do many other things, but that is its primary function.</p>
<p>##A real world example</p>
<p>In this example, we&#8217;ll look at the SSH daemon running on an Ubuntu server. SSH logs, particularly failed login attempts, can be extremely useful in investigating break-in attempts, identifying unusual traffic patterns, and performing regular security audits. It&#8217;s part of a proactive security policy, and critical if you&#8217;re a sysadmin in charge of dozens or hundreds of servers.</p>
<p>We could use other input contexts, but since the ‘net is rife with automated attack traffic, we think security is particularly interesting. If you have a machine connected to the Internet, you will eventually be hit by someone. Hackers use bots to scan the IPv4 space looking for particular ports &#8212; 21 for FTP, 22 for SSH, 23 for Telnet, and so on — and when a server responds to a request on a given port, the bot will try everything it can to log in to the system.</p>
<p>Generally, the bot doesn’t know anything about the target, so it just tries a bunch of random usernames and passwords. By studying this behavior, we can perform some counterintelligence gathering on the hacker community and use the ELK stack to aid in our reconnaissance. For another fun read, see how the developer of Elastichoney uses a <a href="http://jordan-wright.com/blog/2015/05/11/60-days-of-watching-hackers-attack-elasticsearch/" target="_blank" rel="nofollow">similar approach</a> to learn how hackers try to exploit Groovy scripting vulnerabilities in Elasticsearch.</p>
<p>###The Game Plan</p>
<p>We&#8217;ll pass the system logs into Logstash, run some filters to make sure we&#8217;re only seeing what we want, send the results along to Bonsai, and finally, we&#8217;ll examine the data using Kibana.</p>
<p>For the purposes of this example, we&#8217;ll make a few assumptions:</p>
<ol>
<li>
<p>SSH activity is being logged to <code>/var/log/auth.log</code>. This is the default setting for Amazon&#8217;s Ubuntu images, which have not (yet) switched over to systemd. Although you can use systemd with Logstash if you like.</p>
</li>
<li>
<p>You have the ability to install Logstash on the server(s), and have already done so. There are ways to do it with a package manager, and Elastic offers Logstash as a .deb on their downloads page, so it can be pretty easily installed. Read their documentation if you need more help getting it installed wherever you&#8217;re trying to use it.
<strong>In preparing this example, I downloaded the .deb to an Ubuntu machine, so the instructions are based on that. If you use a package manager or another OS, some files might end up in other places and some commands may need minor tweaks.</strong></p>
</li>
<li>
<p>You already have a Bonsai Elasticsearch cluster provisioned. In this example, the URL of that cluster will be <code>http://user123:pass456@logstash-12345.us-east-1.bonsai.io</code></p>
</li>
<li>
<p>You have a Kibana 4 instance set up somewhere. Need more info on this? <a href="http://docs.bonsai.io/docs/using-kibana-with-bonsai" target="_blank" rel="nofollow">We&#8217;ve got you covered</a>.</p>
</li>
</ol>
<p>###Laying the groundwork</p>
<p>Logstash can accept streams from a large number of inputs. They also have an ever-expanding list of input plugins to support more streams and file types. One requirement that all inputs have in common is permissions; that is, Logstash must have permission to access the stream in order to read from it. This probably sounds trivial, but it is a pretty common &#8220;gotcha,&#8221; especially when dealing with daemon log files.</p>
<p>When Logstash is installed in Debian, the OS creates a user called <code>logstash</code>. This is the user that will need to be able to access the log file. We can see the permissions of the auth.log file with <code>ls -lAh /var/log/auth.log</code>:</p>
<pre><code>-rw-r----- 1 syslog adm 6.4M Jan 20 17:24 /var/log/auth.log
</code></pre>
<p>We don&#8217;t want to change these permissions, because that&#8217;s not a best practice and could be dangerous. But what we can do is add the <code>logstash</code> user to the <code>adm</code> group. The <code>adm</code> group typically grants members access to read log files and nothing else. Joining this group would give the Logstash process the ability to read the file, but not modify it in any way. This is the minimum necessary access to do the job, and it doesn&#8217;t expose security details to other programs/users on the server, so it&#8217;s the route we&#8217;ll take:</p>
<pre><code>$ sudo usermod -G adm logstash
</code></pre>
<p>The last thing we need to do is create an index on our Bonsai cluster for Logstash to send data. We&#8217;ll call this index <code>logstash</code>:</p>
<pre><code>$ curl -XPUT http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash
</code></pre>
<p>###Configuring Logstash</p>
<p>Now that we know Logstash will be able to read the input stream, we&#8217;ll need to configure it. Go ahead and edit <code>/etc/logstash/conf.d/logstash.conf</code>:</p>
<pre><code>$ sudo nano /etc/logstash/conf.d/logstash.conf
</code></pre>
<p>Copy/paste the settings below (making sure to substitute the correct values for your cluster URL):</p>
<pre><code>input {
    file {
        path =&gt; "/var/log/auth.log"
        start_position =&gt; beginning
        sincedb_path =&gt; "/dev/null"
    }
}

output {
    elasticsearch {
        hosts     =&gt; ["logstash-12345.us-east-1.bonsai.io:443"]
        user      =&gt; "user123"
        password  =&gt; "pass456"
        ssl       =&gt; true
        index     =&gt; "logstash"
    }
}
</code></pre>
<p>Finally, make sure that Logstash has permission to read its own config file:</p>
<pre><code>$ sudo chown logstash /etc/logstash/conf.d/logstash.conf
</code></pre>
<p>###Test it out!</p>
<p>We&#8217;ve just completed the bare minimum necessary to get Logstash up and running. We&#8217;ve given Logstash access to a log file, defined it as an input stream, created a place in Elasticsearch for the stream to go, and defined that index as an output. Now all we need to do is run it and make sure it works. When I installed the .deb for Logstash, the one thing it didn&#8217;t do was symlink the binary, so I&#8217;ll just use the full path. YMMV:</p>
<pre><code>$ /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/logstash.conf
Logstash startup completed
</code></pre>
<p>Now, if everything worked as expected, I should be able to see the data in my cluster:</p>
<pre><code>$ curl "http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash/_search?pretty&amp;size=1"
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "hits" : {
    "total" : 1975,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "logstash",
      "_type" : "logs",
      "_id" : "AVJdzwbLfbt0kfwq_Tzs",
      "_score" : 1.0,
      "_source":{"message":"Jan 17 04:30:52 ip-XXX-XXX-XXX-XXX sshd[12170]: Invalid user cms from XXX.XXX.XXX.XXX","@version":"1","@timestamp":"2016-01-17T11:30:52.000Z","host":"localhost","path":"/var/log/auth.log"}
    } ]
  }
}
</code></pre>
<p>Neat! Let&#8217;s look at this in Kibana:</p>
<p>So, a partial success. We can see data in the index, and it is coming out of the auth.log file as expected. But the timestamp associated to the event is not when it occurred in real time, but when it was indexed. Not very helpful.</p>
<p>This is a pretty common gotcha as well. I included it here to underscore it as a pitfall (and maybe save everyone a few support tickets). It&#8217;s easily addressed, we just need to do a bit more work.</p>
<p>##A more complex setup</p>
<p>Let&#8217;s go ahead and delete the logstash index on our cluster:</p>
<pre><code>$ curl -XDELETE http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash
</code></pre>
<p>With that out of the way, we can focus on refining our Logstash configuration. We&#8217;ll use some of the more popular filters to extract the most important details, and add those to fields that we can later query and display with Kibana.</p>
<p>Before we do that, let&#8217;s think about what our goal is here. We want to look for failed login attempts over SSH; we&#8217;re not interested in <code>cron</code> tasks or <code>sudo</code> sessions, or anything else that commonly gets logged to auth.log. Let&#8217;s take a look at a sample line:</p>
<pre><code>Jan 17 04:30:52 ip-XXX-XXX-XXX-XXX sshd[12170]: Invalid user cms from XXX.XXX.XXX.XXX
</code></pre>
<p>What kind of data do we have there? I see a date (in a non-standard format, boo!), a private IP address (useful for identifying what server is being hit), as well as a probe at a user name (&#8220;cms,&#8221; really?) and an IP address from where the attempt originated.</p>
<p>So we want to disregard lines that don&#8217;t match this pattern, and extract some useful information from lines that do match the pattern. We&#8217;ll also have to do something about that date &#8212; it will need to be in a standard format to be useful in Kibana. It would also be cool if we could perform some further analysis on the originating IP address, to see if there are any interesting trends. Who&#8217;s hitting us, and where are they coming from, would be good to know.</p>
<p>Fortunately, Logstash has filters that allow us to do all of this:</p>
<ul>
<li>
<p><strong>Grok.</strong> The <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html" target="_blank" rel="nofollow">grok filter</a> allows you to use regular expressions to parse log entries. It has a number of built-in patterns, and allows you to arbitrarily define your own. This is how we&#8217;ll extract the lines of interest.</p>
</li>
<li>
<p><strong>Date.</strong> The <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html" target="_blank" rel="nofollow">date filter</a> allows you to extract timestamps from your logs, then using those as the Logstash timestamp for the event. This will fix that &#8220;single bar&#8221; problem in Kibana.</p>
</li>
<li>
<p><strong>GeoIP.</strong> The <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html" target="_blank" rel="nofollow">geoip filter</a> allows you to take an IP address, run it through the GeoLiteCity database, and save geographic information to a new field. We can use this info in Kibana to make a heat map of where the nefarious traffic is originating.</p>
</li>
</ul>
<p>Update the Logstash configuration file to this, again changing the Elasticsearch output to match your Bonsai URL :</p>
<pre><code>input {
    file {
        path =&gt; "/var/log/auth.log"
        start_position =&gt; beginning
    }
}

filter {
    grok {
        match =&gt; { "message" =&gt; "(?&lt;logdate&gt;%{MONTH} %{HOUR}.%{MINUTE}.%{SECOND}) (?&lt;private_ip&gt;ip-[0-9]{1,3}-[0-9]{1,3}-[0-9]{1,3}-[0-9]{1,3}) sshd\[[0-9]{1,}\]: (?&lt;sshd_message&gt;Invalid user (?&lt;invalid_user&gt;[a-z0-9]{1,}) from (?&lt;invalid_ip&gt;%{IP}))"}
    }
    if "_grokparsefailure" in [tags] {
      drop { }
    }
    date {
        match =&gt; [ "logdate", "MMM dd HH:mm:ss" ]
    }
    geoip {
      source =&gt; "invalid_ip"
      target =&gt; "geoip"
      database =&gt; "/etc/logstash/conf.d/GeoLiteCity.dat"
    }
}

output {
    elasticsearch {
        hosts     =&gt; ["logstash-12345.us-east-1.bonsai.io:443"]
        user      =&gt; "user123"
        password  =&gt; "pass456"
        ssl       =&gt; true
        index     =&gt; "logstash"
    }
}
</code></pre>
<p>We&#8217;ll need to make sure that the GeoLiteCity.dat file is in its proper place:</p>
<pre><code>$ cd /etc/logstash/conf.d/
$ sudo curl -O "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz"
$ sudo gunzip GeoLiteCity.dat.gz
$ sudo chown logstash GeoLiteCity.dat
</code></pre>
<p>Finally, we need to make one little tweak to our index before running Logstash. As of this writing, there is a bug where Elasticsearch will dynamically map the <code>geoip.location</code> as a <code>float</code> instead of a <code>geo_point</code>, which makes it impossible for Kibana to create a tile map with the information. We just need to explicitly set this field up first. We&#8217;ll do that when we recreate the index:</p>
<pre><code>$ curl -XPUT http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash -d '{"mappings":{"logs":{"properties":{"geoip":{"properties":{"location":{"type":"geo_point"}}}}}}}'
</code></pre>
<p>Let&#8217;s spin up Logstash again and see what happens:</p>
<pre><code>$ logstash agent -f /etc/logstash.conf
Settings: Default filter workers: 2
Logstash startup completed
</code></pre>
<p>If we check on the data again, we see:</p>
<pre><code>curl "http://user123:pass456@logstash-12345.us-east-1.bonsai.io/logstash/_search?pretty&amp;size=1"
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "hits" : {
    "total" : 144,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "logstash",
      "_type" : "logs",
      "_id" : "AVJggJNEfbt0kfwq_U0P",
      "_score" : 1.0,
      "_source":{"message":"Jan 17 04:30:52 ip-XXX-XXX-XXX-XXX sshd[12170]: Invalid user cms from XXX.XXX.XXX.XXX","@version":"1","@timestamp":"2016-01-17T11:30:52.000Z","host":"localhost","path":"/var/log/auth.log","logdate":"Jan 17 04:30:52","private_ip":"ip-XXX-XXX-XXX-XXX","sshd_message":"Invalid user cms from XXX.XXX.XXX.XXX","invalid_user":"cms","invalid_ip":"XXX.XXX.XXX.XXX","geoip":{"ip":"XXX.XXX.XXX.XXX","country_code2":"CN","country_code3":"CHN","country_name":"China","continent_code":"AS","region_name":"12","city_name":"Wuhan","latitude":30.580099999999987,"longitude":114.27339999999998,"timezone":"Asia/Shanghai","real_region_name":"Hubei","location":[114.27339999999998,30.580099999999987]}}
    } ]
  }
}
</code></pre>
<p>Nice! So our filters worked. We&#8217;ve dropped from almost 2K lines to just 144. That&#8217;s a big reduction in noise. Let&#8217;s look in Kibana:</p>
<p>Great, so the timestamps are correct as well. We can see a big spike in activity at one point. From a security standpoint, that would be something to investigate. Ideally, it would also be something we could wire up to an alarm and page a sysadmin to let them know there might be an intrusion attempt.</p>
<p>But let&#8217;s take it further. We can create a whole dashboard around this. We can create a Tile Map using the <code>geoip.location</code> field and see where these requests are originating. We can also look at stats like the user account that was probed; this might help us to determine if these attacks are some random bot or script kiddie, or if there is a more sophisticated adversary targeting specific systems.</p>
<p>I&#8217;m sure you could think of even more ways to use this data. For example, imagine that the source data is coming from multiple servers in your network. You could see which ones are being hit hardest and take corrective action. You could set up an alert that notifies you if someone logs in from an IP address overseas &#8212; perhaps they&#8217;ve been compromised. There is a wealth of intelligence that can be gleaned from looking at a setup like this.</p>
<p>###Wrapping up</p>
<p>People use Logstash for all sorts of things. It really is an amazing tool, and we wish it had a larger following. If you&#8217;re using Logstash and want to host your Elasticsearch cluster with Bonsai, we&#8217;d love to work with you. We fully support both Logstash and Kibana, so&#8230;</p>
<p>##Give us a shout!</p>
<p>Got a cool use case you&#8217;d like to share? Maybe an anecdote or useful tip? Did you find this article tedious and boring? We&#8217;d like to hear about that too. Hit us up at support@bonsai.io and share. You can also ping us at <a href="https://twitter.com/bonsaisearch" target="_blank" rel="nofollow">@bonsaisearch</a> on Twitter if that&#8217;s something you&#8217;re in to.</p></div>

Logstash and Bonsai and bots, oh my!

<div><p>A t the core of Elasticsearch’s popularity is the ease and simplicity of setting up a cluster. And with hosted search options like <a href="https://bonsai.io" target="_blank" rel="nofollow">Bonsai.io</a>, powerful, multi-node clusters can be created instantly.</p>
<p>In a lot of ways, ease of use is both a blessing and a curse. When the learning curve isn’t a barrier to entry, it’s easy to start on a path that causes problems later. Many users find that decisions they made early in the product cycle regarding Elasticsearch don’t scale well in production. Some early decisions can become extremely painful or difficult to change later.</p>
<p>At Bonsai, we’ve had the opportunity to work directly with thousands of users around the world. We’ve had the unique experience of getting a peek into projects integrating Elasticsearch at scale, and helping people get the absolute most out of search. This experience has been instrumental in crafting some guidelines for users just starting out.</p>
<p>This is the guide a lot of users <em>wished</em> they’d had in the beginning.</p>
<h3>Why Focus on the Index? Why Not the Cluster?</h3>
<p>The short answer is that you should let your data decide the cluster specifications, not vice-versa. The index will determine how the data is stored and retrieved. If you haven’t thought through what your indices will look like and how they will be configured, you may end up underprovisioning the cluster (hurting performance) or overprovisioning the cluster (hurting your wallet).</p>
<p>This guide will help you understand and reason about the factors that will impact the quality of your search experience over the long term.</p>
<h3>Elasticsearch Has So Many Use Cases, How Relevant is this Advice to Me?</h3>
<p>People use Elasticsearch in so many ways. And each use case is going to benefit from a different set of settings and low level tweaks. However, these are typically changes that can be made at any time, in contrast to immutable settings like primary sharding. In other words, the fine tuning can happen a little later.</p>
<p>Instead, we’re going to focus on a couple of high level characteristics shared by all well-configured indices, and leave the more esoteric stuff for a later series.</p>
<h2>That said, the Ideal Elasticsearch Index is…</h2>
<ol>
<li>Backed by a database</li>
<li>Sharded intelligently</li>
<li>Replicated, but not overreplicated</li>
<li>Has well thought out data structures</li>
</ol>
<p>Let’s look into those in a bit more detail.</p>
<h3>Backing Elasticsearch with a Database</h3>
<p>Most developers have at least a casual understanding of what a database is: some software that stores data in a structured format, and allows applications to interact with that data using a structured query language. And Elasticsearch is basically the same thing, right?</p>
<p>While that is an accurate description of what Elasticsearch does, the devil is in the details. Database software aims for something called ACID compliance. ACID (Atomicity, Consistency, Isolation, Durability) is a set of properties that guarantee validity of data in the event of unexpected service interruptions.</p>
<p>An “ACID-compliant data store” is a fancy way of saying that the technology is highly resistant to corruption and data loss. Some examples of ACID-compliant data stores you may be familiar with: Postgres, SQLite, Oracle, MySQL (with InnoDB), and MongoDB (4.0+). Not listed: Elasticsearch.</p>
<p>Elasticsearch uses a search engine technology called <a href="https://lucene.apache.org/" target="_blank" rel="nofollow">Lucene</a>. Lucene is an information retrieval technology built for speed, not redundancy. It has a radically different architecture that gives it blazing fast performance, at the expense of being more susceptible to data loss.</p>
<p>Data loss can happen in a number of ways, you need to be able to recreate the data if needed. True, Elasticsearch has a snapshot/restore feature, but this process will only ever partially recover you in the event of data loss. Updates made between the most recent snapshot and outage will be lost unless you have another system in place to queue them. Snapshot/restore will also not help in the event of split brain, because there’s no mechanism for reconciling updates to each partition. Updates will just be lost.</p>
<p>This is why <em>every Elasticsearch index you create should be backed by an ACID-compliant data store</em>. The database should be the ultimate source of truth, from which the index is populated. If you ever lose the index — via node loss, outage, or simple fat-finger — you’ll be able to completely recover it.</p>
<h3>Intelligent Sharding</h3>
<p><em>Are you brand new to Elasticsearch? Start by reading <a href="https://docs.bonsai.io/docs/what-are-shards-and-replicas" target="_blank" rel="nofollow">this article on Elasticsearch shards and replicas</a>.</em></p>
<p>Every Elasticsearch index is composed of one or more shards. There are two types of shards: primary, and replica. Primary shards are responsible for managing updates to the index, while replica shards are simply copies on the primary and live on other nodes. <strong>The idea is that if a primary shard is taken offline, the replica will be able to fill the role and keep search from going down</strong>.</p>
<p>Shards are really just abstractions for Lucene indices. When an Elasticsearch index has several primary shards, it can be thought of having the data spread out over several different search engines. This is useful, for example, when you have millions of documents, because it allows queries to be served in parallel, which can save time and IO.</p>
<p>“Oh,” I hear you say, “well then I’ll just create my index with 100 primary shards and be done with it!” That certainly is one approach, but it’s a terrible one. Shards come with overhead; even empty shards require memory and disk. And <a href="https://docs.bonsai.io/docs/what-are-shards-and-replicas" target="_blank" rel="nofollow">tf-idf</a> relevance is calculated on a per-shard basis. Oh, and if you have more shards than CPU cores, they will end up fighting each other for system resources on every request.</p>
<p>In other words, this approach would give you the worst of all worlds unless you already have a lot of data and a lot of nodes.</p>
<p>Another thing to understand is that Elasticsearch uses a naive hashing algorithm to route documents to a given primary shard. This design choice allows documents to be randomly distributed in a reproducible way. This avoids “hot spots” that affect performance and overallocation. However, it has one major downside, which is that <em>the number of primary shards can not be changed after an index has been created. Replicas can be added and removed at will, but the number of primary shards is basically written in stone.</em></p>
<p><strong>This means that we need to have some sense of our scale before even creating the index.</strong> We need a shard scheme that makes sense for our present needs, but with enough headroom to scale up.</p>
<h3>Benchmarking is key</h3>
<p>Benchmarking a subset of your data before you decide on your production settings will give you a sense of how much data will be used per document, both on-disk and in memory. From there, you can use some heuristics to figure out a good launch point.</p>
<p>Something else to think about is the number of nodes you will be using, and what are their capabilities. <strong>A production cluster should not have fewer than three nodes, and should almost always have an odd number</strong>. With that in mind, here are some heuristics:</p>
<p><strong>If you don’t expect data to grow significantly, then:</strong></p>
<ul>
<li>One primary shard is fine if you have less than 100K documents</li>
<li>One primary shard per node is good if you have over 100K documents</li>
<li>One primary shard per CPU core is good if you have a couple million documents</li>
</ul>
<p>If you do anticipate significant growth, then you’ll want to balance your present needs with your expected needs. At a minimum, you’ll want a number of primary cores equal to the number of nodes you would scale to when horizontal scaling is required.</p>
<p>In other words, if you’re starting out on a three node cluster, create the index with at least five primary shards. If you have a large enough collection of documents, you could also go with the product of your current and future number of nodes (in this case, 3 nodes today x 5 nodes at scale-up=15 primary shards). That assumes that the nodes in your three node cluster have at least five CPU cores.</p>
<p>See why this is an interesting problem?</p>
<p>Thinking about your needs and planning a little ahead will go a long way in creating an Elasticsearch index that performs well over the long term.</p>
<h3>Replicating shards, but not Overreplicating them</h3>
<p>Replica shards are copies of primary shards that live on any node other than the primary. Like primary shards, replicas can serve search requests if needed. Unlike primary shards, replicas do not handle update operations; they simply receive updates from the primary at regular intervals. A nice benefit of this design is that replicas are also not subject to immutability like primary shards. <strong>You can add/remove replicas at any time.</strong></p>
<p>Replicas reduce stress on primary shards, and provide protection against data loss, node loss, network partitions, etc. However, too many replicas lead to wasted resources, because shards aren’t free. The cost-benefit ratio of replication gets worse with each new replica shard.</p>
<p><strong>The ideal Elasticsearch index has a replication factor of at least 1. This means for every primary shard — however many there may be — there is at least one replica. And the maximum number of replicas never exceeds (n-1), where n is the number of nodes in the cluster.</strong> (This is a common newbie mistake: creating 10 replicas on a three node cluster and then wondering why the cluster state is perpetually “yellow.” Where are all of those extra replicas supposed to go?)</p>
<p>How do you know how many replicas is “enough”? With a minimum of 1 and maximum of (n-1), what is the “sweet spot?” Incremental benchmarking is a good way to narrow in on that. Add replicas incrementally and measure performance improvements with each new replica. If improvements are not noticeable, replication is probably too high.</p>
<h3>Thinking through data structures</h3>
<p>What are “data structures,” and how are they pertinent to Elasticsearch? Elasticsearch has a concept known as <a href="https://www.tutorialspoint.com/elasticsearch/elasticsearch_mapping.htm" target="_blank" rel="nofollow">“mappings.”</a> Mappings to specify how data will be ingested and queried. Mappings define analyzer chains composed of tokenizers and filters; these allow data (and queries) to be tokenized and mutated before being used.</p>
<p>While a well considered mapping can improve your search results, it’s wise to <strong>start simply, benchmark, and iterate later.</strong> Tweaking mappings and then promoting them to production is easier than you think. That means you don’t have don’t index everything in your database. Don’t try to do All The Things™. Think about what features your application actually needs right now, and worry about new features later.</p>
<p>It’s also carefully considered: <strong>don’t use mappings that you don’t <em>fully</em> understand.</strong> Don’t just copy/paste from the Internet. Read the documentation, know what each component does and how it will affect performance and system resources. If you don’t know what your mappings look like, take an hour to look at them and comprehend how they will work with your data.</p>
<p>The Ideal Elasticsearch Index isn’t necessarily just implementing default data structures, but has <strong>mappings that were honed in small scale testing.</strong> A subset of production data can be used to benchmark the performance and resource demands of a mapping. Figure these things out before taking it to scale.</p>
<h4>Understanding how a mapping can affect performance</h4>
<p>If you’re familiar with the <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/pipes-and-filters" target="_blank" rel="nofollow">Pipe and Filter Pattern</a>, then this will make a lot of sense. But just in case, let’s look at the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-standard-tokenizer.html" target="_blank" rel="nofollow">standard tokenizer</a> for a moment. This component takes a text input and breaks it up using the Unicode Text Segmentation algorithm. The short explanation is that this splits the input up on whitespace. So an input like:</p>
<pre><code>It was the best of times, it was the BLURST of times
</code></pre>
<p>is broken up into an array like so:</p>
<pre><code>[It, was, the, best, of, times, it, was, the, BLURST, of, times]
</code></pre>
<p>The elements of this array are called tokens, and they can be filters out and mutated before reaching Lucene. We can use stopwords filters to get rid of tokens that aren’t likely to be relevant to a search:</p>
<pre><code>[best, times, BLURST, times]
</code></pre>
<p>Then we can pass these tokens through a lowercase filter to get rid of the all-caps, and then through a synonyms filter, which maps <code>blurst =&gt; worst</code>. That will leave us with:</p>
<pre><code>[best, times, worst, times]
</code></pre>
<p>These tokens are now stored in the inverted index along with the corresponding document ID, ensuring queries will be much more relevant. This pattern gives developers a lot of flexibility, however some seemingly innocuous choices will have enormous consequences at scale.</p>
<p>Why? Well, mappings are easy to understand conceptually, but can take time to master in practice. Here’s a classic example: <a href="https://en.wikipedia.org/wiki/N-gram" target="_blank" rel="nofollow">n-grams</a>. This is a filter which breaks up tokens into lots of sub-tokens, with set minimum and maximum sizes. Developers often like to use n-grams for substring matching at search time, and subsequently give the tokens a size from 1 to 100 (or something obnoxiously large).</p>
<p>To see why this is a problem, look at what a token like <code>“terrific”</code> turns into:</p>
<pre><code>["t", "e", "r", "r", "i", "f", "i", "c", "te", "er", "rr", "ri", "if", "fi", "ic", "ter", "err", "rri", "rif", "ifi", "fic", "terr", "erri", "rrif", "rifi", "ific", "terri", "errif", "rrifi", "rific", "terrif", "errifi", "rrific", "terrifi", "errific", "terrific"]
</code></pre>
<p>If you work it out, you find that there is a parabolic relationship between the token length and the number of n-grams that it generates:</p>
<p>Worse, this pattern is generated per token. A standard 500-token document will create 500 times more grams, much of them garbage. How many words can you think of with substrings that are also, unrelated word? “Scatter” contains the totally unrelated word “cat.” If a user searches for “cat,” a document with the term “scatter” could plausibly be a match in this scenario.</p>
<p>And if your gram size is 1, then you’re basically indexing every single character in the alphabet to every single document. Pro-tip: this is not a good way to get relevant results or use system resources intelligently. But it’s a great way to consume lots of disk and memory for no reason.</p>
<p>How about another example? Synonym matching is another feature that is easy to implement, and even easier to screw up. Synonyms allow developers to create rules that group or translate related tokens. You might have a rule that replaces “ram” with “memory,” or a rule that takes the token “cat” and expand it out to the tokens: “cat”, “kitty” and “kitten.”</p>
<p>Elasticsearch allows developers to perform synonym expansion at index time, query time, or both. Expanding at index time means that the tokens are being expanded before being written out to disk. This does lead to larger indices, and has the downside of needing to reindex every time you want to change them.</p>
<p>That said, many developers opt instead for query time expansion. The user searches for a term that the synonym filter matches and expands, and those new tokens become part of the Lucene query. But then in production, the users report lots of unexpected and irrelevant results.</p>
<p>Why? Because multi-word synonyms break phrase queries, and the tf-idf scores for rare terms lead to score boosting in ways that are not immediately obvious. The low level details of why this happens is well beyond the scope of this post, but needless to say, you could spend weeks wrangling with relevancy tuning because of an ill-considered choice when working out the mappings.</p>
<p>These are just a few examples; broader coverage of the kinds of subtle mistakes and inefficiencies that are introduced in early stage data structures would be a whole other series of articles. But it’s just a taste of what can happen when choices are made without understanding the consequences and trade offs.</p>
<h2>Conclusion</h2>
<p>Elasticsearch and search engines are incredibly broad topics, and mastery of these subjects can be incredibly difficult. Fortunately, the fundamentals are easy to grasp and implement. The Ideal Elasticsearch Index is a set of design principles common to most use cases.</p></div>

The Ideal Elasticsearch Index

<div><p>W
e’ve had Elasticsearch 1.7 out and running in production for a few months now, and, as expected, it’s operating like a champ. This is a minor update that doesn’t change user functionality, so we don’t expect any impact. Next week we’ll begin the process of upgrading all remaining 1.5 clusters onto ES 1.7, from Tuesday through Thursday. We will be posting updates on <a href="https://twitter.com/bonsaisearch" target="_blank" rel="nofollow">twitter</a>.</p>
<p>###Gain Improved Versions of Features Available Through Elasticsearch 1.6</p>
<p>Most user-facing changes to Elasticsearch in 1.6 and 1.7 have been minor or subtle additions to existing functionality, so we’re not expecting any notable changes for clusters being upgraded.</p>
<p>One such example from our friends over at <a href="http://www.wework.com" target="_blank" rel="nofollow">WeWork</a>: the <code>field_value_factor</code> function now lets you specify a default value for a field that is missing on a document. A handy improvement if you’re using <code>function_score_query</code> with inconsistent document values!</p>
<ul>
<li><a href="https://github.com/elastic/elasticsearch/pull/10845" target="_blank" rel="nofollow">elastic/elasticsearch#10845</a></li>
</ul>
<p>Other changes to ES 1.6 and 1.7 have been focused on security and operational improvements. While these kinds of improvements are always nice to have, they weren’t critical for us. The security fixes, in particular, do not affect our systems due to our architectural design choices and other preventative measures.</p>
<p>So if ES 1.5 is working out fine, why risk an upgrade? Good question. These newer versions definitely are better, even if subtly. We think it’s a core part of our value as a service provider to provide continuous managed upgrades to our customers, so that you can enjoy the full benefits of open source community built software.</p>
<p>Imagine, if the cost:benefit ratio had us putting this upgrade on the backburner, how much longer might a busy team focused on their app wait to manage their own upgrade!</p>
<p>###A Smoother Transition for Future Releases: 2.0</p>
<p>More importantly here, Elasticsearch 2.0 is around the corner, and that upgrade will not be so trivial. We’re laying the groundwork now to help prepare for that upgrade some time in the new year, and ES 1.6 and 1.7 also help to provide interim tooling toward that end. You will receive at least a month of notification for this update so you can test your app against this particular upgrade.</p>
<p>###How to opt-out of 1.7</p>
<p>If, out of an abundance of caution, you’d like to opt-out of the upgrade, that’s fine too! We can provide long-term version support for any Elasticsearch version on one of our dedicated cluster plans. Just <a href="mailto:support@bonsai.io">contact support</a> and let us know.</p>
<h2><strong>Recap:</strong></h2>
<ul>
<li>We’re upgrading everyone from 1.5 to 1.7</li>
<li>This will happen next week, Dec 22nd — Dec 24th</li>
<li>1.7 improves security and operations, so we don’t expect any user-facing impact</li>
<li>Upgrades to 2.0 are coming in the new year</li>
</ul></div>

Scheduled Maintenance Upgrade to Elasticsearch 1.7

<div><p>We sometimes hear from users concerned about their shard counts. They ask how they can instruct Elasticsearch to add multiple indices to a single shard. This is a surprisingly difficult question to answer in full because the terminology can confuse the issue quite a bit.</p>
<p>In Elasticsearch, every index maps to one or more shards, but the reverse is not true for some pretty sound technical reasons. However, changes to data mapping do afford users the ability to logically separate data on a single shard, thus producing the effect of multiple indices per shard for many use cases.</p>
<p>If that explanation sounds confusing, don&#8217;t worry. In this post, we&#8217;re going to lay out the issue and solution as easily as possible.</p>
<h2>The problem</h2>
<p><a href="https://en.wikipedia.org/wiki/Lucene" target="_blank" rel="nofollow">Lucene</a> is the Java-based information retrieval software that forms the backbone of Elasticsearch. Elasticsearch adds a few layers of abstraction on top of Lucene to provide features that are extremely useful to production applications: aggregations, analysis, distributed search, and more, all controlled through a well-documented and easy to use RESTful API. However, with these features comes a richer glossary of terms, with slightly more subtle definitions.</p>
<p>The biggest point of confusion for users is the word &#8220;index.&#8221; This word means different things, depending on the context. It can be used:</p>
<ul>
<li>As a noun in the context of Lucene</li>
<li>As a noun in the context of Elasticsearch</li>
<li>As a verb in the context of either Lucene or Elasticsearch</li>
</ul>
<p>Catch that? A Lucene index and an Elasticsearch index are very different things, even though Elasticsearch implements Lucene. In the context of Lucene, an index is basically a Lucene instance. In the context of Elasticsearch, an index is&#8230; well, read on.</p>
<p>I mentioned earlier that Elasticsearch offers a rich glossary, with terms like &#8220;shard,&#8221; &#8220;index,&#8221; and &#8220;type,&#8221; which are closely related but distinct concepts. So what distinguishes a shard from an index from a type in Elasticsearch? In the Elasticsearch lexicon:</p>
<ul>
<li>A shard is a single Lucene instance</li>
<li>An index is a logical namespace that points to one or more shards</li>
<li>A type is a piece of metadata used to logically distinguish between documents of different types within an index</li>
<li>A document in Elasticsearch is essentially a Lucene document with some injected metadata</li>
</ul>
<p>To distill this all down:</p>
<blockquote>
<p>An Elasticsearch shard is a Lucene index. An Elasticsearch index is a collection of one or more Lucene indices. An Elasticsearch type is a bit of metadata injected into the document which allows users to organize different types of documents within a single collection of Lucene indices.</p>
</blockquote>
<h2>Pedantry! Just tell me what to do!!</h2>
<p>Let&#8217;s say I have an application and I want to be able to search all of my users. My application has three kinds of users: administrators, moderators and customers. There are a couple of ways I could organize my data. I could create an Elasticsearch index for each type:</p>
<pre><code class="language-bash">PUT localhost:9200/administrators/administrator/1 -d &lt;some fields&gt;
PUT localhost:9200/moderators/moderator/1/ -d &lt;some fields&gt;
PUT localhost:9200/customers/customer/1 -d &lt;some fields&gt;

# Search for customers:
GET localhost:9200/customers/_search
</code></pre>
<p>In this configuration, I have three indices, one for each type of user. Elasticsearch therefore requires a minimum of 3 shards, one for each index. This is kind of a waste, because each index only contains documents of a single type, and Elasticsearch offers the ability have multiple types on an index.</p>
<p>If I want my index to use fewer shards, I could instead create a single index and use types to distinguish between different users. Maybe my index could be called <code>accounts</code>, and my types would be the same as my user types:</p>
<pre><code class="language-bash">PUT localhost:9200/accounts/administrator/1  -d &lt;some fields&gt;
PUT localhost:9200/accounts/moderator/1  -d &lt;some fields&gt;
PUT localhost:9200/accounts/customer/1  -d &lt;some fields&gt;

# Search for customers:
GET localhost:9200/accounts/customer/_search
</code></pre>
<p>In the second configuration, I have one index containing all three types of users. Elasticsearch therefore only requires a minimum of one shard, and uses said types to scope search results to a particular class of user. My goal of reducing the shard count has been completed.</p>
<h2>In closing</h2>
<p>In this post, we focused more on the basic problem and solution to a particular concern without really addressing the underlying question of <em>why</em> you can&#8217;t have multiple indices on a shard. The short answer is that Elasticsearch is magic, and you should just accept this as a reality.</p>
<p>A slightly longer, more technical answer is that it spins up fresh Lucene instances (shards) when an index is created, and handles the coordination of requests between those shards. Elasticsearch takes this approach to make sure that data is spread out for both performance and integrity reasons. That&#8217;s why it doesn&#8217;t make sense to have multiple indices per shard, because that is essentially asking why you can&#8217;t have multiple Lucene instances per Lucene instance, when the Lucene instance is the lowest possible worker unit. That&#8217;s also why types exist. Types are used to achieve the functionality of multiple indices per shard, but the terminology can confuse the issue by masking what&#8217;s going on internally.</p>
<p>Hopefully this has all helped to clarify the issue. If you have questions, concerns, hate mail or songs of praise, feel free to <a href="mailto:support@bonsai.io">gives us a shout</a>!</p></div>

Can I have multiple indices on a shard?

<div><p>We often get asked about security at Bonsai. <em>How do you ensure only I can see my data?</em> <em>What kind of encryption do you offer?</em> <em>What if my credentials are compromised?</em></p>
<h2>1. Unique, Randomized URL</h2>
<p>We offer several levels of security to all customers by default, and it starts
with your cluster&#8217;s hostname. Each cluster is assigned a unique, randomized
subdomain known only to the customer. This is a fairly standard practice for
cloud hosting providers (consider, for example, AWS Elastic Load Balancer or
CloudFront URLs).</p>
<p>The randomized hostname is security by obscurity: a good start, but not the full
story. A useful global identifier against which we can attach more tools.</p>
<p>Clusters are also provided with access credentials, which are required for all
requests. The extra layer of credentials makes it impossible for an attacker to
randomly guess the right combination to your cluster, and also provide a means
of periodically replacing access credentials.</p>
<p>Altogether, this cluster URL with credentials should be treated as sensitively
as any other database password, and only accessed from an application you
control and trust.</p>
<h2>2. TLS/SSL</h2>
<p>We support HTTPS on all of our clusters by default, providing encryption of all
traffic in transit via recent TLS encryption algorithms.</p>
<p>Encryption in transit works alongside your randomized URL and credentials,
helping to keep both private from any prying eyes. Not to mention the contents
of your data and search queries as your application interacts with the cluster.</p>
<h2>3. Firewall</h2>
<p>Like any good network administrators, we operate under the principle of least
privilege. Our servers are all tightly firewalled away from communication with
the wider internet, for obvious security reasons.</p>
<p>Each cluster&#8217;s private URL — with proper authorization over an encrypted
protocol — makes for a tightly controlled access path through our networks
and over to your cluster&#8217;s resources.</p>
<p>This access to Elasticsearch is controlled and mediated by our custom-built,
high-performance layer 7 routing proxy. This proxy serves as a central choke
point where we can authorize and audit all of the traffic incoming to your
traffic.</p>
<p>In addition to logging and access control, this proxy layer applies intelligent
connection management policies to help ensure best access patterns and
interacting with Elasticsearch itself. This lets us protect your cluster against
unexpected traffic surges, but that&#8217;s a whole different kind of security that
deserves its own article!</p>
<h2>4. IP and Network CIDR Whitelisting</h2>
<p>An additional benefit of our proxy layer is that it gives us tools to support
more advanced networking functionality that are not present in a stock
Elasticsearch deployment.</p>
<p>Currently in beta with select customers, we have the ability to also enforce IP
and network CIDR whitelisting. This provides an extra level of comfort (or
security compliance!) for customers whose deployments all route their outgoing
traffic through a predictable network gateway.</p>
<h2>5. Encrypted, Offsite Backups</h2>
<p>Bonsai also ensures regular backups are encrypted and stored in a completely
separate AWS account than the one that runs our servers. We use the bare minimum
in credentials for access to backups to ensure that your data is safe from
scrutiny, and always available for disaster recovery should the worst occur.</p>
<h2>6. Encryption at Rest</h2>
<p>Finally, we offer encryption at rest as a feature of single tenant clusters.</p>
<p>Generally speaking, we also have experience in designing and deploying clusters
built to customer specifications in order to support sensitive data and comply
with a range of regulatory policies, like HIPAA and Safe Harbor. <a href="mailto:support@bonsai.io">Contact
us</a> with any specific questions you might have on
security and compliance.</p></div>

Six Important Facts about Bonsai Security

Founder, CEO

<div><blockquote>
<p>Announcing a major change in the industry: Elastic is ending the open source licensing for future versions and AWS announces its forks of Elasticsearch and Kibana.</p>
</blockquote>
<p>Last week, <a href="https://www.elastic.co/blog/licensing-change" target="_blank" rel="nofollow">Elastic announced</a> that future releases of Elasticsearch and Kibana will be changing their licensing. The current 7.10 release will be the last under the open source Apache 2.0 license. Future versions of Elasticsearch and Kibana will be dual-licensed under the $1&#8217;source-available&#8217; Elastic license and <a href="https://www.mongodb.com/licensing/server-side-public-license" target="_blank" rel="nofollow">Server Side Public License</a> (SSPL).</p>
<p>On Thursday, <a href="https://aws.amazon.com/blogs/opensource/stepping-up-for-a-truly-open-source-elasticsearch/" target="_blank" rel="nofollow">AWS announced it will maintain its own fork of Elasticsearch</a>.</p>
<p>Our customers naturally have questions about what comes next. After a week of study, and consulting with industry experts, we’re now sharing our understanding of what possibilities exist to ensure continuity of high quality service for our customers.</p>
<p><strong>It’s worth emphasizing: nothing changes immediately, nothing changes retroactively.</strong> All Bonsai clusters are running on Apache 2.0 licensed releases, and will be supported indefinitely, according to our normal support schedule.</p>
<h2>What are our available options?</h2>
<p>While there are plenty of subtle combinations, we have identified four paths moving forward.</p>
<ol>
<li>Comply with the terms of the SSPL</li>
<li>Negotiate distribution terms with Elastic</li>
<li>Participate in a community fork</li>
<li>Endorse and support the AWS fork</li>
</ol>
<h3>Comply with the SSPL</h3>
<p>Insomuch as a market of providers exists around Elasticsearch, and insomuch as that market can credit its existence to Elasticsearch, Elastic has taken the position that providers selling Elasticsearch as a service should exist in a more ‘collaborative’ relationship with them.</p>
<p>The SSPL identifies such commercial providers. It doesn’t exclude them from offering a paid service, but it does provide a unique copyleft-style stipulation. Providers of a paid service are required under the SSPL to release their entire management stack under the terms of the SSPL.</p>
<p>This is a bold requirement, and I am not aware of any company who has decided to go this route under the SSPL. It’s plausible that many companies would find this kind of stipulation to be completely infeasible.</p>
<p>For a small team like ours, who places much more value on the humans within the service than the code that supports them, it’s not entirely impossible.</p>
<p>It’s true that we have good and innovative technology. Our customers certainly would not enjoy our service if our technology was bad—or we would find it unprofitable to support. But our customers don’t choose us because of our technology. They choose us because of our people, and our values. Because we are committed to their availability and performance, and because we make ourselves available to give incredible advice and guidance.</p>
<p>If we went the SSPL route, we may well be the first company to do so. It’s a novel situation and a novel license whose language is not without problems. We’re engaged with experts in the relevant IP law as we consider this path.</p>
<p>Complying with the SSPL would be a gesture to take Elastic at face value, and continue participating in an ecosystem that they’ve helped create, within terms they find agreeable. However, given that the license&#8217;s language is problematic, we think this is an unlikely direction for us.</p>
<h3>Negotiate distribution terms with Elastic</h3>
<p>As a prominant alternative in their FAQ, Elastic invites providers to reach out to negotiate custom license terms.</p>
<p>This seems like a completely sound and viable business decision, especially when customers strongly care both about future access to the latest releases of the mainline Elasticsearch product, and also having access to more diverse and competitive service providers.</p>
<p>While researching the eventual decisions of MongoDB providers in 2018 and 2019, I could only find examples of companies coming to an agreement with MongoDB.</p>
<p>We’ve long been open to the idea of working more closely with Elastic. In fact, Bonsai was a member of Elastic’s own nascent partner program back in 2014 and 2015. With circumstances changing in the market, we’re open to the possibility of new opportunities.</p>
<p>Toward that end, we have been in touch with Elastic to discuss and understand their goals, and explore possible licensing terms.</p>
<h3>Participate in a community fork</h3>
<p>One of the core premises of open source is that, once opened, a project cannot be closed again. Any company or entity who decides to take their own proprietary path leaves the community with a choice: to follow that path, or to go their own.</p>
<p>It seems eminently likely — and eminently reasonable — that certain businesses within the larger Elasticsearch ecosystem will remain committed to the ideal of preserving and maintaining an open source alternative.</p>
<p>It is certainly true that Elastic employs a large number of core Elasticsearch engineers who can continue contributing bug fixes and new features. However, in our experience, current versions of Elasticsearch are quite mature. Even a relatively feature-frozen fork, focused on bug-fixes and security patches, can still be incredibly useful and competitive into the future. It would take a helluva lot of work, or a revolutionary idea, or both, for a brand new project to overtake what is available in Elasticsearch 7.10.</p>
<p>In this section we&#8217;re intentionally focusing on a <em>community-led</em> effort. That&#8217;s because sophisticated open-source consumers have long paid attention to governance. This is an important aspect of open source that is not reflected in the license itself.</p>
<p>Elastic and AWS have provided a fresh and poignant case study for the community on governance. When a project is controlled exclusively by a single commercial interest, then the market challenges and struggles and forces that entity faces become consequences that its community of users and contributers have to reconcile with.</p>
<p>A community-led effort to rename, maintain, and enhance the Apache 2.0 branch of Elasticsearch would be in the interests of companies&#8217; who value a more diverse selection of Elasticsearch service providers. It would also go a long way to reassuring a wider industry about the continued viability of open source.</p>
<h3>Endorse and support the AWS fork</h3>
<p>AWS has since announced their intent to <a href="https://aws.amazon.com/blogs/opensource/stepping-up-for-a-truly-open-source-elasticsearch/" target="_blank" rel="nofollow">maintain an Apache 2.0 licensed fork</a> of Elasticsearch. This is good news for customers of AWS&#8217;s managed Elasticsearch service who would like an assurance of continued future viability of that service. In those terms, it seemed clear from the beginning that AWS would likely pursue their own open source fork.</p>
<p>With this project officially now available, we will be giving it serious consideration. Particularly if there is some opportunity created here for meaningful partnership with a broader community, alleviating some of the concerns of using a project that&#8217;s controlled by a single entity.</p>
<h3>Again, these changes are not urgent</h3>
<p>New licenses will apply to new releases, and it is a rare deployment that absolutely <em>needs</em> to be on the bleeding edge. Licensing changes here only have urgency insomuch as your app needs new features or specific bugs fixed. In practice, urgent bug-fix releases are, thankfully, increasingly rare in a mature open source technology.</p>
<p>Indeed, rather than rushing to newer versions, many of our customers find value from the stability of long-term support on older versions, which is a feature that our <a href="https://bonsai.io/pricing" target="_blank" rel="nofollow">Business and Enterprise</a> clusters enjoy. While a majority of Bonsai clusters are deployed on Elasticsearch 7.x, we have a healthy mix of clusters deployed onto versions as far back as 1.7.5.</p>
<p>The Elasticsearch codebase has been around a long time now, and after nearly a decade supporting it, we are quite comfortable with the level of maturity and stability offered in current versions. We feel comfortable that this affords us and our customers plenty of time to thoughtfully consider and navigate the best path for supporting future releases.</p>
<h2>A thriving ecosystem of providers</h2>
<p>We don’t really need to make the case here that Elasticsearch is an incredibly valuable and innovative piece of software.</p>
<p>However, that value doesn’t exist in a vacuum. There’s a lot of work that happens between the search engine at the bottom of the stack and the end-user perusing the results of their query at the top. Within an ecosystem, the diversity of business and end-user needs are what create the opportunities for creative solutions and a diverse marketplace to arise and flourish.</p>
<p>At Bonsai we’re clear that we didn’t invent the search engine — indeed, we have deliberately resisted building our own! Rather, we focus on providing high quality management and scaling, encompassed within our “Evergreen” promise: that our customers’ clusters are never anything but healthy and fast; if something is wrong, it&#8217;s our responsibility to fix.</p>
<p>We started offering our managed Elasticsearch services in early 2012, before Elastic, Inc. even existed. Our best and only understanding of our responsibilities at the time was the project’s Apache 2.0 license, and the emergence of a thriving ecosystem around it. We dove into the deep-end of scaling and supporting Elasticsearch through its at-times troubled adolescence. And as Elastic the company grew, acquiring another startup to form their Elastic Cloud platform, and as AWS made their entry, we didn&#8217;t worry too much. We remained committed to supporting our customers as our first-to-market position evolved into a niche of quiet specialists.</p>
<h2>We’re here to serve our customers</h2>
<p>At Bonsai our stakeholders are our customers and our team. We remain focused on our customers and providing them the most productive tools and software available so they can stay focused on the work of delivering great search experiences. Tools and licenses and communities shift and evolve, but our commitments will not change.</p>
<p>Whether you’re a customer, or a member of the community, I would love to hear from you! Drop me a line any time at <a href="mailto:nick@bonsai.io">nick@bonsai.io</a>.</p></div>

Open letter to Bonsai customers

<div><p>Does your application have a search bar? Are you recording what your users are searching for? If not, you are missing out on your customers literally telling you what they want from you. Paying attention to these asks can help guide you to customer delight.</p>
<p>If not, you are missing out on one of the strongest signals that your customer can send. By paying attention to what they are asking you for you can give them exactly what they want.</p>
<p>For the rest of the blog post I&#8217;ll be referencing this code snippet from a Ruby on Rails project. You could just as easily write this same snippet of code in your favorite language and framework, it&#8217;s not all that technical or interesting. Its the data that we end up capturing that is the star of this story.</p>
<pre><code class="language-ruby">class ProductSearchController &lt; ApplicationController
  # other actions elided...
  def post
    query = params['q']
    offset = params[:page] * params[:per_page]

    # Perform the search
    @products = Product.search(
      query,
      limit: params[:per_page],
      offset: offset
    )

    # Record the search
    SearchLog.create(
      user: @current_user,
      session: session.id,
      query: query,
      doc_ids: @products.map(&amp;:id),
    )
  end
end
</code></pre>
<p>In the code snippet above, I&#8217;m demonstrating a chunk of code that is executing a search on behalf of the user. After the search is executed, we record a log of that search including the current session, the user executing the query, the raw query from the user, the docs that were returned, and what page was the user on.</p>
<h2>Help Customers Win</h2>
<p>This simple set of data is enough for us to start hearing a story that our users are trying to tell us. Let&#8217;s look at an example user&#8217;s logs from a recent visit.</p>
<pre><code class="language-md">user   | session_id | query      | doc_ids
-------+------------+------------+---------------
user:1 | 20265a...  | windshield | []
user:1 | 20265a...  | fairing    | [34, 55, 768]
</code></pre>
<p>Even with this small contrived example, we can see something that is interesting. <code>user:1</code>$1&#8217;s first search for <code>windshield</code> returned zero hits. Frustrated, the user went back to the search bar and tried the correct term <code>fairing</code> (the motorcycle term for a windshield). Now that they are using the correct term, they are able to start shopping which hopefully leads to them checking out.</p>
<p>Using this information, we could go back to our search index and add an alias from <code>windshield</code> to <code>fairing</code>, which would let users succeed with one less step. Once deployed, if a user searches for <code>windshield</code>, they will get the same results as a customer who typed in <code>fairing</code>.</p>
<pre><code class="language-md">user   | session_id | query      | doc_ids
-------+------------+------------+---------------
user:5 | 4b3ad5...  | windshield | [34, 55, 768]
</code></pre>
<blockquote>
<p>Bonsai recently released a plugin called <code>elasticsearch-stored-synonyms</code> that will let you store your synonyms in the Elasticsearch cluster but importantly allows hot reloading of synonyms without having to recreate the index.</p>
<p>For more details check out: <a href="https://github.com/omc/elasticsearch-stored-synonyms" target="_blank" rel="nofollow">omc/elasticsearch-stored-synonyms</a></p>
</blockquote>
<h2>Giving your customers what they are asking for</h2>
<p>Using this same set of data points let&#8217;s imagine a different scenario.</p>
<pre><code class="language-md">user    | session_id | query                 | doc_ids
--------+------------+-----------------------+--------
user:29 | 67f4ab...  | thing you don't carry | []
user:43 | 0b959c...  | thing you don't carry | []
user:54 | b96969...  | thing you don't carry | []
user:76 | 54e233...  | thing you don't carry | []
user:98 | 3d3483...  | thing you don't carry | []
user:34 | bfc08f...  | thing you don't carry | []
</code></pre>
<p>In this dataset we can see six different users that are all looking for the same item, but it&#8217;s not one that you currently carry. If this started to show up &#8220;out of the blue,&#8221; then that might give us pause to reflect. Is this something that we want or should carry? If it meets the right criteria, you have just found another product that you can offer to your users. Pretty soon, that same dataset might start looking like:</p>
<pre><code class="language-md">user    | session_id | query               | doc_ids
--------+------------+---------------------+----------
user:29 | 67f4ab...  | thing you now carry | [98, 102]
user:43 | 0b959c...  | thing you now carry | [98, 102]
user:54 | b96969...  | thing you now carry | [98, 102]
user:76 | 54e233...  | thing you now carry | [98, 102]
user:98 | 3d3483...  | thing you now carry | [98, 102]
user:34 | bfc08f...  | thing you now carry | [98, 102]
</code></pre>
<p>If that then tracks to conversions, then we&#8217;ve identified a new stream of revenue simply by listening to what our customers are telling us.</p>
<h2>Closing</h2>
<p>We&#8217;ve only scratched the surface of what we can learn from our customers by listening to what they are telling us in the search bar. Hopefully, you can appreciate the insights that can be gathered from your search infrastructure.</p>
<blockquote>
<p>Want to improve your search analytics? Schedule a call now with Bonsai.</p>
</blockquote></div>

Listening to your users

<div><p>One More Cloud Inc. is a unique company that loves learning, collaboration, and curiosity. In our efforts to grow as a company, we want to learn from our team and their experiences. We spoke with Designer and Front-End Engineer Allison Zadrozny about OMC’s company culture and hiring process.</p>
<h2>Can you tell me a little bit about OMC’s culture?</h2>
<p>At OMC, we’re highly collaborative and interdependent with one another. We’re a small team so everyone works in cross-functional ways; we can’t afford hyper-specialization. When things are going great, it’s because of the team working well together. We’re not looking for heroes nor martyrs because neither of those work well with our company mindset.</p>
<h2>How does OMC operate differently from other companies?</h2>
<p>OMC has two different modes of how the company runs. The first is operational, which is hierarchical and must lean on process, formality, and seniority. The second is collaborative: a flat, informal, and creative style where we <a href="https://www.amazon.com/Yes-Improvisation-Reverses-Creativity-Collaboration-Lessons/dp/0062248545" target="_blank" rel="nofollow">&#8220;say yes&#8221;</a> to each other and play with ideas to find the best outcomes.</p>
<p>Operational modes of working with strict hierarchy is a pretty familiar culture in American business. For us, this shows up in higher-stakes situations involving things like server operations and legal admin work. If we’re running an incident response or wrangling specifics of a legal requirement, we don’t have time to be super collaborative. Instead we are in operational mode, relying on specific skillsets and decision-making responsibility.</p>
<p>Then there’s the creative and collaborative side. In this mode, everyone’s voice is allowed to contribute. The caveat here is that in order for others to take my participation seriously, I must put my skin in the game by adding value to the process with curiosity, willingness to learn, personal research, and generating some sort of output - like writing, PR’s, or visual prototyping work. At a lot of companies, you get people into a room for creative work, and literally whoever is the loudest is probably going to win that argument or “brainstorming session.” It’s difficult to protect against these sort of pluralistic and social biases, but we try. Working remotely and generating work commentary asynchronously helps, as well as having a company posture of openness and curiosity.</p>
<p>These modes have to be employed in a balance. It’s irresponsible to believe one can run all aspects of a business in a flat hierarchy. Likewise, strict top-down management squashes creativity. There is space for both. It isn’t easy, but we strive to have discernment about what mode to use depending on the task at hand.</p>
<h2>What is OMC’s hiring process?</h2>
<p>Historically we’ve used our network to fill positions. But that’s changing. For this last round of hiring, we posted on tons of job boards — some specifically for BIPOC and underrepresented individuals in tech (like <a href="https://www.diversifytech.co/job-hunt" target="_blank" rel="nofollow">diversifytech.org</a>, <a href="https://www.womenwhocode.com/" target="_blank" rel="nofollow">womenwhocode.com</a>, <a href="https://www.2050.work/" target="_blank" rel="nofollow">2050.work</a>, <a href="https://www.include.io/" target="_blank" rel="nofollow">include.io</a>, and others), and within the Austin tech meetup space. There’s a lot of decision-making that happens before we post. We’re very clear about what our “why” is behind hiring.</p>
<p>The first step for a candidate is a quick screening call. After that, we conduct a longer interview to assess skills on a video call. One of the final steps is a panel interview. Participants in this interview are the interviewee, the hiring manager, and a couple of people that the candidate would work closely with. We try to have the panel made up so that different perspectives are provided. One important recent change we’ve employed in the last couple of years is using interviewing scorecards that each panelist must complete on their own after each interview. The topics on the card include things from general professionalism to specific role-based questions we ask during the interview, like “Explain your approach to concurrency for multi-core and multi-node applications.” Scorecards take a lot of effort, but they help mitigate bias and have offered good jumping off points for some tough conversations internally.</p>
<p>Our process here is constantly improving, but one thing we try to be amazing at is good and timely communication. We all know how stressful it feels to be on the other side of the table waiting to hear back from a company or getting vague information. We don’t want to ever make other people go through that, so we try to make our hiring as humane and respectful as possible.</p>
<h1>How are you trying to make sure you get the most diverse candidates?</h1>
<p>The most important question for us: how do we include a greater diversity, equity, and inclusion within our hiring process? Our scorecards diminish <a href="https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555" target="_blank" rel="nofollow">system 1 thinking</a> about candidates, but that doesn’t matter if we only have the same kind of people in our applicant pool. This is all about reaching out and getting a broader pool of applicants. I’m not going to say we’re great at this - <a href="https://omc.io/" target="_blank" rel="nofollow">just look at the makeup of our current team</a>. We’ve used a lot of job boards for BIPOC and underrepresented groups in tech, but I’d like to see us get a larger volume of applicants from these places and work internally to make sure our culture is a place where anyone, regardless of their identities, class background, and race, can feel like they are valued and fairly compensated for their skills and work.</p>
<p>It’s important to me to be real about equity and representation at our company — why wait for someone else to call us on our BS when we can do our own honest self-reflection and be open about it? I think one thing that we’ve thought about a lot is that we don’t want anyone that’s coming on to feel like they are “just the diversity hire,” without a true say in the team and the projects they are working on, or without hope for promotion. We want to make sure that those we’re hiring feel confident as a teammate: “I have badass skills, I’m here for a job, and my colleagues respect me.”</p>
<p>That’s just more of the philosophy part, which is pretty big. We’ve got a lot of work to do as a small company, but I’m confident that we are making ourselves an excellent team to work with no matter who you are.</p></div>

How OMC thinks about hiring

<div><h2>How To Test Elasticsearch</h2>
<p>We’ve recently kicked off a push to overhaul <a href="https://docs.bonsai.io/" target="_blank" rel="nofollow">our documentation</a>, and make our <a href="https://docs.bonsai.io/collection/85-quickstart-guides" target="_blank" rel="nofollow">Quickstart Guides</a> more useful to users. The goal has been to make them easier to use, add more real-world examples, and take the reader through the process from start to finish.</p>
<p>We started with <a href="https://docs.bonsai.io/article/97-ruby-on-rails" target="_blank" rel="nofollow">Ruby on Rails</a>, using the official Elasticsearch gems. As someone with Rails experience, but not with the <a href="https://github.com/elastic/elasticsearch-rails" target="_blank" rel="nofollow">elasticsearch-rails gem</a> specifically, it was a great opportunity for me to get some hands-on experience with that code. To give readers of our documentation some concrete examples, we made a <a href="https://github.com/omc/bonsai-examples/tree/master/rails" target="_blank" rel="nofollow">basic Rails app</a> and went through the process of making one of the models searchable with Elasticsearch.</p>
<p>Tested code is better code. You can execute several helpful tests to ensure your Elasticsearch code is cleaner, more reliable, and easier to maintain. Like many concepts in Elasticsearch, this process may seem conceptually easy, but there are a lot of hidden pitfalls. I’ll call them out as we go.</p>
<h3>Pitfall 1: Version Management</h3>
<p>Wow, a pitfall before we even start! In Elasticsearch, there are often breaking changes between major versions, and sometimes even between minor versions. The official gems are targeted to specific versions of Elasticsearch, and it’s important to specify the version of the gem(s) that is compatible with the version of Elasticsearch you’re running in production.</p>
<p>A similar problem exists for Rails developers, in that different projects are built using different versions of Ruby. When switching between projects, you need to ensure that the version of Ruby in your console is exactly what the current project requires. There are some tools for this, namely <code>rbenv</code> and <code>rvm</code>. With this in mind, I looked around to see if something similar exists for Elasticsearch.</p>
<p>There are a couple solutions out there. I think <a href="https://github.com/duydo/evm" target="_blank" rel="nofollow">evm</a> is one of the better ones, although it’s not quite as flexible as rvm or rbenv. evm has a familiar set of command line options, and it downloads Elasticsearch binaries to <code>~/.evm</code>. The one thing I’m not a fan of is that it doesn’t set or override a system-wide Elasticsearch binary. Instead, it uses a link: <code>~/.evm/elasticsearch</code>. So you have to somehow alias <code>elasticsearch</code> to <code>~/.evm/elasticsearch/bin/elasticsearch</code>. And evm doesn’t support <a href="https://en.wikipedia.org/wiki/Dot_file" target="_blank" rel="nofollow">dotfiles</a>.</p>
<p>We’re heavy users of rbenv at OMC, and I love how it uses dotfiles to ensure the correct version of Ruby is automatically loaded for any given project. I decided to follow this pattern and created a dotfile for the Elasticsearch version we need for the app. This way we can write tests to ensure our other tests are running against the correct version of Elasticsearch.</p>
<h3>Pitfall 2: System Configuration</h3>
<p>Another pitfall involves where Elasticsearch is installed in your system. I wanted to test against an actual running instance of Elasticsearch, but wanted some control over how the cluster is started, stopped and configured. That’s a pretty complicated task to manage by hand.</p>
<p>This is made much easier with the excellent <a href="https://github.com/elastic/elasticsearch-ruby/tree/master/elasticsearch-extensions" target="_blank" rel="nofollow">elasticsearch-extensions gem</a>. This gem is an extension of the elasticsearch-ruby gem, and it offers some really neat features. The one we’re going to make use of here is the ability to automatically spin up and tear down local test clusters.</p>
<p>Something to keep in mind about the elasticsearch-extensions gem: it uses your system version of Elasticsearch by default. In other words, it will try to run the first <code>elasticsearch</code> program it finds in your <code>$PATH</code>. This is a problem if: A) you don’t have Elasticsearch installed on your system, or B) your system-wide Elasticsearch is not in the <code>$PATH</code> of your test suite, or C) you want to test a version of Elasticsearch that is different from the one installed to your system. Or any other number of configuration eccentricities you might have.</p>
<p>Fortunately, you can override this setting with a <code>command</code> parameter. We’ll cover that in the examples below.</p>
<h2>Getting Started</h2>
<p>For this project, we’re targeting Elasticsearch 7.4.0. In our Gemfile, we already have this:</p>
<pre><code class="language-ruby"># As of this writing, the master branch is for Elasticsearch 7.x.
# Check the README.md for current compatibility:
# https://github.com/elastic/elasticsearch-rails/blob/master/README.md#compatibility

gem 'elasticsearch-model', github: 'elastic/elasticsearch-rails', branch: 'master'
gem 'elasticsearch-rails', github: 'elastic/elasticsearch-rails', branch: 'master'
</code></pre>
<p>Those gems will ensure the app is able to instantiate a client that is compatible with Elasticsearch 7.x. I then created a dotfile called <code>.elasticsearch-version</code>:</p>
<pre><code class="language-shell">echo '7.4.0' &gt; .elasticsearch-version
</code></pre>
<p>This means that my app should expect to work with Elasticsearch 7.x, and we have explicitly defined a version we’re targeting. Next, you need to add <code>gem 'elasticsearch-extensions'</code> to your <code>Gemfile</code> inside the <code>:test</code> block, and run <code>bundle install</code> in your command line. That should ensure we have all the gems that we need.</p>
<p>Next, we need to configure <code>spec_helper.rb</code> to make the elasticsearch-extensions’ test features available to RSpec. Add <code>require 'elasticsearch/extensions/test/cluster'</code> at the top of your <code>spec/spec_helper.rb</code> . Then inside the <code>RSpec.configure</code> block, add:</p>
<pre><code class="language-ruby">cluster = Elasticsearch::Extensions::Test::Cluster::Cluster.new(port: 9250, number_of_nodes: 1, timeout: 120)
</code></pre>
<p>If you are using evm, or have a specific binary you want to test with, then you’ll want something like this:</p>
<pre><code class="language-ruby">cluster = Elasticsearch::Extensions::Test::Cluster::Cluster.new(port: 9250, number_of_nodes: 1, timeout: 120, command: '~/.evm/elasticsearch/bin/elasticsearch')
</code></pre>
<p>This line creates a data structure representing an Elasticsearch cluster with one node, running on port 9250 (if you have more nodes, you will get multiple instances of Elasticsearch running on ports incrementing from 9250). It assigns this to a variable, which we’ll use to start and stop the cluster.</p>
<h3>Pitfall #3: Client Configuration</h3>
<p>Rails apps implementing Elasticsearch are probably using either the elasticsearch-rails gem, or Searchkick. Each of these gems will default to looking for Elasticsearch on <code>localhost:9200</code>, unless specifically instructed not to. If your production app is not running on the same machine(s) as your Elasticsearch cluster, then you probably want to override this default. Otherwise your tests will fail (or they will be run against a different cluster from the one we are creating explicitly for testing).</p>
<p>If you’re a Bonsai customer, and especially if you are using the <a href="https://github.com/ankane/searchkick" target="_blank" rel="nofollow">Searchkick</a> OR <a href="https://github.com/omc/bonsai-elasticsearch-rails" target="_blank" rel="nofollow">bonsai-elasticsearch-rails</a> gems, then add this line:</p>
<pre><code class="language-ruby">ENV['BONSAI_URL'] = "localhost:#{cluster.arguments[:port]}"
</code></pre>
<p>This sets an environment variable that is used to correctly set up the Elasticsearch client to point to the test cluster.</p>
<p>Otherwise, you might want an initializer or setting in <code>config/environments/test.rb</code> to specify that the Elasticsearch cluster is located at <code>localhost:9250</code> in a test context. Do the thing that makes the most sense in your code base.</p>
<p>The last step here is to configure RSpec to stand up and tear down Elasticsearch automatically.
While researching for this process, I came across a <a href="https://medium.com/@rowanoulton/testing-elasticsearch-in-rails-22a3296d989" target="_blank" rel="nofollow">great post by Rowan Oulton</a> that takes a neat approach. It injects <code>before</code> and <code>after</code> blocks into any RSpec block that is tagged with <code>elasticsearch: true</code>.</p>
<p>So now, in <code>spec_helper.rb</code>, inside the <code>RSpec.configure</code> block, I have this code:</p>
<pre><code class="language-ruby"># Create a data structure representing the cluster to build up and tear down:
cluster = Elasticsearch::Extensions::Test::Cluster::Cluster.new(port: 9250, number_of_nodes: 1, timeout: 120)

# Make sure the Elasticsearch client can find the cluster we want to use. The
# BONSAI_URL variable is used by the bonsai-elasticsearch-rails gem to automatically
# configure the Elasticsearch client:
ENV['BONSAI_URL'] = "localhost:#{cluster.arguments[:port]}"

# Create a local cluster for all tests within an RSpec block tagged with
# `elasticsearch: true`. Credit to Rowan Oulton:
# https://medium.com/@rowanoulton/testing-elasticsearch-in-rails-22a3296d989
config.before :all, elasticsearch: true do
  cluster.start unless cluster.running?
  ActiveRecord::Base.descendants.each do |model|
    if model.respond_to?(:__elasticsearch__)
      begin
        model.__elasticsearch__.create_index!
        model.__elasticsearch__.refresh_index!
      rescue Elasticsearch::Transport::Transport::Errors::NotFound =&gt; e
        # This kills "Index does not exist" errors being written to console
        # by this: https://github.com/elastic/elasticsearch-rails/blob/738c63efacc167b6e8faae3b01a1a0135cfc8bbb/elasticsearch-model/lib/elasticsearch/model/indexing.rb#L268
      rescue =&gt; e
        STDERR.puts "There was an error creating the elasticsearch index for #{model.name}: #{e.inspect}"
      end
    end
  end
end

# Stop elasticsearch cluster after test run
config.after :suite do
  ActiveRecord::Base.descendants.each do |model|
    if model.respond_to?(:__elasticsearch__)
      begin
        model.__elasticsearch__.delete_index!
      rescue Elasticsearch::Transport::Transport::Errors::NotFound =&gt; e
        # This kills "Index does not exist" errors being written to console
        # by this: https://github.com/elastic/elasticsearch-rails/blob/738c63efacc167b6e8faae3b01a1a0135cfc8bbb/elasticsearch-model/lib/elasticsearch/model/indexing.rb#L268
      rescue =&gt; e
        STDERR.puts "There was an error removing the elasticsearch index for #{model.name}: #{e.inspect}"
      end
    end
  end
  cluster.stop if cluster.running?
end
</code></pre>
<p>Finally, I added <code>elasticsearch: true</code> to the RSpec block in the file <code>spec/models/user_spec.rb</code>. This is needed so that the code above can be invoked:</p>
<pre><code class="language-ruby"># spec/models/user_spec.rb

require 'rails_helper'

# Note the `elasticsearch: true` tag in this block:
RSpec.describe User, elasticsearch: true, type: :model do
  ... tests go here
end
</code></pre>
<p>Now let’s get to testing!</p>
<h2>Writing the Tests</h2>
<p>I used <code>rspec</code> for my tests. I also integrated the <a href="https://github.com/thoughtbot/factory_bot_rails" target="_blank" rel="nofollow">FactoryBot gem</a> for rails which makes creating test data much less time consuming in a few easy steps:</p>
<ul>
<li>Add <code>gem 'factory_bot_rails``'</code> to your test and development blocks in you <code>Gemfile</code>.</li>
<li>Run <code>bundle install</code> from the console.</li>
<li>Add this line of code to your <code>rails_helper</code> file:</li>
</ul>
<pre><code class="language-ruby">RSpec.configure do |config|
  config.include FactoryBot::Syntax::Methods
end
</code></pre>
<ul>
<li>Create <code>spec/factories/user.rb</code> in your file directory and add this code to the file:</li>
</ul>
<pre><code class="language-ruby">FactoryBot.define do
  factory :user do
    sequence(:first_name) { |n| "Sam#{n}" }
    sequence(:last_name) { |n| "Smith#{n}" }
    sequence(:email) { |n| "samsmith#{n}@gmail.com" }
    address { '123 Happy St' }
    city { 'Denver' }
    zip_code { 80_015 }
    company { |n| "Smith&amp;Co#{n}" }
    company_description { |n| "Smith&amp;Co#{n} dry goods" }
  end
end
</code></pre>
<p>Now that your factories are set up, let’s write the first test!</p>
<p>I decided to start with a test that would help me make sure that the version of the test cluster and the version in the dotfile aligned, so that we know we are testing the same version in test and in production.</p>
<p>This is what I added in the <code>user_spec</code> file inside a describe block:</p>
<pre><code class="language-ruby">it 'should have the right version' do
  es_version = User.__elasticsearch__.client.info['version']['number']
  dotfile = "#{File.open(".elasticsearch-version", &amp;:readline).chomp}"
  expect(dotfile).to eql(es_version)
end
</code></pre>
<p>This is a great sanity test for us. If we start getting errors, we’ll be able to see very plainly that it’s due to a version mismatch between the cluster we’re testing on, and the version of Elasticsearch we’re targeting.</p>
<p>After I got this out of the way, I wanted to be able to know that the <code>Elasticsearch::Model::Callbacks</code> library is working as expected. This library injects Elasticsearch calls into the ActiveRecord lifecycle. The <code>User</code> model in our demo app uses it so that when a User record is created/updated/destroyed, the corresponding Elasticsearch document is automatically changed too.</p>
<p>Initially, the cluster should have no User records in it. In the <code>user_spec</code> file, inside a describe block, I tested this by expecting that a search would return 0 results:</p>
<pre><code class="language-ruby">it 'should initially have no User records' do
  expect(User.search('*:*').records.length).to eq(0)
end
</code></pre>
<p>To test what happens when we try and add a record, I created a different test that creates a user, refreshes the index, and then searches for the user that was just created:</p>
<pre><code class="language-ruby">it 'should update ES when the object is created' do
  user = create(:user)
  User.__elasticsearch__.refresh_index!
  expect(User.search("id:#{user.id}").records.length).to eq(1)
end
</code></pre>
<p>When this test passes, it means that we know that our indexing works. Yay for us!</p>
<p>We can also validate that deleting users in Rails will also delete the indexed record:</p>
<pre><code class="language-ruby">it 'should update ES when the object is destroyed' do
  user = create(:user)
  user.destroy!
  expect(User.search("id:#{user.id}").records.length).to eq(0)
end
</code></pre>
<p>That’s a fairly reasonable set of tests to ensure that the records are created and destroyed as expected. Let’s test searches by creating a few users and make sure that we can find them in Elasticsearch:</p>
<pre><code class="language-ruby">it 'should return correct results when queried' do
  user_1 = create(:user)
  user_2 = create(:user)
  user_3 = create(:user)

  User.__elasticsearch__.refresh_index!
  expect(User.search("first_name:#{user_1.first_name}").records.length).to eq(1)
  expect(User.search("first_name:#{user_1.first_name}").records.first.first_name).to eq(user_1.first_name)
end
</code></pre>
<p>By default, the elasticsearch-rails gem will create indices with 5 primary shards. This is significantly over-provisioned for most users, and we recommend overriding this to use a single primary shard. In our demo app, we configured the User model with:</p>
<pre><code class="language-ruby">settings index: { number_of_shards: 1 }
</code></pre>
<p>We can check this setting is being respected with a test like this:</p>
<pre><code class="language-ruby">it 'creates the right number of primary shards' do
  model_shards = User.settings.to_hash.dig(:index, :number_of_shards)
  es_shards = User.__elasticsearch__.client.perform_request(:get, 'users/_settings').body.to_hash["users"]["settings"]["index"]["number_of_shards"].to_i
  expect(model_shards).to eq(es_shards)
end
</code></pre>
<p>You can use a similar approach to test your mappings (or number of replicas).</p>
<p>This seems like a pretty reasonable time to run RSpec. If all tests are green, your Elasticsearch implementation is working. Go you!</p>
<h2>Wrapping up</h2>
<p>In closing, I found the process for testing fun but also challenging. I now understand why there aren’t a lot of resources on testing Elasticsearch since the process involves quite a few &#8220;gotchas&#8221;. I hope this tutorial can shed some light on those pitfalls and help people in their testing journey.</p>
<p>I can’t wait to write more tests that allow me to test my customization of Elasticsearch for my projects. If you have any settings you are excited to see tested, just let me know at <a href="mailto:support@onemorecloud.com">support@onemorecloud.com</a>. I love a challenge. If you’d like to see this in action, check out the <a href="https://github.com/omc/bonsai-examples/tree/master/rails" target="_blank" rel="nofollow">demo app</a> I used for writing these tests.</p></div>

How To Test Your Elasticsearch Integration with RSpec

<div><p>If you&#8217;ve ever tweaked Elasticsearch queries to improve search performance or relevance, you&#8217;ve likely noticed that Elasticsearch queries can be very long. They can take a considerable amount of time to create, debug, and optimize. There&#8217;s also no easy way to collaborate on search queries. That’s why we created our latest feature, Search Clips.</p>
<p>Search Clips allow you to query any of your clusters and view readable, real-time responses. Team members within your Bonsai account can view and edit your Search Clips which makes collaborating on search queries easy. Search Clips can also be exported to JSON or CSV to be shared with external parties or imported into third party systems such as Geckboard, Google Sheets, or custom tools.</p>
<h2>What’s now available in Search Clips</h2>
<p>Right now, you might use curl or the Kibana Console to test queries. Kibana doesn’t offer anything that actually <em>saves</em> your query, making iteration and collaboration pretty messy. Search Clips focus on a single query and its response, allowing you to iterate on a query until you see the expected results in the Response View.</p>
<h2>Helpful features of Search Clips include:</h2>
<ol>
<li>Structured Semantic Completion. Our knowledge of Elasticsearch query structure and the fields present in your data enable auto completion with that information, making query building easier.</li>
<li>Results will be automatically refreshed as you type if it recognizes a valid query.</li>
<li>Search the whole cluster or specify individual indices to run the query against.</li>
<li>Search Clips are read-only, so you can be sure that the data in your cluster will not be modified while working on a Clip.</li>
<li>Response View has tabs for Hits, Aggregations, Metadata, and Raw JSON for easy viewing of data in different ways.</li>
<li>Create a Clip and share it with your co-workers instead of copying and pasting queries. Your colleague can see exactly what you see.</li>
</ol>
<h2>Collaborate across teams within your company</h2>
<p>Bonsai wants you to be able to use the best tools for the task at hand. Like our Metrics Dashboard, data from Search Clips can be exported into third party services. Now any team in your company can view your search results in tools like Geckoboard or Google Sheets.</p>
<h2>Try it out today</h2>
<p>Search Clips are available on all Bonsai plans. Try it on your own. If you have feedback, let us know at product(at)bonsai.io.</p></div>

Collaborate on search at a more rapid pace with Search Clips

<div><h2>What to consider when designing your application’s search functionality</h2>
<p>Elasticsearch is a popular tool for application search. It has a short learning curve and a plethora of tools and clients to help integrate it into new projects and existing apps. That’s probably why it now boasts tens of millions of users, including tech superstars such as Netflix, Slack, SoundCloud, StackOverflow, and many more.</p>
<p>But building app search is kind of like snowboarding. It’s easy to get up and running, but not necessarily easy to master. It’s easy to overlook many of the features within Elasticsearch that can help you deliver a great search experience. Since most developers are using a database as their mental model for search, this document explores:</p>
<ol>
<li>why the requirements for app search are different than requirements for a database and</li>
<li>what to consider when designing your requirements.</li>
</ol>
<h2>Different tools for different requirements</h2>
<p>Search engines and databases perform similar functions on the surface: you push data in, and then it can be queried later. But these technologies were designed to address very different problems and constraints. With a database, the primary goal is “query millions of records by id or a range of numbers”, whereas a search engine’s primary goal is “query millions of records for relevant results by a value in the record.”</p>
<h2>Why are databases not the best search engines?</h2>
<p>Databases were designed to access data by a “key”, and databases can do this extremely well. Looking up a record by its primary key is an optimized solution in databases. We are always asking a database to find me the row in the persons table where the id equals 42. Databases are also really good at giving you all records between a given set of numbers.</p>
<p>However, databases are terrible at finding records where the word “milk” is somewhere in the notes field of the persons table. For this we need a new tool that is optimized for this task, let’s welcome the search engine. Search engines use a special data structure called an Inverted Index to achieve this record breaking speed. Because this is such a different way of thinking about how to access data we have to think about designing our data with a different mind set.</p>
<h2>What are the requirements for a good search experience?</h2>
<p>Since Elasticsearch must think about our data in such a different way than our databases do, we also have to change the way we think about our data. Additionally, in order to provide a great search experience we have to consider brand new concepts like misspellings that are non-existent in a database.</p>
<p>A good search engine should:</p>
<ol>
<li>Understand that we are dealing with fallible humans and correct for typos.</li>
<li>Understand that different people will use different words to find the same concepts. For instance, a &#8220;crisp&#8221; and a &#8220;chip&#8221; describe the same thing.</li>
<li>Effectively rank results with little real context, which is a fascinating world into itself.</li>
<li>Understand that a plural of a word and the singular form should return the same documents, a process handled in the tokenization phase.</li>
<li>Require as little work as possible on search, so we need to front load all of the work at the time of writing, not during the query.</li>
<li>Remove low value words to improve the quality of search results.</li>
<li>Handle ambiguities that are otherwise hard for computers like “milk chocolate” and “chocolate milk.”</li>
<li>Understand that we use different words when we search for things versus when we describe them.</li>
</ol>
<p>And the hardest part? All of this has to happen in milliseconds.</p>
<p>A good search experience doesn’t just involve your Elasticsearch index though. Content and UX also come into play. Are you planning to use faceting to surface search categories, or are you keeping things simple with just a search box? You&#8217;ll have to choose what data will be included for searching and what will be included for drawing the UI. None of these are factors when designing a database.</p>
<h2>Database + Elasticsearch = &lt;3</h2>
<p>So how do you choose a data store to use for your app that is reliable, but is also fast and can handle a variety of search queries? We recommend using both. In this arrangement, the database acts as a reliable “source of truth,” from which the Elasticsearch cluster is populated.</p>
<p>This approach has a couple of benefits:</p>
<ol>
<li>The transactions that make it into Elasticsearch have already been validated by the database</li>
<li>If anything happens to the cluster, it can be repopulated from a more durable source; and</li>
<li>You can conduct blue-green deployments much more easily.</li>
</ol>
<h2>Questions to Consider When Designing your Requirements</h2>
<p>Now that we’ve established how your search requirements are different than database requirements, what should your team decide before digging too deep into the Elasticsearch API?</p>
<ol>
<li>What will our search UX look like?</li>
<li>What fields will be stored in the index for showing on the UI vs what fields are just needed to support search?</li>
<li>How often will we test and tweak our mappings to improve search relevance?</li>
</ol>
<p>Understanding these requirements can take time, but will ultimately help your team in the long run.</p></div>

Designing app search <> designing a database

<div><p>After various tests by our team, Elasticsearch 7.2 is now available on Bonsai. This is a fairly major release, and has quite a few benefits as well as changes to note. We’re highlighting the ones that will likely impact our customers the most.</p>
<h2>Bonsai &lt;3 Rank Feature</h2>
<p>Elasticsearch is now exposing the <a href="https://lucene.apache.org/core/7_4_0/core/org/apache/lucene/document/FeatureField.html" target="_blank" rel="nofollow">Lucene FeatureField</a> type. This field type will let you store a precomputed value in the field, rather than having to compute the value during query time. Offloading this work to the indexing process will help keep your customers happy with snappy search response times.</p>
<p>In Elasticsearch, you can leverage this new feature by mapping your properties to the type <code>rank_feature</code>. Read up on the specific <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/rank-feature.html" target="_blank" rel="nofollow">Elasticsearch mapping syntax</a>. Once you’ve indexed your documents, you can they start to use the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-rank-feature-query.html" target="_blank" rel="nofollow">Elasticsearch query syntax</a> to search them using these new features.</p>
<h2>Script Scoring, Oh, My!</h2>
<p>Here is another hot off the presses feature that we are excited about. <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-score-query.html" target="_blank" rel="nofollow">Script Score Query</a> is Elasticsearch’s replacement of the “Function Score” which will soon be deprecated. Both of these features will allow you to tweak the scoring of matching documents after they have been retrieved, which reduces the amount of computation needed to score the various documents.</p>
<p>With Script Score Queries, you are no longer limited to a pre-defined set of functions but can leverage the scripting capabilities of Elasticsearch to their full extent, giving you more power to implement just the right relevance needed for you and your customers.</p>
<p>One trick that I really like is using the time based decay functions to temporarily boost new documents that might not get picked up by other boosting measures, such as a boosting based on popularity.</p>
<h2>Improved Search Response Time for Large Corpuses</h2>
<p>When you have a large number of documents in your index, and any given query might return thousands of documents, your customers may not be concerned with the exact number of documents that match the query. If there are over 10,000 documents that match my query, is your user really planning to page through all of those results, or are they only going to look at the first page or two?</p>
<p>With Elasticsearch 7, we can now specify the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-track-total-hits.html" target="_blank" rel="nofollow"><code>track_total_hits</code></a> parameter in a query. This will help reduce the amount of time a query takes, since it will no longer have to count the full result set of low value scoring documents. For any result set less than the value set in <code>track_total_hits</code>, Elasticsearch will return the correct number. For values over that amount, the query will tell you that it was greater than the number. Then you can draw your UI correctly for the user.</p>
<p>Faster top k queries is a key new feature for improving search results. Elasticsearch 7.0 implements a new algorithm that provides a huge speed boost when retrieving top hits for any given search query.</p>
<h2>Upgrade with no downtime</h2>
<p>Upgrading to Elasticsearch 7.0 is pretty straightforward. Check out <a href="https://docs.bonsai.io/article/126-upgrading-major-versions" target="_blank" rel="nofollow">these step-by-step instructions</a>.</p></div>

Bonsai now supports Elasticsearch 7.2

<div><p><img src="/images/blog/haystackconf.jpg" alt=""></p>
<p>The <a href="https://haystackconf.com/" target="_blank" rel="nofollow">Haystack Conference</a> is the premiere event for search relevance. It took place April 22-25th
in Charlottesville, VA. A few members of the Bonsai team were in attendance to hear about the latest innovations in search
relevance. If you’re looking to improve your search results but aren’t sure where to start, here are some takeaways from a
couple of talks that will help you get started.</p>
<h2>I. A lot of teams were using the <a href="https://github.com/o19s/elasticsearch-learning-to-rank" target="_blank" rel="nofollow">Learning to Rank</a> plugin.</h2>
<p>This plugin was written by <a href="https://opensourceconnections.com/" target="_blank" rel="nofollow">Open Source Connections</a>, and uses Machine Learning to improve
search ranking. Bonsai customer Snag is a jobs marketplace, and is a big user of this plugin. Xun Wang at <a href="https://www.snagajob.com" target="_blank" rel="nofollow">Snag</a> gave the talk
<a href="https://haystackconf.com/2019/human-judgement/" target="_blank" rel="nofollow">“Towards a Learning to Rank Ecosystem at Snag.”</a> Here are some of the issues
they overcame to make LTR successful:</p>
<ol>
<li>Bots are common on Snag’s platform. Bot behavior can contaminate results even if they are just 5-10 percent of training data.</li>
<li>Keyword stuffing in job descriptions is also common. Snag overcame this by using topical keywords.</li>
<li>Intent can be difficult to infer. If a user doesn’t click on a result from the page, are they actively skipping them, or
was it passive neglect? To better train their learning models, Snag incorporates the last clicked result into their query
logs.</li>
<li>Deduping job results in a Search Engine Results Page (SERP) improves search quality. Job seekers don’t want to see too many
jobs from the same company. To dedupe their SERPs, Snag uses <a href="https://en.wikipedia.org/wiki/Locality-sensitive_hashing" target="_blank" rel="nofollow">Locality-Sensitive Hashing</a>
to tag duplicates/near duplicates,
and then tunes results accordingly.</li>
<li>Snag’s big measurement of success is “Yield.” Snag must be good at producing relevant results for users, AND benefit
job posters. Yield is a measurement created by the Snag team that factors in both. Search has multiple stakeholders, and
it’s important to bring in these stakeholders when defining what success looks like.</li>
</ol>
<h2>II. Many teams also use experts to gauge the quality of search results.</h2>
<p><a href="https://lexisnexis.com" target="_blank" rel="nofollow">LexisNexis</a> is a provider of legal, government, business, and high-tech information sources.
Currently, they reference three petabytes of legal and news data with 65 billion documents, which is three times the size
of Wikipedia. LexisNexis hires legal experts to evaluate their findings. Here is their processed laid out in their talk
<a href="https://haystackconf.com/2019/human-judgement/" target="_blank" rel="nofollow">“Making the case for human judgment relevance testing”</a> by Tara Diedrichsen and Tito Sierra.</p>
<ol>
<li>First, they select raters based on their subject matter expertise, cost, and capacity. Each rater is given metrics to
gauge their success.</li>
<li>Then, they select queries to measure. LexisNexis uses actual user queries as much as possible, but sometimes they test
constructed queries.</li>
<li>The raters are then asked to rank the results for a query on a scale from 1-4, based on a document’s permissibility
in court. LexisNexis also gathers verbatim comments for these results.</li>
<li>Scores are then generated. Like many search relevance teams, the LexisNexis team uses the industry standard <a href="https://en.wikipedia.org/wiki/Discounted_cumulative_gain" target="_blank" rel="nofollow">Discounted
Cumulative Gain</a> to measure the quality of search results. This allows them to compare the current search results with the
feedback from raters.</li>
<li>The results are then communicated to various stakeholders within LexisNexis, including Development, Product, and other
teams within the organization.</li>
</ol>
<p>These were just a few out of many great talks from Haystack. If you’re interested in watching the videos when they are
posted, tune into the <a href="https://haystackconf.com" target="_blank" rel="nofollow">Haystack website</a>.</p></div>

Haystackconf Recap

<div><p>Countless blog posts and Hacker News threads have asked a simple question: Heroku or AWS? Sometimes you want the convenience of Heroku, and sometimes, you need the scale and control of AWS. We want to make it easy to completely secure Bonsai with any app, no matter where you host it.</p>
<p>It’s always been easy to deploy Bonsai from the Heroku CLI, and now you can deploy Bonsai within your own AWS account using a <a href="https://bonsai.io/fully-managed-elasticsearch-in-aws-vault" target="_blank" rel="nofollow">Bonsai Vault</a>. We’ve had a lot of demand for this option. Thus far, our beta customers are quite pleased.</p>
<p><img src="/images/blog/vault-intro.svg" alt=""></p>
<h2>What are the primary advantages of using a Bonsai Vault if you’re on AWS?</h2>
<ol>
<li><strong>It’s secure and can help meet compliance requirements.</strong> Bonsai Vaults are actually deployed in your AWS account using
the AWS Organizations API. Clusters deployed in Bonsai Vaults do not communicate with your app over the public internet.
It’s one extra measure your team can take to protect against hackers and ransomware.</li>
<li><strong>You can save money by using your own AWS reserved instances to host your cluster.</strong> Reserved instances can be up to 75%
cheaper than on-demand instances. By opting for this option, you’re saving money while still getting the convenience of Bonsai. It’s a win/win.</li>
<li><strong>It’s fully managed and supported by a staff of Elasticsearch experts.</strong> Get the benefit of cost at scale, without
the hassle of managing that scale on your own. Our team will still offer the same kind of white glove support you’d get
on our <a href="https://bonsai.io/pricing/enterprise" target="_blank" rel="nofollow">Enterprise tier</a>.</li>
</ol>
<h2>What if you’re still using Heroku?</h2>
<p>While Heroku doesn’t enable add-ons to be deployed in Heroku Private Spaces, we do offer VPC peering with them.
It’s an increase in security and can help meet compliance requirements. See <a href="https://bonsai.io/fully-managed-elasticsearch-in-heroku-vault" target="_blank" rel="nofollow">our overview on Heroku Vaults</a> for more details.</p>
<h2>Getting started with a Bonsai Vault</h2>
<p>Bonsai Vaults have been in beta mode for months, and are now available in General Availability. <a href="https://bonsai.io/contact" target="_blank" rel="nofollow">Contact our team</a> if you’re interested in setting up your own Bonsai Vault.</p></div>

Deploy Bonsai in your own AWS account with Bonsai Vaults

<div><p>Black Friday is coming up soon, and a lot of eCommerce companies are doing everything they can to prepare. Elasticsearch has some handy under-the-hood features that can improve how much money your search engine can generate for your site. Whether you’re using Elasticsearch for an eCommerce site or not, it never hurts to know different ways to better tune your search results.</p>
<p>Say a customer comes to your website searching for “gloves.” Your search engine diligently goes off and retrieves every document with the word “gloves” in it. You could easily imagine there would be a multitude of listings. To drive the point home, here is a hypothetical data set for this:</p>
<pre><code>[{
  "item_name": "Basic Gloves",
  ... various attributes
  "meta_sales_count": 90,
  "meta_margin": .20,
  "meta_age_days": 450
},{
  "item_name": "Luxury Gloves",
  ... various attributes
  "meta_sales_count": 40,
  "meta_margin": .30,
  "meta_age_days": 2
}]
</code></pre>
<p>If you are a keen business person, you might notice to your surprise that the Luxury Gloves are ranking significantly lower than a group of Basic Gloves. But the margin on those Luxury Gloves is so much higher! How do I get the Luxury Gloves to show up higher in search results than the Basic Gloves?</p>
<p>Thankfully, in this example the developer has indexed into these documents a hidden (in that it’s not shown on the search results page) meta field that describes the item’s margin called “meta_margin”.</p>
<p>Elasticsearch makes it easy to alter the default scoring mechanism. By default Elasticsearch only scores your documents based on your query - basically, on how closely the word “gloves” matches the text in the document. In order to adjust the document scores based on the “meta_margin” field, we are going to use an Elasticsearch feature called boost. Using boost you can, at query time, change where a document will show up in the search results, by applying a positive “factor” to the field we can get higher margin products to sort to the top (If you use a negative number they will go down the search results page, and is often called a “bury”). It’s important to remember that with the great power to alter the search results comes greater responsibility to give our customers what they are actually looking for. Therefore, in practice, I will prefer this boost to be a relatively small number, my goal is that it becomes more of a tie-breaker, rather than something used to dramatically change results.</p>
<p>Example query using boost:</p>
<pre><code>GET ~/_search

{
  "query": {
    "function_score": {
      "query": {
        "match": {
          "item_name": {
            "query": "gloves"
          }
        }
      },
      // https://www.elastic.co/guide/en/elasticsearch/guide/2.x/boosting-by-popularity.html#_modifier
      "field_value_factor": {
        "field":    "meta_margin",
        "modifier": "log1p",
        "factor":   1.01
      },
      "boost_mode": "sum"
    }
  }
}
</code></pre>
<p>As with any change to your query methodology, you will want to test the before and after results of this change and make sure that you aren’t impacting your search results in unexpected ways.</p></div>