Introducing: A search & relevancy assessment from engineers, not theorists. Learn more
Upgrading your Bonsai cluster to a more powerful one is a straightforward process, designed to minimize downtime and maintain the integrity of your data. This document outlines the steps involved, the expected downtime, and the effort required for a successful upgrade.
Bonsai utilizes a two-phase snapshot and restore process to migrate your data from one hardware cluster to another. A snapshot of your data is taken from the current hardware and restored to the new new hardware. There is no downtime or production impact to your cluster during this phase. This phase of the upgrade can take time, depending on how much data you have on your existing hardware.
Once this is completed, a second snapshot-restore operation is performed. This operation is simply a delta — covering only the data that has been changed since the phase-1 shapshot was taken. Usually this second phase lasts a few seconds, or maybe a minute. During this phase, your cluster is placed into a read-only mode. Search traffic is not impacted, but writes are blocked until the restore completes.
Once this final restore has completed, the cluster will be running entirely on the new hardware, with no data loss.
403 Forbidden along with a JSON response indicating that Cluster is currently read-only for maintenance. Please try your request again in a few minutes. See [status.bonsai.io](<http://status.bonsai.io/>) or contact [support@bonsai.io](<mailto:support@bonsai.io>) for updates.. It's essential to handle this gracefully in your application.
To begin, navigate to the cluster’s Plan under Settings on the cluster dashboard.
If you haven’t added billing information yet, please do so at Account Billing. Detailed steps can be found in Add a Credit Card.
Once there is billing information listed on the account, you will see the different options for changing the cluster’s plan. Select the plan you would like to change to and click the green Change to … Plan button. In this example, this `documentation_cluster` is on a Sandbox plan and it will be upgraded to a Standard Micro plan.
A successfully changed plan will notify you at the top with Plan scheduled for update. The Plan has been upgraded to the Standard Micro plan in this example.
Please note that downgrading a plan may fail to update if it puts the cluster in an Overage State for the new plan. For example, downgrading a Standard Micro plan to a Sandbox plan will fail if there is 11 shards on the cluster (Sandbox plan has a shard limit of 10).
Bonsai clusters can be provisioned in a few different ways. These instructions are for users who want to be billed directly by Bonsai. If you are a Heroku customer and are adding Bonsai to your app, then you will not need these instructions. Instead, check out:
Creating an account on Bonsai is the first step towards provisioning and scaling an Elasticsearch cluster on demand. The following guide has everything you need to know about signing up.
To start the sign up process, head to our Sign up page, where you’ll begin by providing account information. Click Create Account to continue.
In this section, you can provide some extra project details to allow our project team to best support your cluster. We have a few questions for you to answer to tailor your experience when using Bonsai.
Click Next to move on.
Fill out your cluster name, then select a release and a region. All hobby and testing clusters are provisioned with the latest version. More information about our version and region support can be found here.
You don’t have to sign up with payment details - your first testing cluster is on us! If you wish to provision or upgrade to a larger cluster with more resources, you’ll need to add a credit card after confirming your email.
Click Provision Cluster to complete your signup.
If everything checks out, you should receive an email from us shortly with a confirmation button.
Navigate to your email client and click on Confirm email.
You’re all set! Thank you for signing up.
If you don't receive the sign up email, please check your Spam folder. You can also reach out to us for help.
A: We validate emails using RFC 2822 and RFC 3696. If you have a non-conforming email address, let us know at support@bonsai.io.
A: Yes! We’re a security-conscious bunch, but we don’t have any arcane rules about what kinds of characters you must use for your password. Why? We’ll let xkcd explain it, but the tl;dr is our password policy simply enforces a minimum length of 10 characters. We also reject common passwords that have been pwned. Sadly, correct horse battery staple appears in our blacklist.
We also highly recommend using a password manager like 1Password to generate unique, secure passwords.
A: If you have signed up for Bonsai in the past using this email address, you will receive an email directing you to log in using it.
A: You will want to select a region that’s as close to where your application is hosted as possible to minimize latency. Doing so will ensure the fastest search and best user experience for your application.
Bonsai is built on Amazon Web Services (AWS) and Google Cloud Platform (GCP), and we run clusters in several of their regions. Users are able to provision clusters in the following regions:
These regions are supported due to broad demand. We can support other regions as well, but pricing will vary. Shoot us an email if you’d like to learn more about getting Bonsai running in a region not listed above.
A: Users are able to manage multiple clusters through their Bonsai dashboard once they have confirmed their email. Not only is the cluster name a label for you to distinguish between different applications and environments, but it’s also used as part of the cluster’s unique hostname.
For example, if you name your cluster “Erin’s Exciting Elasticsearch Experiment,” that will be the name of your cluster. The host name of your cluster URL will be automatically generated into something like erins-exciting-elasticsea-123456.
Cluster names can be changed later, but the host name that is generated when the cluster is first created is immutable.
As always, feel free to email us at support@bonsai.io if you have further questions.
The Bonsai cluster dashboard’s Overview provides a series of useful metrics about your cluster.
This article will cover the following:
Overview provides general information about your cluster at the top:
The account name, cluster’s name, and health status dot (which will be green, yellow or red) is found here. Below that, the region the cluster is provisioned in, the version of Elasticsearch it’s running, and the subscription plan tier is display.
The first component is the Performance heatmap:
This heatmap reveals how fast requests are. Each column represents a “slice” of time. Each row, or “bucket”, in the slice represents duration speed. The"hotter" a bucket is colored, the more requests there are in that bucket. To further help visualize the difference in the quantity of requests for each bucket, every slice of time can be viewed as a histogram on hover.
You can check out our Metrics documentation for a more detailed dive into cluster metrics.
Traffic Summary highlights several statistics in the last 24 hours:
This component shows the cluster’s current usage versus the limits of your plan for 3 items:
This component indicates how your data is allocated across your cluster. If the allocation seems radically unbalanced, that can be an indication that you should reindex your data with an updated shard scheme. Documentation on this can be found in our Capacity Planning documentation.
If you upgrade to a Business / Enterprise cluster, you may see some extra nodes appear here, and may further observe that these nodes have few or 0 shards allocated. This is expected. These extra nodes will be cleaned up and removed later.
Enterprise subscriptions support private multitenancy. For clusters running on these subscriptions, there will also be a Tenants table that lists tenants on the cluster:
Bonsai cluster dashboard’s Metrics is the place to troubleshoot cluster traffic issues and view performance metrics. This article will cover:
Metrics is located in each cluster’s dashboard. Log into Bonsai, click on your cluster, and click on Metrics within the left sidebar:
Use this selector to choose between four window sizes for metrics in the:
Click the left and right arrows to go back or forth in time within the selected window size.
Click on the timezone to toggle between displaying the graph timestamps in UTC time or your local browser timezone.
You can drill down to smaller windows of time on any graph by clicking and dragging to select a time range.
More information doesn’t necessarily mean more clarity. When something happens to your traffic and cluster responses, it’s important to know how to see your metrics and draw conclusions.
We’ll cover what each graph displays and some examples of what they will look like given certain use cases (such as periods of high-traffic, or clusters in a normal state compared to ones that are experiencing downtime). We’ll start with the most information-dense graph: the Requests heat map.
This graph reveals how fast requests are. Each column in the graph represents a “slice” of time. Each row, or “bucket”, in the slice represents duration speed. The "hotter" a bucket is colored, the more requests there are in that bucket. To further help visualize the difference in the quantity of requests for each bucket, every slice of time can be viewed as a histogram on hover.
Example 1
This heat map displays a cluster with consistent traffic with most requests in the 200-300ms range to complete.
Example 2
This cluster has light, sporadic traffic. It’s important to note that the "heat" color of every bucket is determined relative to the other data in the graph - so a side-by-side comparison of two request heat maps using color won’t be accurate.
This graph shows the number of requests handled by the cluster at a given time.
This graph, similar to the Requests heat map, shows a distribution of request speed based on 3 percentiles of the requests in that time slice: p50 (50%), p95 (95%), and p99 (99%). This is helpful in determining where the bulk of your requests sit in terms of speed and how slow the outliers are.
Proxy Queue time is the total amount of time requests were queued (or paused) at our load balancing layer. Queue time is ideally 0; however, in the event that you send many requests in parallel, our load balancer will queue up requests while waiting for executing requests to finish. This is part of our Quality of Service layer.
Concurrency shows the number of requests that are happening at the same time. Since clusters are limited on concurrency, this can be an important one to keep an eye on. When you reach your plan’s max concurrency, you will notice queue time start to consistently increase.
This graph shows the amount of data crossing the network - going into the cluster (shown in green) and coming from the cluster (in blue).
We expect most bandwidth graphs to look something like the graph below — a relatively higher count of "From Client" data (read requests) compared to "To Client" data (write or indexing requests).
The relationship between green to blue bars in this graph really depends on your use-case. A staging cluster, for example, might see a larger ratio of Write:Read data. It’s important to note that this graph deals exclusively in data - a high-traffic cluster will probably see a lot of data coming “From” the cluster, but a low-traffic cluster with very complicated queries and large request bodies will also have a larger “From Client” data than would otherwise be expected. Therefore, it’s helpful to look at request counts to get a feeling for the average "size" of a request.
This graph can do two things:
It’s important to note while reading this graph, that 5xx requests don’t necessarily mean that your cluster is down. A common situation on multitenant plans is a cluster that’s getting throttled by a noisy neighbor who’s taking up a lot of resources on the server. This can interrupt some(but not all) normal behavior on your cluster, resulting in a mix of 2xx and 5xx requests.
Tolerance for a few 5xx requests every now and then should be expected with any cloud service. We’re committed to getting all production clusters a 99.99% uptime (i.e. expected 0.001% 5xx responses), and we often have a track record of four 9’s and higher.
We have a lot of people that are very sensitive to 5xx requests. In these cases, it’s usually best to be on a higher plan or a single tenant plan. Reach out to us at support@bonsai.io if this is something your team needs.
Clusters running on Business and Enterprise grade clusters will have access to some additional metrics. If you would like to get access to these metrics for your cluster, please reach out and our team will walk you through the process.
System load is the average number of processes waiting on a resource within a given period of time. This is often reported in 1, 5 and 15 minute windows. The Bonsai dashboard shows the system load average over the past minute.
It is helpful to think of system load as how saturated a node is with tasks; as long as the node's load average is lower than the number of its available CPUs, the node is able to handle all of its work without getting backed up. If the load average is larger than the number of its available CPUs, that means that some tasks are being scheduled. When tasks are delayed like this, performance suffers, and the performance impact is correlated to how high the load average gets.
Elasticsearch utilizes a number of thread pools for handling various tasks. These pools are also backed by a queue, so that if an tasks is created and a thread is not available to execute it, the task is queued until a thread becomes available. This metric shows the total number of tasks sitting in an Elasticsearch queue.
It's important to note that this is not the same as a request queue. A single request can result in multiple tasks being created within Elasticsearch. It is also important to note that these queues have a finite length. If the queue is full and an additional task is created, Elasticsearch will reject it with a message like "rejected execution (queue capacity 50)" and an HTTP 429 response.
This metric shows the number of _bulk requests that have been processed over time. Bulk requests are an efficient way to insert or update data in your cluster. Naturally, your payload sizes need to be more than 1-2 documents in order to get the benefits of bulk updates. Usually batches of 50-500 are ideal.
This metric shows the number of search requests that have been processed by Elasticsearch. It's important to distinguish between user searches and shard searches. A user search is performed by your application (often in response to some user action or search in the app), and may translate into multiple shard searches. This is true if your indices have multiple primary shards, or if you're searching across multiple indices. A query for the top X results will be passed to all relevant shards; each shard will perform the search and return the top X results. The coordinating node is then responsible for collating those results, sorting, and returning the top X.
Elasticsearch does utilize a thread pool specifically for searches, so depending on the types of searches you're running, and the volume of search traffic, it's possible to get a message like "rejected execution (queue capacity 50)" and an HTTP 429 response.
Elasticsearch is a Java-based search engine that runs in the Java Virtual Machine (JVM). The JVM has a special area of memory where objects are stored, called "heap space." This space is periodically garbage-collected, meaning objects that are no longer in use are destroyed to free up space in memory.
This metric shows the percentage of heap space that is currently occupied by Elasticsearch's objects and arrays. Lower is ideal, because high heap usage generally means more frequent, and longer-lasting garbage collection pauses, which manifests as slower performance and higher latency.
This metric shows the amount of time in milliseconds that the JVM spent reclaiming memory from new or short-lived objects. This type of garbage collection is expected and shouldn't lead to HTTP 503 or 504 errors. However, if it is chronic and frequent, it may lead to slow performance.
This metrics shows the amount of time in milliseconds the JVM spent reclaiming memory from long-surviving objects. The JVM periodically pauses the application so it can free up heap space, which means some operations are stopped for a period of time. This can result in perceived slow response times, and in some extreme cases can lead to system restarts and HTTP 503 and 504 responses.
This metric shows the percentage of the time that the CPU(s) waited on IO operations. This means that an operation requested IO (like reading or writing to disk) and then had to wait for the system to complete the request. A certain amount of IOWait is expected for any IO operation, and usually it's on the order of nanoseconds. This isn't indicative of a problem on its own.
Excessive wait times are problematic though. It is usually correlated to high system load and a high volume of updates. Sometimes that can be addressed through hardware scaling or better throttling of updates. In rare cases it can indicate a problem with the hardware itself, like an SSD drive failing.
This metric shows the percentage of the time that the CPU(s) were executing code in user space. The user space is where all code runs, outside of the operating system's kernel. On Bonsai, this space is primarily dedicated to Elasticsearch, so the metric is roughly the amount of time the CPU spent processing instructions by the Elasticsearch code.
The metric can vary widely between clusters, based on application, hardware and use case. There is not necessarily an ideal value or range for this metric. However, large spikes or long periods of high processing times can manifest as poor performance. It often indicates that the hardware is not able to keep up with the demands of the application, although it can sometimes indicate a problem with the hardware itself.
Bonsai offers support for automatically removing old indices from your cluster to save space. If you’re using Bonsai to host logs or some application that creates regular indices following a time-series naming convention, then you can specify one or more prefix patterns and Bonsai will automatically purge the oldest indices that match each pattern. The pattern will only match from the start of the index name.
This feature is found in the Cluster dashboard, under the Trimmer section:
For example, assume you’re indexing time-series data, say, number of support tickets in a day. You’re putting these in time-series indices like "support_tickets-201801010000", "support_tickets-201801020000", and so on.
With this feature, you could specify a pattern like "support_tickets-", and we’ll auto-prune the oldest indices first when you reach the size limit specified for the pattern. Indices scheduled for removal will be highlighted in red.
Please note we will not purge the last remaining index that matches the pattern, even if the size is above the limit.
The trimmer feature only allows you to delete whole indexes and only if there are more than one with the same trimmer pattern. The trimmer does not delete just certain documents in an index. To remove a number of documents in an index your best option is to use delete_by_query.
Here is an example for deleting the 50 oldest documents, according to a hypothetical "date" field:
<div class="code-snippet w-richtext"><pre><code fs-codehighlight-element="code" class="hljs language-javascript">POST //_delete_by_query?conflicts=proceed
{
"query": {
"match_all": {}
},
"sort": {
"_script": {
"type": "date",
"order": "asc" }
},
"size": 50
}</code></pre></div>
The Elasticsearch Search API can be used to identify documents to delete. Please use your Console (or an equivalent tool) and build your query using the Search API first, and verify that you will only be deleting the documents you want. Do this before running the delete_by_query call against your query.
To navigate to your personal profile, click on your initials in the upper right corner and select Profile Settings from the dropdown menu. Then navigate to the Security tab.
Single Sign-On (SSO) is the ability to have a third party service validate your identity. You can enable Google SSO which offers additional security like multi-factor authentication (MFA). Bonsai also supports Okta.
To use this feature, your identity provider must match your Bonsai.io account email address. For example, if your Google email address is "bob.smith@gmail.com," then your Bonsai.io account must use this same email address in order to verify your identity.
Once you have SSO set up, you will no longer be able to log in with your username/password. Logging in will need to be done through the identity provider.
To revert back to username/password authentication, you will need to disable SSO. To do so, simply click on Disable SSO.
If you see this section greyed out then your account admin has required that you use SSO.
To update your password, enter your old password and a new password. Bonsai strongly recommends using a password manager like 1Password or LastPass to keep your passwords secure, and to help randomly generate new passwords.
Protip: Use a strong password
We’re a security-conscious bunch, and we don’t have any arcane rules about what kinds of characters you must use for your password. Why? We’ll let xkcd explain it. Tl;dr: our password policy simply enforces a minimum length of 10 characters. We also reject common passwords that have been pwned. Sadly, correct horse battery staple appears in our blacklist.
Note: updating your password will revoke all of your active sessions and force you to log in again.
View and revoke your active sessions by scrolling down to Active Sessions. If you have a session on another device, you can see its IP address and information about the device.
You can revoke sessions individually, or revoke all. Revoking all sessions will also revoke your current session that you are using to view your profile, and doing so will require you to log in again.
You can add or delete a credit card, update your payment profile, or add coupon codes in the Billing tab of Account Settings.
To begin adding a credit card to the account, click on Add credit card to bring up a new form.
Fill in the credit card information and click Add Card to proceed. Click Cancel to go back.
Your Financial Data, Secured
When you add a credit card in Bonsai, the information is encrypted and passed to our payment processor. We have verified that this service is Level 1 PCI Compliant. This level of compliance is the highest level of security a business can offer. We do not host or store any financial data, and your credit card details can not be accessed by anyone on our team or within the payment processor.
A successfully added credit card will be listed.
Once there is at least one card listed in Credit Cards, click on the overflow menu icon to reveal a drop-down menu to View Details on that card.
Card Details lists the credit card information.
You can associate multiple credit cards to your account. If you only have one credit card on file, it will automatically be the default. If you have multiple credit cards, you have the option to update the default credit card for payment.
To set another credit card as default, click on the overflow menu icon to reveal a drop-down menu to Set Default on a that card.
To remove a credit card from an account, click on the overflow menu icon to reveal a drop-down menu to Delete a credit card.
Please note, you can delete a default credit card if all 3 of the following statements are true:
If you are having trouble deleting a default card, let us know at support@bonsai.io.
Once a credit card has been successfully added, the Address section will appear under Credit Cards. Adding an address here will aid in applying taxes based on your location. If tax is charged, then at what rate depends on this address. See Bonsai's Policy on Collecting Sales Tax for further information.
Clicking on the Add Address button will take you to a form where you can use an existing address populated from your credit card or fill out a new address.
A successfully added address will take you back to the Billing tab.
If you have a coupon code, you can add it in the Coupon Code section. Simply enter the code and click on Apply Coupon.
If the code is accepted, it will be listed under Coupon Codes.
If the code is not valid, there will be an error message.
If you have a code that isn’t working but should, shoot us an email at support@bonsai.io and we’ll check it out.
Download and view the account statements.
You and the people you refer can earn credit that will be applied to your account to on paid Bonsai plans. We'll cover how this process works in the following guide.
How it works:
<ol>
<li>Send your unique referral URL to a new user.
<ul>
<li>Enter the email of the person you are referring under your Referrals tab, and we’ll automatically send them an invite email with everything they need to know.</li>
and/or
<li>Copy your shareable link, and share your URL via email, social media, or whatever method works for you.</li>
</ul>
</li>
<li>The new user signs up using your referral URL.</li>
<li>The new user pays for their first month on a paid cluster plan.</li>
<li>You earn $50 AND they receive $50 toward a paid Bonsai plan.</li>
<li>You can send your referral URL to as many people as you want, but you will stop earning credit once you have reached the limit of $50.</li>
</ol>
To begin, navigate to Account Settings then to the Referrals tab.
Under the Referrals tab you will be able to access your unique referral URL and see how many referrals have been sent and accepted:
Entering a new user's email address and clicking Invite will automatically send them an email with a detailed explanation of how to sign up for Bonsai using your referral URL. A success message on the dashboard will notify you that your invite has been successfully sent.
The email we send to new users looks like this:
Clicking Copy Link will automatically copy your referral URL to your clipboard.
Paste your unique referral URL to share it with new users through your personal email, social media, or wherever!
<ol>
<li>When a new user is directed to our Sign Up page using your referral URL, they will see the following page:
<figure class="w-richtext-figure-type-image w-richtext-align-center"><div><img src="https://global-uploads.webflow.com/63c81e4decde60e815417fc3/6464200ec961bced99877436_6410e020d4469.png" alt=""></div></figure>
</li>
<li>By completing the sign up process through your referral URL, their account will be marked as having been created with your referral code</li>
<li>Once the new user has upgraded their cluster to a Staging plan or higher, you and the new user will both receive a credit of $50 towards the next monthly bill. Hooray!</li>
</ol>
A: Only accounts with Standard an up clusters have access to Referrals.
The Referrals tab will show up in your Account Settings once you have at least one paid cluster in your account.
A: In order to receive the credit your friend has to both sign up for a new account with your referral URL and upgrade to their cluster to Standard, Business, or Enterprise. If you don't see your credit after that, please let us know at support@bonsai.io. We're happy to get this sorted out for you.
A: No, your credit will automatically be applied to the following month’s payment.
A: No. Since this is a one-time credit, it can only be applied to the following month.
Logs are real-time they only appear on this page while requests are happening. Logs do not persist.
Bonsai provides a real-time stream of all the requests hitting your cluster. There is also a subtab for the Top 20 Slow Requests. This will begin to populate if requests slow down measurably.
The streaming logs show a timestamp, HTTP verb and endpoint, along with how long it took Elasticsearch to respond and what HTTP response code was returned. Request/response payloads are not captured.
Bonsai does not expose server logs at this time.
Credential management is imperative to tracking who and what has access to your cluster. At Bonsai, regardless of plan level, every request made to your cluster requires a username and password. Security is a default, not an upgrade.
With the Credential Management section of the cluster dashboard, you can add and remove access credentials. In this guide, we will cover:
You can see your current credentials and generate new ones by logging into your cluster dashboard and navigating to the Credentials section.
Every cluster has a Master credential that is created when the cluster is provisioned. The Master credential can never be revoked, only rotated.
There are three types of credentials you can generate:
With custom auth controls you can specify things like:
You would want to regenerate your default credentials if your fully-qualified URL has been linked (say, if somehow was copy-pasted into an email, GitHub issue, or, perish the thought, StackOverflow). To do that, simply click the yellow Regenerate button. This will instantly regenerate a new, randomized authentication pair.
The old credentials will remain active for two minutes or so. After that time the old keys are revoked. The purpose of the two minute warning is to give administrators the opportunity to update their application with the new credentials before the old ones expire.
To create a new credential, click on the Create Credential button.
Choose one of the three types (full-access, read-only, or custom), give it a name, and then click Generate.
We advise giving your credential a human-friendly name, like ACME_frontend, python_indexer, or docs_search_component. It’s an easy way to help you and your teammates remember how each credential is used. When you generate a new credential, Bonsai shows your credential details.
Credential Details
You can view who created the credential (in this example, Leo), the access key and secret (username and password), the allowed settings, and some quick links.
This displays what indices are accessible, which Elasticsearch actions are allowed, and if there are whitelisted IP’s or CIDR blocks. If your cluster is on the Business tier and above, these fields are customizable.
The Elasticsearch access url is excellent for pasting into a terminal and executing curl commands. Use Kibana access to launch your Bonsai-hosted instance Kibana, included with every Bonsai cluster. Read more about Bonsai hosted Kibana here.
Full-access tokens are best used for back-end applications that handle indexing or act as a proxy for user’s input for querying.
When choosing this type, the form will pre-populate with the allowed action (or ‘privilege’) `indices:data/read/*`. This allowed read-only specific actions: count, explain, get, exists, mget, get indexed scripts, more like this, multi percolate/search/termvector), percolate, scroll, clear_scroll, search, suggest, tv.
If you have specific security needs, generate a custom credential. Increase your team’s security tolerance by using custom credentials for things like limiting index actions to only certain IP addresses, or making certain indices search-only. There are three fields for custom credentials: Indices, Actions, and IP/CIDRs.
This sections allowed you to list a set of indices that are permitted, or create a pattern such as "logs_2019-12-*":
Leaving this blank will allow all indices present on the cluster accessible.
Specify access privileges from the searchable dropdown:
If you ever need help figuring out exactly which actions map to your needs, please email support and we’ll point you in the right direction. Leaving this blank will allow all access privileges.
Use this section to control where you allow requests to be made from. Whitelist individual IP addresses for monitoring privileges, or write a CIDR block that only allows your company to access an internal-only index or cluster. Leaving this blank will allow any IP address by default.
Further reading:
If you receive a notice to upgrade for access to read-only or custom credential management, you’ll need to navigate to Manage tab and upgrade your cluster to Standard, Business, or Enterprise:
Clicking on the Upgrade this cluster link will take you to your management dashboard, where you can upgrade to a Business or Enterprise subscription.
Changing a cluster's name is quick and easy! From the cluster dashboard, simply navigate to the Settings link under the Manage header:
Under the "Edit Cluster Name" form, enter the new name for the cluster, and click on "Save."
Changing your cluster's name will not result in downtime or loss of data. It will also not change the cluster's URL. If the name is already taken by an active cluster in your account, you will receive an error.
Destroying your cluster is simple. In this doc we cover:
If you need to deprovision(destroy) a cluster, navigate to your cluster, and head to the settings section of the dashboard.
Once you verify your account password in the form, you’ll be able to click the Deprovsion button.
Once a cluster has been deprovisioned, it is destroyed immediately. The data is deleted, it will instantly stop responding, and all requests will return an HTTP 404.
When you deprovision a paid cluster on Bonsai, you will automatically be credited with a prorated amount. This credit will apply to future invoices. For example, if you had a $50/mo cluster, which you destroyed after 3 weeks, you would get(roughly) $12.50 credit applied to your account.
Note
This section does not apply to Heroku users. If you're a Heroku user, check out:
I don’t see the Settings section on my dashboard.
If you can’t find the Settings section on a cluster dashboard, then that means your Role on your team is either Member or Billing , neither of which can deprovision a cluster. Contact your team Admin to change your role, or handle the deprovision themselves. Read more about how teams on Bonsai work.
What if I accidentally deprovisioned my cluster?
The data in your cluster will be automatically deleted. Bonsai retains hourly snapshots for the past 24 hours, and daily snapshots for the past 7 days. So if you accidentally deleted data you needed, our team might be able to provide a partial restore, depending on when we are alerted. Contact us for more help.
Bonsai offers support for integrating with other services. This menu can be found by clicking on the Integrations tab of your cluster dashboard.
Bonsai currently offers the support for the following integrations:
We are always looking for new services to integrate with, so if you would like to see Bonsai add support for another service, please reach out and let us know.
The Console is a web-based UI for interacting with your cluster. Think of it like a user-friendly version of curl. If you want to try out some queries or experiment with the Elasticsearch API, this is a great place to start:
The UI allows the user to select an HTTP verb, enter an endpoint and run the command. The results are shown in the navy console on the right. The box below the verb and endpoint boxes can be used to create a request body.
Some places in the Elasticsearch documentation suggest using a GET request even when passing a request body. An example would be something like:
<div class="code-snippet w-richtext"><pre><code fs-codehighlight-element="code" class="hljs language-javascript">GET /_search
{
"query" : {
"term" : {
"user" : "george"
}
}
}</code></pre></div>
Without getting bogged down in pedantry, GET is a questionable verb to use in this case. A POST is more appropriate. It’s what the UI is assuming anyway. If you paste in a request body and use a GET verb, the body will be ignored.
Need to update a credit card or provide access to your clusters to a team member? Use the sections below to manage all things account-related.
To access your Account Settings, click on the upper right-hand dropdown and select Account Settings.
From your account settings page, you can…
Control security within your whole team in your Account Security tab by enabling Single-sign on for your Account.
Single-sign on (SSO) is the ability to have a third party service validate your identity, like Google and Github. Bonsai currently support Google SSO. You can enable Google SSO which offers additional security like multi-factor authentication (MFA).
Individual team members can finalize this setup in their Profile Setting's Security tab.
Once Google SSO is enabled on your account, you will be able to require your team members to log in with Google.
Search Clips allow you to query any of your clusters and view readable, real-time responses. The Query Builder helps you navigate the Elasticsearch DSL with auto-completion and syntax error highlighting, and the results are exportable as JSON or CSV.
With Search Clips, you can build and share queries with your team members. You can invite some collaboration on the query structure while looking at the same results. Utilizing Search Clips in this way could create loops of building queries, learning from the output, refining, and ultimately lead to a well designed query that then gets integrated into an application.
To manage team members on an account, navigate to Managing Team Members.
In this document we will cover the following actions:
Click on the icon to the left of the account drop down on the upper right hand corner of the screen, and click on Search Clips from that menu. You will be able to view and edit Clips related to clusters in any of your accounts created by you or your team members.
Accessing Search Clips for the first time? Click Get started with Search Clips
Click the New Clip button in the upper right hand corner of the Search Clips page.
This will open a modal to enter information for your new Clip. Enter a name and select the cluster and indices you wish to query. All indices can be queried at once by selecting '_all', or select individual indices. Clicking on Create Clip will create and redirect you to the Search Clip.
The main sections of the Search Clip view are the Query Editor on the left, and the Response View on the right. Updating the query in the editor will automatically refresh results, or you can click the refresh button on the upper right hand corner of the Response View. The name of the Clip and the name of the cluster that is currently being queried are also shown on this page.
Update the cluster, indices, and name of the Clip by clicking on Settings in the upper right hand corner.
The Response View has tabs for Hits, Aggregations (Aggs), Metadata, and the Raw response. The Hits table shows the returned hits in a readable format, and allows you to toggle which columns you would like to see. If your query has any aggregations, they will be summarized in the Aggs tab.
Click on the Export button to export your Clip results to JSON or CSV.
Clips can be deleted from the Search Clips page by clicking on the toggle to the right of the Clip you would like to delete, and selecting the Delete option.
Bonsai occasionally sends out updates about new features, and can also send out weekly reports about your cluster's usage and performance. You can manage whether or not you receive these emails in the Notification Preferences tab.
You can edit your name and company size in the account profile.
You can create a new account from the upper right-hand dropdown by selecting Add Account within Switch Account's drop-down menu
Give your new account a name and click Create Account.
You will then be prompted to create your free sandbox cluster in the new account. You can switch between your accounts from the drop-down menu:
You can invite multiple team members to your Account, each with specialized roles. Account admins can invite new users to join the Account. At least one Billing and one Admin role are needed for each account. If you only have one team member, that person assumes the Admin and Billing role by default.
Click here to go directly to Team Members in the dashboard
You can edit your name and contact information in the Profile tab.
Need to change your email, password, or change the notifications you receive about clusters? Use the sections below to manage your personal profile.
To navigate to your personal profile, click on your initials in the upper right corner and select Profile Settings from the dropdown menu.
Navigate to Cancel Account to begin the cancellation process.
If you want to cancel your account, you’ll need to first make sure that any active clusters you have are deprovisioned first. If you have any active clusters on your account, you’ll see a notice like this:
For instructions on how to deprovision your clusters, please visit Cluster Dashboard.
Once you have deprovisioned all of the clusters on your account, you will be able to cancel your account completely. You will see a screen requesting your account password to confirm the cancellation.
So long, farewell, auf Wiedersehen, good night!
We’re sorry that you no longer want to have an account with us, but wish you the very best with your application and search. If there’s anything you feel we could do better, please don’t hesitate to send us an email with comments.
Migrating from a Heroku Account to a Direct Account requires minor configuration changes in the Heroku application. This will require redeployment of the application.
That’s it! Migrations are zero-downtime and take only a few minutes once our support team takes the ticket.
Hugo is a static site generator written in Go. It is conceptually similar to Jekyll, albeit with far more speed and flexibility. Hugo also supports generating output formats other than HTML, which allows users to pipe content directly into an Elasticsearch cluster.
In this guide, we are going to use this feature to tell Hugo to generate the exact format needed to submit the file to the _bulk endpoint of Elasticsearch.
In order to make use of this documentation, you will need Hugo installed and configured on your system
Use the URL for your cluster. A Bonsai URL looks something like this:
<div class="code-snippet w-richtext"><pre><code fs-codehighlight-element="code" class="hljs language-javascript">curl -XPUT https://user123:pass456@my-awesome-cluster-1234.us-east-1.bonsai.io/hugo</code></pre></div>
Hugo’s configuration settings live in a file called <span class="inline-code"><pre><code>config.toml</code></pre></span> by default. This file may also have a <span class="inline-code"><pre><code>.json</code></pre></span>. or <span class="inline-code"><pre><code>.yaml</code></pre></span>/<span class="inline-code"><pre><code>yml</code></pre></span>yml extension. Add the following snippet based on your config file format:
TOML:
<div class="code-snippet-container"><a fs-copyclip-element="click-1" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-1" class="hljs language-javascript">[outputs]
home = ["HTML", "RSS", "Bonsai"]
[outputFormats.Bonsai]
baseName = "bonsai"
isPlainText = true
mediaType = "application/json"
notAlternative = true
[params.bonsai]
vars = ["title", "summary", "date", "publishdate", "expirydate", "permalink"]
params = ["categories", "tags"]</code></pre></div></div>
JSON:
<div class="code-snippet-container"><a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">
{
"outputs": {
"home": [
"HTML",
"RSS",
"Bonsai"
]
},
"outputFormats": {
"Bonsai": {
"baseName": "bonsai",
"isPlainText": true,
"mediaType": "application/json",
"notAlternative": true
}
},
"params": {
"bonsai": {
"vars": [
"title",
"summary",
"date",
"publishdate",
"expirydate",
"permalink"
],
"params": [
"categories",
"tags"
]
}
}
}</code></pre></div></div>
YAML:
<div class="code-snippet-container"><a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript"> outputs:
home:
- HTML
- RSS
- Bonsai
outputFormats:
Bonsai:
baseName: bonsai
isPlainText: true
mediaType: application/json
notAlternative: true
params:
bonsai:
vars:
- title
- summary
- date
- publishdate
- expirydate
- permalink
params:
- categories
- tags</code></pre></div></div>
This snippet defines a new output called “Bonsai”, and specifies some associated variables.
Hugo needs to have a template for rendering data in a way that Elasticsearch will understand. To do this, we will define a JSON template that conforms to the Elasticsearch Bulk API.
Create a template called <span class="inline-code"><pre><code>layouts/_default/list.bonsai.json</code></pre></span> and give it the following content:
<div class="code-snippet-container"><a fs-copyclip-element="click-7" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-7" class="hljs language-javascript">{{/* Generates a valid Elasticsearch _bulk index payload */}}
{{- $section := $.Site.GetPage "section" .Section }}
{{- range .Site.AllPages -}}
{{- if or (and (.IsDescendant $section) (and (not .Draft) (not .Params.private))) $section.IsHome -}}
{{/* action / metadata */}}
{{ (dict "index" (dict "_index" "hugo" "_type" "doc" "_id" .UniqueID)) | jsonify }}
{{ (dict "objectID" .UniqueID "date" .Date.UTC.Unix "description" .Description "dir" .Dir "expirydate" .ExpiryDate.UTC.Unix "fuzzywordcount" .FuzzyWordCount "keywords" .Keywords "kind" .Kind "lang" .Lang "lastmod" .Lastmod.UTC.Unix "permalink" .Permalink "publishdate" .PublishDate "readingtime" .ReadingTime "relpermalink" .RelPermalink "summary" .Summary "title" .Title "type" .Type "url" .URL "weight" .Weight "wordcount" .WordCount "section" .Section "tags" .Params.Tags "categories" .Params.Categories "authors" .Params.Authors) | jsonify }}
{{- end -}}
{{- end }}</code></pre></div></div>
When the site is generated, this will result in creating a file called public/bonsai.json, which will have the content stored in a way that can be pushed directly into Elasticsearch using the Bulk API.
To get the site’s data into Elasticsearch, render it by running <span class="inline-code"><pre><code>hugo</code></pre></span> on the command line. Then send it to your Bonsai cluster with <span class="inline-code"><pre><code>curl</code></pre></span>:
<div class="code-snippet-container"><a fs-copyclip-element="click-4" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-4" class="hljs language-javascript">curl -H "Content-Type: application/x-ndjson" -XPOST "https://user123:pass456@my-awesome-cluster-1234.us-east-1.bonsai.io/_bulk" --data-binary @public/bonsai.json</code></pre></div></div>
You should now be able to see your data in the Elasticsearch cluster:
<div class="code-snippet-container"><a fs-copyclip-element="click-5" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-5" class="hljs language-javascript">$ curl -XGET "https://user123:pass456@my-awesome-cluster-1234.us-east-1.bonsai.io/_search"{"took":1,"timed_out":false,"_shards":{"total":2,"successful":2,"failed":0},"hits":{"total":1,"max_score":1.0,"hits":[{"_index":"hugo","_type":"doc","_id":...</code></pre></div></div>
Bonsai supports the ElasticHQ monitoring and management interface. This open source software gives you insight into the state of your cluster. If you’re looking to see details about performance, check out Cluster Metrics.
The GitHub repo has tons of documentation and how-to guides. This article lays out some common methods of running ElasticHQ locally.
<div class="code-snippet-container"><a fs-copyclip-element="click-1" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-1" class="hljs language-javascript">mkdir -p ~/.pow/elastichq
git clone git@github.com:royrusso/elasticsearch-HQ.git ~/.pow/elastichq/public</code></pre></div></div>
Navigate your browser to http://elastichq.dev/. You should see the ElasticHQ dashboard. Enter your Bonsai cluster URL in the field for the cluster location and click on “Connect.” The dashboard will bring up information about your cluster.
<div class="code-snippet-container"><a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">git clone git@github.com:royrusso/elasticsearch-HQ.git
cd elasticsearch-HQ
git checkout master
npm install
grunt server</code></pre></div></div>
You should see the ElasticHQ dashboard at <span class="inline-code"><pre><code>http://localhost:9000/</code></pre></span>. Enter your bonsai.io cluster URL in the field for the cluster location and click on “Connect.” The dashboard will bring up information about your cluster.
First, make sure Apache is running. Typically this means you can access a directory via your browser at <span class="inline-code"><pre><code>http://localhost:80/</code></pre></span>. Your install may be different, so use whatever URL/folder is appropriate for you.
<div class="code-snippet-container"><a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript">cd # The root directory of your HTTP server
git@github.com:royrusso/elasticsearch-HQ.git
cd elasticsearch-HQ/
git checkout master</code></pre></div></div>
Open up your browser to <span class="inline-code"><pre><code>http://localhost:80/elasticsearch-HQ/</code></pre></span> and you should see the ElasticHQ dashboard. Enter your bonsai.io cluster URL in the field for the cluster location and click on “Connect.” The dashboard will bring up information about your cluster.
Logstash is a data processing tool for ingesting logs into Elasticsearch. It plays a prominent role in the Elastic suite, and a common question is whether Bonsai offers support for it.
The answer is a qualified “yes.” Logstash is a server-side tool, meaning it runs outside of Bonsai’s infrastructure and Bonsai is not involved in its configuration or management. But as a host, Bonsai is not opinionated about where your cluster’s data comes from. So if you have Logstash running on your servers, you can configure an output to your Bonsai cluster, and it will work.
Connecting your Logstash instance to a Bonsai cluster is as easy as adding an output to the Logstash configuration file like so:
<div class="code-snippet w-richtext"><pre><code fs-codehighlight-element="code" class="hljs language-javascript">output {
elasticsearch {
# https://randomuser:randompass@something-12345.us-east-1.bonsai.io
# would be entered as:
hosts => ["something-12345.us-east-1.bonsai.io:443"]
user => "randomuser"
password => "randompass"
ssl => true
index => ""
}
}</code></pre></div>
If an application sends data to an index which does not exist, Elasticsearch will create that index and assume its mappings from the data in the payload. This feature is called autocreation, and it is supported in a limited capacity on Bonsai. Certain base names can be used for autocreation. Those base names are:
This means your Logstash index must start with one of these index names, or it will not be automatically created.
We have made a number of tests and verified that we are fully compatible with Logstash, as of version 1.5+. Older versions of Logstash don’t have support for SSL/TLS or HTTP Basic Auth; these older versions can work with Bonsai, but only without the benefits of encryption or authentication.
If you have any issues getting Logstash to pass data along to Bonsai, check the documentation to make sure it’s set up correctly. If that doesn’t help, feel free to reach out to us at support@bonsai.io and we’ll do our best to get you pointed in the right direction.
Kibana is an open-source data visualization and dashboard tool built for rich analytics. It takes advantage of Elasticsearch’s full-text querying and aggregation capabilities to built highly flexible and interactive dashboards.
All Bonsai clusters support Kibana out of the box. It is possible to use Kibana through one of several ways: via your Bonsai cluster dashboard, as a free Heroku app, or locally/as a private server.
Bonsai provides Kibana instances to clusters running on Elasticsearch versions 5.x and up. You can launch your Kibana instance right from your dashboard:
Clicking on the Kibana link will open up your Kibana instance:
Please be patient, as it may take a few seconds for Kibana to load.
If you have a Heroku account, there is a GitHub project that offers a click to deploy button. Clicking on the button will walk you through the process of deploying a free Heroku app running Kibana, which can be configured with a URL to an Elasticsearch cluster.
If you don’t have a cluster yet, a free Bonsai cluster will be created and attached to the Kibana app. If you already have a Bonsai cluster, you can link to it during the build process.
You may also download Kibana and run it locally or on a private server. Not all versions of Kibana are compatible with all versions of Elasticsearch, so make sure to check the compatibility matrix and download a version that will work with your Bonsai cluster.
( Note: You can also install Kibana using a repository and package manager, but this will likely involve downloading the latest version and may not be compatible with your cluster)
Once you have Kibana downloaded, you’ll need to configure it to point at your Bonsai cluster. Open up the <span class="inline-code"><pre><code>config/kibana.yml</code></pre></span> file and to set the value for <span class="inline-code"><pre><code>elasticsearch_url</code></pre></span>. For example:
<div class="code-snippet w-richtext"><pre><code fs-codehighlight-element="code" class="hljs language-javascript">elasticsearch_url: "https://:@something-12345.us-east-1.bonsai.io"</code></pre></div>
In some later versions of Kibana, you may need to separately specify your Bonsai cluster’s username/password as configuration options:
<div class="code-snippet w-richtext"><pre><code fs-codehighlight-element="code" class="hljs language-javascript">elasticsearch_url: "https://:@something-12345.us-east-1.bonsai.io:443"
elasticsearch.username: ""
elasticsearch.password: ""</code></pre></div>
Once Kibana has been configured, you can run it with bin/kibana (or bin\kibana.bat on Windows). This will start up the Kibana server with the settings pointing to your Bonsai cluster.
Last, open up a browser to http://localhost:5601 to finish setting up Kibana and get started. Note that if you’re running Kibana on a remote server, you’ll need to replace localhost with the IP address or domain of the remote server.
You must have an Enterprise account to enable Okta Single Sign-On. Once enabled for your account, all users must login via Okta. This can be done from the login form with any password, or from the Okta End-User Dashboard.
To enable Okta you must:
Filebeat is a lightweight shipper for forwarding and centralizing log data. It monitors the log files or locations that you specify, collects log events, and forwards them to Elasticsearch for indexing. A common question is whether Bonsai offers support for it.
The answer is a qualified “yes.” Filebeat is a server-side tool, meaning it runs outside of Bonsai’s infrastructure and Bonsai is not involved in its configuration or management. But as a host, Bonsai is not opinionated about where your cluster’s data comes from. So if you have Filebeat running on your servers, you can configure an output to your Bonsai cluster, and it will work.
To connect Filebeat to a Bonsai cluster you just need to add your Bonsai URL to the filebeat.yml file like this:
<div class="code-snippet-container"><a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">output.elasticsearch:
hosts: ["wp-play-8646224217.us-east-1.bonsaisearch.net:443"]
protocol: "https"
username: "aaa" # The randomly-generated username for your cluster
password: "xxx" # The randomly-generated password for your cluster</code></pre></div></div>
Autocreation and Bonsai
If an application sends data to an index which does not exist, Elasticsearch will create that index and assume its mappings from the data in the payload. This feature is called autocreation, and it is supported in a limited capacity on Bonsai. Certain base names can be used for autocreation. Those base names are:
This means your Filebeat index must start with one of these index names, or it will not be automatically created.
It is important to note that Filebeat requires the OSS distribution of Elasticsearch, so for this process to work the OSS version of Filebeat needs to be used.
WordNet is a huge lexical database that collects and orders English words into groups of synonyms. It can offer major improvements in relevancy, but it is not at all necessary for many use cases. Make sure you understand the tradeoffs (discussed below) well before setting it up.
There are two ways to use WordNet with Bonsai. Users can add a subset of the list using the Elasticsearch API, or use the WordNet file that comes standard with all Bonsai clusters.
First, a brief background on synonyms and WordNet. If you want to jump around, the main sections of this document are:
Let’s say that you have an online store with a lot of products. You want users to be able to search for those products, but you want that search to be smart. For example, say that your user searches for “bowtie pasta.” You may have a product called “Funky Farfalle” which is related to their search term but which would not be returned in the results because the title has “farfalle” instead of “bowtie pasta”. How do you address this issue?
Elasticsearch has a mechanism for defining custom synonyms, through the Synonym Token Filter. This lets search administrators define groups of related terms and even corrections to commonly misspelled terms. A solution to this use case might look like this:
<div class="code-snippet w-richtext"><pre><code fs-codehighlight-element="code" class="hljs language-javascript">{
"settings": {
"index" : {
"analysis" : {
"filter" : {
"synonym" : {
"type" : "synonym",
"synonyms" : [
"bowtie pasta, farfalle"
]
}
}
}
}
}
}</code></pre></div>
This is great for solving the proximate issue, but what it can get extremely tedious to define all groups of related words in your index.
WordNet is essentially a text database which places English words into synsets - groups of synonyms - and can be considered as something of a cross between a dictionary and a thesaurus. An entry in WordNet looks something like this:
<div class="code-snippet w-richtext"><pre><code fs-codehighlight-element="code" class="hljs language-javascript"></code></pre></div>
Let’s break it down:
This line expresses that the word ‘kitty’ is a noun, and the first word in synset 102122298 (which includes other terms like “kitty-cat,” “pussycat,” and so on). The line also indicates ‘kitty’ is the fourth most commonly used term according to semantic concordance texts. You can read more about the structure and precise definitions of WordNet entries in the documentation.
The WordNet has become extremely useful in text processing applications, including data storage and retrieval. Some use cases require features like synonym processing, for which a lexical grouping of tokens is invaluable.
Relevancy tuning can be a deeply complex subject, and WordNet – especially when the complete file is used – has tradeoffs, just like any other strategy. Synonym expansion can be really tricky and can result in unexpected sorting, lower performance and more disk use. WordNet can introduce all of these issues with varying severity.
When synonyms are expanded at index time, Elasticsearch uses WordNet to generate all tokens related to a given token, and writes everything out to disk. This has several consequences: slower indexing speed, higher load during indexing, and significantly more disk use. Larger index sizes often correspond to memory issues as well.
There is also the problem of updating. If you ever want to change your synonym list, you’ll need to reindex everything from scratch. And WordNet includes multi-term synonyms in its database, which can break phrase queries.
Expanding synonyms at query time resolves some of those issues, but introduces others. Namely, performing expansion and matching at query time adds overhead to your queries in terms of server load and latency. And it still doesn’t really address the problem of multi word synonyms.
The Elasticsearch documentation some really great examples of what this means. The takeaway is that WordNet is not a panacea for relevancy tuning, and it may introduce unexpected results unless you’re doing a lot of preprocessing or additional configuration.
tl;dr: Do not simply assume that chucking a massive synset collection at your cluster will make it faster with more relevant results.
Elasticsearch supports several different list formats, including the WordNet format. WordNet synonyms are maintained in a Prolog file called <span class="inline-code"><pre><code>wn_s.pl</code></pre></span>. To use these in your cluster, you’ll need to download the WordNet archive and extract the <span class="inline-code"><pre><code>wn_s.pl</code></pre></span> file. You’ll then need to create your synonyms list by reading this file into a request to your cluster.
The target index could be created with settings like so:
<div class="code-snippet w-richtext"><pre><code fs-codehighlight-element="code" class="hljs language-javascript">PUT https://randomuser:randompass@something-12345.us-east-1.bonsai.io/some_index
{
"settings": {
"analysis": {
"filter": {
"wn_synonym_filter": {
"type": "synonym",
"format" : "wordnet",
"synonyms" : [
"s(100000001,1,"abstain",v,1,0).",
"s(100000001,2,"refrain",v,1,0).",
"s(100000001,3,"desist",v,1,0).",
#... more synonyms, read from wn_s.pl file
]
}
},
"analyzer": {
"my_synonyms": {
"tokenizer": "standard",
"filter": [
"lowercase",
"wn_synonym_filter"
]
}
}
}
}
}</code></pre></div>
There are a number of ways to generate this request. You could do it programmatically with a language like Python, or Bash scripts with <span class="inline-code"><pre><code>curl</code></pre></span>, or any language with which you feel comfortable.
A benefit of using a subset of the list would be more control over your mappings and data footprint. Depending on when your analyzer is running, you could save IO by not computing unnecessary expansions for terms not in your corpus or search parameters. Reducing the overhead will improve performance overall.
If you would rather use the official WordNet list, it is part of our Elasticsearch deployment. You can follow the official Elasticsearch documentation for WordNet synonyms, and link to the file with <span class="inline-code"><pre><code>analysis/wn_s.pl</code></pre></span>. For example:
<div class="code-snippet w-richtext"><pre><code fs-codehighlight-element="code" class="hljs language-javascript">PUT https://username:password@my-awesome-cluster.us-east-1.bonsai.io/some_index
{
"settings": {
"index" : {
"analysis" : {
"analyzer" : {
"synonym" : {
"tokenizer" : "whitespace",
"format" : "wordnet",
"filter" : ["synonym"]
}
},
"filter" : {
"synonym" : {
"type" : "synonym",
"format" : "wordnet",
"synonyms_path" : "analysis/wn_s.pl"
}
}
}
}
}
}</code></pre></div>
WordNet is a large subject and a great topic to delve deeper into. Here are some links for further reading:
Setting up your Java app to use Bonsai Elasticsearch is quick and easy. Just follow these simple steps:
You’ll need to add the <span class="inline-code"><pre><code>elasticsearch-rest-high-level-client</code></pre></span> library to your pom.xml file like so:
org.elasticsearch.client
elasticsearch-rest-client
6.2.3
Bonsai requires basic authentication for all read/write requests. You’ll need to configure the client so that it includes the username and password when communicating with the cluster. The following code is a good starter for integrating Bonsai Elasticsearch into your app:
<div class="code-snippet-container"><a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-java">package io.omc;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.DefaultConnectionKeepAliveStrategy;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import java.net.URI;
public class Main {
public static void main(String[] args) {
String connString = System.getenv("BONSAI_URL");
URI connUri = URI.create(connString);
String[] auth = connUri.getUserInfo().split(":");
CredentialsProvider cp = new BasicCredentialsProvider();
cp.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(auth[0], auth[1]));
RestHighLevelClient rhlc = new RestHighLevelClient(
RestClient.builder(new HttpHost(connUri.getHost(), connUri.getPort(), connUri.getScheme()))
.setHttpClientConfigCallback(
httpAsyncClientBuilder -> httpAsyncClientBuilder.setDefaultCredentialsProvider(cp)
.setKeepAliveStrategy(new DefaultConnectionKeepAliveStrategy())));
// https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/java-rest-high-search.html
SearchRequest searchRequest = new SearchRequest();
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(QueryBuilders.matchAllQuery());
searchRequest.source(searchSourceBuilder);
try {
SearchResponse resp = rhlc.search(searchRequest);
// Show that the query worked
System.out.println(resp.toString());
} catch (Exception ex) {
// Log the exception
System.out.println(ex.toString());
}
// Need to close the client so the thread will exit
try {
rhlc.close();
} catch (Exception ex) {
}
}
}</code></pre></div></div>
Feel free to adapt that code to your particular app. Keep in mind that the core elements here can be moved around, but really shouldn’t be changed or further simplified.
For example, the snippet above parses out the authentication credentials from the <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span> environment variable. This is so the username and password don’t need to be hard coded into your app, where they could pose a security risk.
Bonsai Elasticsearch integrates with your .Net app quickly and easily, whether you’re running on Heroku or self hosting.
First, make sure to add the Elasticsearch.Net and NEST client to your dependencies list:
<div class="code-snippet-container"><a fs-copyclip-element="click-1" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-1" class="hljs language-javascript">...
....</code></pre></div></div>
You’ll also want to make sure the client is installed locally for testing purposes:
<div class="code-snippet-container"><a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">$ dotnet restore</code></pre></div></div>
Self-hosted users: when you sign up with Bonsai and create a cluster, it will be provisioned with a custom URL that looks like https://user:pass@my-awesome-cluster.us-east-1.bonsai.io. Make note of this URL because it will be needed momentarily.
Heroku users: you’ll want to make sure that you’ve added Bonsai Elasticsearch to your app. Visit our addons page to see available plans. You can add Bonsai to your app through the dashboard, or by running:
<div class="code-snippet-container"><a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript">$ heroku addons:create bonsai: -a</code></pre></div></div>
Update your Main.cs file with the following:
<div class="code-snippet-container"><a fs-copyclip-element="click-4" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-4" class="hljs language-javascript">using System;
using Nest;
namespace search.con
{
class Program
{
static void Main(string[] args)
{
var server = new Uri(Environment.GetEnvironmentVariable("BONSAI_URL"));
var conn = new ConnectionSettings(server);
// use gzip to reduce network bandwidth
conn.EnableHttpCompression();
// lets the HttpConnection control this for optimal access (default: 80)
conn.ConnectionLimit(-1);
var client = new ElasticClient(conn);
var resp = client.Ping();
if (resp.IsValid)
{
Console.WriteLine("All is well");
}
else
{
Console.WriteLine("Elasticsearch cluster is down");
}
}
}
}
</code></pre></div></div>
The code above does several things:
Self-hosted users: You will need to take an additional step and add your Bonsai URL to your server/dev environment variables. Something like this should work:
<div class="code-snippet-container"><a fs-copyclip-element="click-5" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-5" class="hljs language-javascript">$ export BONSAI_URL="https://username:password@my-awesome-cluster-123.us-east-1.bonsai.io"</code></pre></div></div>
Heroku users: You don’t need to worry about this step. The environment variable is automatically populated for you when you add Bonsai Elasticsearch to your app. You can verify that it exists by running:
<div class="code-snippet-container"><a fs-copyclip-element="click-6" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-6" class="hljs language-javascript">$ heroku config:get BONSAI_URL</code></pre></div></div>
Go ahead and spin up your .Net app with <span class="inline-code"><pre><code>dotnet run</code></pre></span>. Heroku users can also push the changes to Heroku, and monitor the logs with <span class="inline-code"><pre><code>heroku logs --tail</code></pre></span>. Either way, you should see something like this:
Elasticsearch INFO: 2016-02-03T23:44:41Z
Adding connection to https://nodejs-test-012345.us-east-1.bonsai.io/
Elasticsearch DEBUG: 2016-02-03T23:44:41Z
Request complete
Elasticsearch TRACE: 2016-02-03T23:44:41Z
-> HEAD https://nodejs-test-012345.us-east-1.bonsai.io:443/?hello=elasticsearch
<- 200
The above output indicates that the client was successfully initiated and was able to contact the cluster.
Questions? Problems? Feel free to ping our support if something isn’t working right and we’ll do what we can to help out.
Bonsai Elasticsearch integrates with your node.js app quickly and easily, whether you’re running on Heroku or self hosting.
First, make sure to add the Elasticsearch.js client to your dependencies list:
package.json:
<div class="code-snippet-container"><a fs-copyclip-element="click-1" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-1" class="hljs language-javascript">"dependencies": {
"elasticsearch": "10.1.2"
....</code></pre></div></div>
You’ll also want to make sure the client is installed locally for testing purposes:
<div class="code-snippet-container"><a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">$ npm install --save elasticsearch
$ npm install</code></pre></div></div>
Self-hosted users: when you sign up with Bonsai and create a cluster, it will be provisioned with a custom URL that looks like https://user:pass@my-awesome-cluster.us-east-1.bonsai.io. Make note of this URL because it will be needed momentarily.
Heroku users: you’ll want to make sure that you’ve added Bonsai Elasticsearch to your app. Visit our addons page to see available plans. You can add Bonsai to your app through the dashboard, or by running:
<div class="code-snippet-container"><a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript">$ heroku addons:create bonsai: -a</code></pre></div></div>
Update your index.js file with the following:
index.js:
<div class="code-snippet-container"><a fs-copyclip-element="click-4" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-4" class="hljs language-javascript">var bonsai_url = process.env.BONSAI_URL;
var elasticsearch = require('elasticsearch');
var client = new elasticsearch.Client({
host: bonsai_url,
log: 'trace'
});
// Test the connection:
// Send a HEAD request to "/" and allow
// up to 30 seconds for it to complete.
client.ping({
requestTimeout: 30000,
}, function (error) {
if (error) {
console.error('elasticsearch cluster is down!');
} else {
console.log('All is well');
}
});</code></pre></div></div>
The code above does several things:
Self-hosted users: You will need to take an additional step and add your Bonsai URL to your server/dev environment. Something like this should work:
<div class="code-snippet-container"><a fs-copyclip-element="click-5" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-5" class="hljs language-javascript">$ export BONSAI_URL="https://username:password@my-awesome-cluster-123.us-east-1.bonsai.io"</code></pre></div></div>
Heroku users: You don’t need to worry about this step. The environment variable is automatically populated for you when you add Bonsai Elasticsearch to your app. You can verify that it exists by running:
<div class="code-snippet-container"><a fs-copyclip-element="click-6" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-6" class="hljs language-javascript">$ heroku config:get BONSAI_URL</code></pre></div></div>
Go ahead and spin up your node app with <span class="inline-code"><pre><code>npm start</code></pre></span>. Heroku users can also push the changes to Heroku, and monitor the logs with <span class="inline-code"><pre><code>heroku logs --tail</code></pre></span>. Either way, you should see something like this:
<div class="code-snippet w-richtext"><pre><code fs-codehighlight-element="code" class="hljs language-javascript">Elasticsearch INFO: 2016-02-03T23:44:41Z
Adding connection to https://nodejs-test-012345.us-east-1.bonsai.io/
Elasticsearch DEBUG: 2016-02-03T23:44:41Z
Request complete
Elasticsearch TRACE: 2016-02-03T23:44:41Z
-> HEAD https://nodejs-test-012345.us-east-1.bonsai.io:443/?hello=elasticsearch
<- 200</code></pre></div>
The above output indicates that the client was successfully initiated and was able to contact the cluster.
Questions? Problems? Feel free to ping our support if something isn’t working right and we’ll do what we can to help out.
Getting Elasticsearch up and running with a PHP app is fairly straightforward. We recommend using the official PHP client, as it is being actively developed alongside Elasticsearch.
Like Heroku, we recommend Composer for dependency management in PHP projects, so you’ll need to add the Elasticsearch library to your <span class="inline-code"><pre><code>composer.json file</code></pre></span>:
<div class="code-snippet-container"><a fs-copyclip-element="click-1" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-1" class="hljs language-javascript">{
"require": {
"elasticsearch/elasticsearch": "1.0"
}
}</code></pre></div></div>
Heroku will automatically add the library when you deploy your code.
The elasticsearch-php library is configured almost entirely by associative arrays. To initialize your client, use the following code block:
<div class="code-snippet-container"><a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-php">// Initialize the parameters for the cluster
$params = array();
$params['hosts'] = array (
getenv("BONSAI_URL"),
);
$client = new Elasticsearch\Client($params);</code></pre></div></div>
Note that the host is pulled from the environment. When you add Bonsai to your app, the cluster will be automatically created and a <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span> variable will be added to your environment. The initialization process above allows you to access your cluster without needing to hardcode your authentication credentials.
Bonsai does not support lazy index creation, so you will need to create your index before you can send over your documents. You can specify a number of parameters to configure your new index; the code below is the minimum needed to get started:
<div class="code-snippet-container"><a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-php">$indexParams = array();
$indexParams['index'] = 'my_index'; //index
$client->indices()->create($indexParams);</code></pre></div></div>
Once your index has been created, you can add a document by specifying a body (an associative array of fields and data), target index, type and (optionally) a document ID. If you don’t specify an ID, Elasticsearch will create one for you:
<div class="code-snippet-container"><a fs-copyclip-element="click-4" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-4" class="hljs language-php">$params['body'] = array('testField' => 'abc');
$params['index'] = 'my_index';
$params['type'] = 'my_type';
$params['id'] = 'my_id';
// Document will be indexed to my_index/my_type/my_id
$ret = $client->index($params);</code></pre></div></div>
The official Elasticsearch Python client is not supported on Bonsai after version 7.13. This is due to a change introduced in the 7.14 release of the client. This change prevents the Python client from communicating with open-sourced versions of Elasticsearch 7.x, as well as any version of OpenSearch.
If you are receiving an error stating "The client noticed that the server is not a supported distribution of Elasticsearch," then you'll need to downgrade the client to 7.13 or lower.
Setting up your Python app to use Bonsai Elasticsearch is quick and easy. Just follow the steps below:
You’ll need to add the elasticsearch library to your <span class="inline-code"><pre><code>Pipfile</code></pre></span> file like so:
<div class="code-snippet-container"><a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-python">pipenv install elasticsearch elasticsearch-dsl</code></pre></div></div>
Bonsai requires basic authentication for all read/write requests. You’ll need to configure the client so that it includes the username and password when communicating with the cluster. The following code is a good starter for integrating Bonsai Elasticsearch into your app:
<div class="code-snippet-container"><a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript">import os, base64, re, logging
from elasticsearch import Elasticsearch
# Log transport details (optional):
logging.basicConfig(level=logging.INFO)
# Parse the auth and host from env:
bonsai = os.environ['BONSAI_URL']
auth = re.search('https\:\/\/(.*)\@', bonsai).group(1).split(':')
host = bonsai.replace('https://%s:%s@' % (auth[0], auth[1]), '')
# Optional port
match = re.search('(:\d+)', host)
if match:
p = match.group(0)
host = host.replace(p, '')
port = int(p.split(':')[1])
else:
port=443
# Connect to cluster over SSL using auth for best security:
es_header = [{
'host': host,
'port': port,
'use_ssl': True,
'http_auth': (auth[0],auth[1])
}]
# Instantiate the new Elasticsearch connection:
es = Elasticsearch(es_header)
# Verify that Python can talk to Bonsai (optional):
es.ping()
</code></pre></div></div>
Feel free to adapt that code to your particular app. Keep in mind that the core elements here can be moved around, but really shouldn’t be changed or further simplified.
For example, the snippet above parses out the authentication credentials from the <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span> environment variable. This is so the username and password don’t need to be hard coded into your app, where they could pose a security risk. Also, the <span class="inline-code"><pre><code>host</code></pre></span>, <span class="inline-code"><pre><code>port</code></pre></span> and <span class="inline-code"><pre><code>use_ssl</code></pre></span> parameters are important for SSL encryption to work properly. Simply using your <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span> as the host will not work because of limitations in urllib. It needs to be a plain URL, with the credentials passed to the client as a tuple.
Users of Django/Haystack can easily integrate with Bonsai Elasticsearch! We recommend using the official Python client, as it is being actively developed alongside Elasticsearch.
Haystack does not yet support Elasticsearch 6.x or 7.x, which is the current default on Bonsai. Heroku users can provision a 5.x cluster by adding Bonsai via the command line and passing in a <span class="inline-code"><pre><code>--version</code></pre></span> flag like so:
<div class="code-snippet-container">
<a fs-copyclip-element="click-1" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-1" class="hljs language-javascript">heroku addons:create bonsai --version=5</code></pre>
</div>
</div>
Note that some versions require a paid plan. Free clusters are always provisioned on the latest version of Elasticsearch, regardless of which version was requested. See more details here.
Let’s get started:
You’ll need to add the elasticsearch-py and django-haystack libraries to your requirements.txt file:
<div class="code-snippet-container">
<a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">elasticsearch>=1.0.0,<2.0.0
django-haystack>=1.0.0,<2.0.0</code></pre>
</div>
</div>
Bonsai requires basic authentication for all read/write requests. You’ll need to configure the client so that it includes the username and password when communicating with the cluster. We recommend adding the cluster URL to an environment variable, <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span>, to avoid hard-coding your authentication credentials.
The following code is a good starter for integrating Bonsai Elasticsearch into your app:
<div class="code-snippet-container">
<a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript">ES_URL = urlparse(os.environ.get('BONSAI_URL') or 'http://127.0.0.1:9200/')
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': ES_URL.scheme + '://' + ES_URL.hostname + ':443',
'INDEX_NAME': 'haystack',
},
}
if ES_URL.username:
HAYSTACK_CONNECTIONS['default']['KWARGS'] = {"http_auth": ES_URL.username + ':' + ES_URL.password}</code></pre>
</div>
</div>
The sample code above uses port 443, which is the default for the https:// protocol. If you’re not using SSL/TLS and want to use http:// instead, change this value to 80.
One of the most common issues users see relates to SSL certs. Bonsai URLs are using TLS to secure communications with the cluster, and our certificates are signed by an authority (CA) that has verified our identity. Python needs access to the proper root certificate in order to verify that the app is actually communicating with Bonsai; if it can’t find the certificate it needs, you’ll start seeing exception messages like this:
<div class="code-snippet w-richtext">
<pre><code fs-codehighlight-element="code" class="hljs language-javascript">Root certificates are missing for certificate</code></pre>
</div>
The fix is straightforward enough. Simply install the certifi package in the environment where your app is hosted. You can do this locally by running <span class="inline-code"><pre><code>pip install certifi</code></pre></span>. Heroku users will also need to modify their <span class="inline-code"><pre><code>requirements.txt</code></pre></span>, per the documentation:
<div class="code-snippet-container">
<a fs-copyclip-element="click-4" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-4" class="hljs language-javascript">certifi==0.0.8</code></pre>
</div>
</div>
If certifi and root cert management isn’t possible, you can simply bypass verification by modifying your <span class="inline-code"><pre><code>HAYSTACK_CONNECTIONS</code></pre></span> like so:
<div class="code-snippet-container">
<a fs-copyclip-element="click-5" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-5" class="hljs language-javascript">HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': ES_URL.scheme + '://' + ES_URL.hostname + ':443',
'INDEX_NAME': 'documents',
'KWARGS': {
'use_ssl': True,
'verify_certs': False,
}
},
}</code></pre>
</div>
</div>
This instructs Haystack to use TLS without verifying the host. This does allow for the possibility of MITM attacks, but the probably of that happening is pretty low. You’ll need to weigh the expediency of this approach against the unlikely event of someone eavesdropping, and decide whether there is an acceptable security risk of leaking data in that way.
Haystack does not yet support Elasticsearch 6.x and up. Please ensure your Bonsai cluster version is 5.x or less.
Users of Django/django-elasticsearch-dsl can easily integrate with Bonsai Elasticsearch! This library is built on top of elasticsearch-dsl which is built on top of the low level elasticsearch-py maintained by Elastic.
Let’s get started:
You’ll need to add the django-elasticsearch-dsl library to your <span class="inline-code"><pre><code>requirements.txt</code></pre></span> file:
<div class="code-snippet-container">
<a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">django-elasticsearch-dsl>=0.5.1
elasticsearch-dsl>=6.0,<6.2</code></pre>
</div>
</div>
Full Instructions can be found here
Bonsai requires basic authentication for all read/write requests. You’ll need to configure the client so that it includes the username and password when communicating with the cluster. We recommend adding the cluster URL to an environment variable, <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span>, to avoid hard-coding your authentication credentials.
The following code is a good starter for integrating Bonsai Elasticsearch into your app:
<div class="code-snippet-container">
<a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript">ES_URL = urlparse(os.environ.get('BONSAI_URL') or 'http://127.0.0.1:9200/')
ELASTICSEARCH_DSL={
'default': {
'hosts': ES_URL
},
}</code></pre>
</div>
</div>
The sample code above uses port 443, which is the default for the https:// protocol. If you’re not using SSL/TLS and want to use http:// instead, change this value to 80.
Getting started with Ruby on Rails and Bonsai Elasticsearch is fast and easy. In this guide, we will cover the steps and the bare minimum amount of code needed to support basic search with Elasticsearch. Users looking for more details and advanced usage should consult the resources at the end of this page.
Throughout this guide, you will see some code examples. These code examples are drawn from a very simple Ruby on Rails application, and are designed to offer some real-world, working code that new users will find useful. The complete demo app can be found in this GitHub repo.
The official Elasticsearch Ruby client is not supported on Bonsai after version 7.13. This is due to a change introduced in the 7.14 release of the gem. This change prevents the Ruby client from communicating with open-sourced versions of Elasticsearch 7.x, as well as any version of OpenSearch. The table below indicates compatibility:
<table>
<thead>
<tr><th>Engine</th><th>Version Highest Compatible Gem Version</th></tr>
</thead>
<tbody>
<tr><td>Elasticsearch 5.x</td><td>7.13</td></tr>
<tr><td>Elasticsearch 6.x</td><td>7.14 (sic)</td></tr>
<tr><td>Elasticsearch 7.x</td><td>7.13</td></tr>
<tr><td>OpenSearch 1.x</td><td>7.13</td></tr>
</tbody>
</table>
If you are receiving a <span class="inline-code"><pre><code>Elasticsearch::UnsupportedProductError</code></pre></span>, then you'll need to ensure you're using a supported version of the Elasticsearch Ruby client.
In this example, we are going to connect to Elasticsearch using the official Elasticsearch gems. There are other gems for this, namely SearchKick, which is covered in another set of documentation.
Make sure that there is a Bonsai Elasticsearch cluster ready for your app to interact with. This needs to be set up first so you know which version of the gems you need to install; Bonsai supports a large number of Elasticsearch versions, and the gems need to correspond to the version of Elasticsearch you’re running.
Bonsai clusters can be created in a few different ways, and the documentation for each path varies. If you need help creating your cluster, check out the link that pertains to your situation:
When you have successfully created your cluster, it will be given a semi-random URL called the Elasticsearch Access URL. You can find this in the Cluster Dashboard, in the Credentials tab:
Heroku users will also have a <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span> environment variable created when Bonsai is added to the application. This variable will contain the fully-qualified URL to the cluster.
When you have a Bonsai Elasticsearch cluster, there are a few ways to check the version that it is running. These are outlined below:
The easiest is to simply get it from the Cluster Dashboard. When you view your cluster overview in Bonsai, you will see some details which include the version of Elasticsearch the cluster is running:
You can also use the Interactive Console. In the Cluster Dashboard, click on the Console tab. It will load a default view, which includes the version of Elasticsearch. The version of Elasticsearch is called “number” in the JSON response:
You can copy/paste your cluster URL into a browser or into a tool like <span class="inline-code"><pre><code>curl</code></pre></span>. Either way, you will get a response like so:
<div class="code-snippet-container">
<a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">curl https://abcd123:efg456@my-cluster-123456.us-west-2.bonsaisearch.net:443
{
"name" : "ip-172-31-14-16",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "jVJrINr5R5GVVXHGcRhMdA",
"version" : {
"number" : "7.2.0",
"build_flavor" : "oss",
"build_type" : "tar",
"build_hash" : "508c38a",
"build_date" : "2019-06-20T15:54:18.811730Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}</code></pre>
</div>
</div>
The version of Elasticsearch is called “number” in the JSON response.
There are a few gems you will need in order to make all of this work. Add the following to your Gemfile outside of any blocks:
<div class="code-snippet-container">
<a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript">gem 'elasticsearch-model', github: 'elastic/elasticsearch-rails', branch: 'master'
gem 'elasticsearch-rails', github: 'elastic/elasticsearch-rails', branch: 'master'
gem 'bonsai-elasticsearch-rails', github: 'omc/bonsai-elasticsearch-rails', branch: 'master'</code></pre>
</div>
</div>
This will install the gems for the latest major version of Elasticsearch. If you have an older version of Elasticsearch, then you should follow this table:
<table>
<thead>
<tr><th>Branch</th><th>Elasticsearch Version</th></tr>
</thead>
<tbody>
<tr><td>0.1</td><td>-> 1.x</td></tr>
<tr><td>2.x</td><td>-> 2.x</td></tr>
<tr><td>5.x</td><td>-> 5.x</td></tr>
<tr><td>6.x</td><td>-> 6.x</td></tr>
<tr><td>master</td><td>-> master</td></tr>
</tbody>
</table>
For example, if your version of Elasticsearch is 6.x, then you would use something like this:
<div class="code-snippet-container">
<a fs-copyclip-element="click-4" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-4" class="hljs language-javascript">gem 'elasticsearch-model', github: 'elastic/elasticsearch-rails', branch: '6.x'
gem 'elasticsearch-rails', github: 'elastic/elasticsearch-rails', branch: '6.x'
gem 'bonsai-elasticsearch-rails', github: 'omc/bonsai-elasticsearch-rails', branch: '6.x'</code></pre>
</div>
</div>
Make sure the branch you choose corresponds to the version of Elasticsearch that your Bonsai cluster is running.
Once the gems have been added to your Gemfile, run <span class="inline-code"><pre><code>bundle install</code></pre></span> to install them.
Any model that you will want to be searchable with Elasticsearch will need to be configured. You will need to require the `elasticsearch/model` library and include the necessary modules in the model.
For example, this demo app has a <span class="inline-code"><pre><code>User</code></pre></span> model that looks something like this:
<div class="code-snippet-container">
<a fs-copyclip-element="click-5" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-5" class="hljs language-javascript">require 'elasticsearch/model'
class User < ApplicationRecord
include Elasticsearch::Model
include Elasticsearch::Model::Callbacks
settings index: { number_of_shards: 1 }
end</code></pre>
</div>
</div>
Your model will need to have these lines included, at a minimum.
You can optionally avoid (or increase) replicas by amending the settings like this:
<div class="code-snippet-container">
<a fs-copyclip-element="click-6" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-6" class="hljs language-javascript">require 'elasticsearch/model'
class User < ApplicationRecord
include Elasticsearch::Model
include Elasticsearch::Model::Callbacks
settings index: {
number_of_shards: 1, # 1 primary shard; probably fine for most users
number_of_replicas: 0 # 0 replicas; fine for dev, not for production.
}
end</code></pre>
</div>
</div>
If you have more questions about shards and how many is enough, check out our Shard Primer and our documentation on Capacity Planning.
You will need to set up a route to handle searching. There are a few different strategies for this, but we generally recommend using a dedicated controller for it, unrelated to the model(s) being indexed and searched. This is a more flexible approach, and keeps concerns separated. You can always render object-specific partials if your results involve multiple models.
In our example Rails app, we have one model, <span class="inline-code"><pre><code>User</code></pre></span>, with a handful of attributes. It looks something like this:
<div class="code-snippet-container">
<a fs-copyclip-element="click-7" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-7" class="hljs language-javascript">require 'elasticsearch/model'
class User < ApplicationRecord
include Elasticsearch::Model
include Elasticsearch::Model::Callbacks
settings index: { number_of_shards: 1 }
end</code></pre>
</div>
</div>
To implement search, we created a file called <span class="inline-code"><pre><code>app/controllers/search_controller.rb</code></pre></span> and added this code:
<div class="code-snippet-container">
<a fs-copyclip-element="click-8" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-8" class="hljs language-javascript">class SearchController< ApplicationController
def run
# This will search all models that have `include Elasticsearch::Model`
@results = Elasticsearch::Model.search(params[:q]).records
end
end</code></pre>
</div>
</div>
We then created a route in the <span class="inline-code"><pre><code>config/routes.rb</code></pre></span> file:
<div class="code-snippet-container">
<a fs-copyclip-element="click-9" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-9" class="hljs language-javascript">post '/search', to: 'search#run'</code></pre>
</div>
</div>
Next, we need to have some Views to render the data we get back from Elasticsearch. The run controller action will be rendered by creating a file called <span class="inline-code"><pre><code>app/views/search/run.html.erb</code></pre></span> and adding:
<div class="code-snippet-container">
<a fs-copyclip-element="click-10" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-10" class="hljs language-javascript"> Search Results
<% if @results.present? %>
<%= render partial: 'search_result', collection: @results, as: :result %>
<% else %>
Nothing here, chief!
<% end %></code></pre>
</div>
</div>
This way if there are no results to show, we simply put a banner indicating as such. If there are results to display, we will iterate over the collection (assigning each one to a local variable called <span class="inline-code"><pre><code>result</code></pre></span>), and passing it off to a partial. Create a file called <span class="inline-code"><pre><code>app/views/search/_search_result.html.erb</code></pre></span> and add:
<div class="code-snippet-container">
<a fs-copyclip-element="click-11" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-11" class="hljs language-javascript">
<%= link_to "#{result.first_name} #{result.last_name} <#{result.email}>", user_path(result) %>
<%= result.company %>
<%= result.company_description %></code></pre>
</div>
</div>
This partial simply renders a search result using some of the data of the matching ActiveRecord objects.
At this point, the <span class="inline-code"><pre><code>User</code></pre></span> model is configured for searching in Elasticsearch, and has routes for sending a query to Elasticsearch. The next step is to render a form so that a user can actually use this feature. This is possible with a basic <span class="inline-code"><pre><code>form_with</code></pre></span> helper in a Rails view.
In this demo app, we added this to the navigation bar:
<div class="code-snippet-container">
<a fs-copyclip-element="click-12" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-12" class="hljs language-javascript"><%= form_with(url: "/search", method: "post", class: 'form-inline my-2 my-lg-0', local: true) do %>
<%= text_field_tag(:q, nil, class: "form-control mr-sm-2", placeholder: "Search") %>
<%= button_tag("Search", class: "btn btn-outline-info my-2 my-sm-0", name: nil) %>
<% end %></code></pre>
</div>
</div>
This code renders a form that looks like this:
Please note that these classes use Bootstrap, which may not be in use with your application. The ERB scaffold should be easily adapted to your purposes.
We’re close to finishing up. We just need to tell the app where the Bonsai cluster is located, then push our data into that cluster.
By default, the gems will try to connect to a cluster running on <span class="inline-code"><pre><code>localhost:9200</code></pre></span>. This is a problem because your Bonsai cluster is not running on a localhost. We need to make sure Elasticsearch is pointed to the correct URL.
If you are using the bonsai-elasticsearch-rails gem, then all you need to do is ensure that there is an environment variable called <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span> set in your application environment that points at your Bonsai cluster URL.
Heroku users will have this already, and can skip to the next step. Other users will need to make sure this environment variable is manually set in their application environment. If you have access to the host, you can run this command in your command line:
<div class="code-snippet-container">
<a fs-copyclip-element="click-13" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-13" class="hljs language-javascript"># Substitute with your cluster URL, obviously:
export BONSAI_URL="https://abcd123:efg456@my-cluster-123456.us-west-2.bonsaisearch.net:443"</code></pre>
</div>
</div>
You will only need to write an initializer if:
If you need to do this, then you can create a file called <span class="inline-code"><pre><code>config/initializers/elasticsearch.rb</code></pre></span>. Inside this file, you will want to put something like this:
<div class="code-snippet-container">
<a fs-copyclip-element="click-14" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-14" class="hljs language-javascript"># Assuming you can set the BONSAI_URL variable:
Elasticsearch::Model.client = Elasticsearch::Client.new url: ENV['BONSAI_URL']</code></pre>
</div>
</div>
If you’re one of the few who can’t set the <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span> variable, then you’ll need to do something like this:
<div class="code-snippet-container">
<a fs-copyclip-element="click-15" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-15" class="hljs language-javascript"># Use your personal URL, not this made-up one:
Elasticsearch::Model.client = Elasticsearch::Client.new url: "https://abcd123:efg456@my-cluster-123456.us-west-2.bonsaisearch.net:443"</code></pre>
</div>
</div>
If you’re wondering why we prefer to use an environment variable instead of the URL, it’s simply a best practice. The cluster URL is considered sensitive information, in that anyone with the fully-qualified URL is going to have full read/write access to the cluster.
So if you have it in an initializer and check it into source control, that creates an attack vector. Many people have been burned by committing sensitive URLs, keys, passwords, etc to git, and it’s best to avoid it.
Additionally, if you ever need to change your cluster URL, updating the initializer will require another pass through CI and a deployment. Whereas you could otherwise just change the environment variable and restart Rails. Environment variables are simply the better way to go.
Assuming you have a database populated with data, the next step is to get that data into Elasticsearch.
Something to keep in mind here: Bonsai does not support lazy index creation, and even Elastic does not recommend using this feature. You’ll need to create the indices manually for each model that you want to search before you try and populate the cluster with data.
If you have the <span class="inline-code"><pre><code>elasticsearch-rails</code></pre></span> gem, you can use one of the built-in Rake tasks. To do that, create a file called <span class="inline-code"><pre><code>lib/tasks/elasticsearch.rake</code></pre></span> and simply put this inside:
<div class="code-snippet-container">
<a fs-copyclip-element="click-16" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-16" class="hljs language-javascript">require 'elasticsearch/rails/tasks/import'</code></pre>
</div>
</div>
Now you will be able to use a Rake task that will automatically create(or recreate) the index:
<div class="code-snippet-container">
<a fs-copyclip-element="click-17" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-17" class="hljs language-javascript">bundle exec rake environment elasticsearch:import:model CLASS='User'</code></pre>
</div>
</div>
You’ll need to run that Rake task for each model you have designated as destined for Elasticsearch. Note that if you’re relying on the <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span> variable to configure the Rails client, this variable will need to be present in the environment running the Rake task. Otherwise it will populate your local Elasticsearch instance(or raise some exceptions).
If you don’t use the Rake task, then you can also create and populate the index from within a Rails console with:
<div class="code-snippet-container">
<a fs-copyclip-element="click-18" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-18" class="hljs language-javascript"># Will return nil if the index already exists:
User.__elasticsearch__.create_index!
# Will delete the index if it already exists, then recreate:
User.__elasticsearch__.create_index! force: true
# Import the data into the Elasticsearch index:
User.import</code></pre>
</div>
</div>
At this point you should have all of the pieces you need to search your data using Elasticsearch. In our demo app, we have this simple list of users:
This search box is rendered by a form that will pass the query to the <span class="inline-code"><pre><code>SearchController#run</code></pre></span> action, via the route set up in <span class="inline-code"><pre><code>config/routes.rb</code></pre></span>:
This query will reach the <span class="inline-code"><pre><code>SearchController#run</code></pre></span> action, where it will be passed to Elasticsearch. Elasticsearch will search all of the indices it has (just <span class="inline-code"><pre><code>users</code></pre></span> at this point), and return any hits to a class variable called <span class="inline-code"><pre><code>@results</code></pre></span>.
The SearchController will then render the appropriate views. Each result will be rendered by the partial <span class="inline-code"><pre><code>app/views/search/_search_result.html.erb</code></pre></span>. It looks something like this:
Congratulations! You have implemented Elasticsearch in Rails!
This documentation demonstrated how to quickly get Elasticsearch added to a basic Rails application. We added the gems, configured the model, set up the search route, and created the views and partials needed to render the results. Then we set up the connection to Elasticsearch and pushed the data into the cluster. Finally, we were able to search that data through our app.
Hopefully this was enough to get you up and running with Elasticsearch on Rails. This documentation is not exhaustive. However, there are a number of other use cases not discussed here. There are other additional changes and customizations that can be implemented to make search more accurate and resilient.
You can find information on these additional subjects below. And if you have any ideas or requests for additional content, please don’t hesitate to let us know!
Here’s how to get started with Bonsai Elasticsearch and Ruby on Rails using Chewy.
Chewy is built on top of the official Elasticsearch Ruby client, which is not supported on Bonsai after version 7.13. This is due to a change introduced in the 7.14 release of the gem. This change prevents the Ruby client from communicating with open-sourced versions of Elasticsearch 7.x, as well as any version of OpenSearch. The table below indicates compatibility:
<table>
<thead>
<tr><th>Engine</th><th>Version Highest Compatible Gem Version</th></tr>
</thead>
<tbody>
<tr><td>Elasticsearch 5.x</td><td>7.13</td></tr>
<tr><td>Elasticsearch 6.x</td><td>7.14 (sic)</td></tr>
<tr><td>Elasticsearch 7.x</td><td>7.13</td></tr>
<tr><td>OpenSearch 1.x</td><td>7.13</td></tr>
</tbody>
</table>
EngineVersionHighest Compatible Gem VersionElasticsearch5.x7.13Elasticsearch6.x7.14+ ( sic)Elasticsearch7.x7.13OpenSearch1.x7.13
If you are receiving a <span class="inline-code"><pre><code>Elasticsearch::UnsupportedProductError</code></pre></span>, then you'll need to ensure you're using a supported version of the Elasticsearch Ruby client.
As of January 2021, Chewy supports up to Elasticsearch 5.x. Users wanting to use Chewy will need to ensure they are not running anything later than 5.x. Support for later versions is planned.
In order to use the Chewy gem, add the gem to the Gemfile like so:
<div class="code-snippet-container">
<a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">gem 'chewy'
</code></pre>
</div>
</div>
Then run <span class="inline-code"><pre><code>bundle install</code></pre></span> to install it.
A Chewy initializer will need to be written so that the Rails app can connect to your Bonsai cluster.
You can include your Bonsai cluster URL (located in Credentials of Bonsai dashboard) in the initializer, but hard-coding your credentials is not recommended. Instead, export them manually in the application environment to an environment variable called <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span>.
We recommend using something like dotenv for this, but you can also set it manually like so:
<div class="code-snippet-container">
<a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript"># Substitute with your own Bonsai cluster URL:
export BONSAI_URL="https://abcd123:efg456@my-cluster-123456.us-west-2.bonsaisearch.net:443"</code></pre>
</div>
</div>
Heroku users will not need to do so, as their Bonsai cluster URL will already be in a Config Var of the same name.
Create a file called <span class="inline-code"><pre><code>config/initializers/chewy.rb</code></pre></span>. With the environment variable in place, save the initializer with the following:
<div class="code-snippet-container">
<a fs-copyclip-element="click-4" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-4" class="hljs language-javascript">Chewy.settings = {
host: ENV['BONSAI_URL']
}</code></pre>
</div>
</div>
The official Chewy documentation has more details on ways to modify and refine Chewy’s behavior.
Any Rails model that you will want to be searchable with Elasticsearch will need to be configured to do so by creating a Chewy index definition and adding model-observing code.
For example, here’s how we created an index definition for our demo app in <span class="inline-code"><pre><code>app/chewy/users_index.rb</code></pre></span>:
<div class="code-snippet-container">
<a fs-copyclip-element="click-5" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-5" class="hljs language-javascript">class UsersIndex < Chewy::Index
define_type User
end</code></pre>
</div>
</div>
Then, the User model is written like so:
<div class="code-snippet-container">
<a fs-copyclip-element="click-6" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-6" class="hljs language-javascript">class User < ApplicationRecord
update_index('users#user') { self }
end</code></pre>
</div>
</div>
You can read more about configuring how a model will be ingested by Elasticsearch in the official Chewy documentation.
The Chewy gem comes with some Rake tasks for getting your documents into the cluster. Once you have set up your models as desired, run the following in your application environment:
<div class="code-snippet-container">
<a fs-copyclip-element="click-7" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-7" class="hljs language-javascript">bundle exec rake chewy:deploy
</code></pre>
</div>
</div>
That will push your data into Elasticsearch and synchronize the data.
Chewy is a mature ODM with plenty of great features. Toptal (the maintainers of Chewy) have some great documentation exploring some of what Chewy can do: Elasticsearch for Ruby on Rails: A Tutorial to the Chewy Gem.
Getting started with Ruby on Rails and Bonsai Elasticsearch is fast and easy with Searchkick. In this guide, we will start with a very basic Ruby on Rails application and add the bare minimum amount of code needed to support basic search with Elasticsearch. Users looking for more details and advanced usage should consult the resources at the end of this page.
Throughout this guide, you will see some code examples. These code examples are drawn from a very simple Ruby on Rails application, and are designed to offer some real-world, working code that new users will find useful. The complete demo app can be found in this GitHub repo.
<div class="callout-warning">
<h3>Warning</h3>
<p>SearchKick uses the official Elasticsearch Ruby client, which is not supported on Bonsai after version 7.13. This is due to a change introduced in the 7.14 release of the gem. This change prevents the Ruby client from communicating with open-sourced versions of Elasticsearch 7.x, as well as any version of OpenSearch. The table below indicates compatibility:</p>
<table>
<thead>
<tr><th>Engine</th><th>Version Highest Compatible Gem Version</th></tr>
</thead>
<tbody>
<tr><td>Elasticsearch 5.x</td><td>7.13</td></tr>
<tr><td>Elasticsearch 6.x</td><td>7.14+ (sic)</td></tr>
<tr><td>Elasticsearch 7.x</td><td>7.13</td></tr>
<tr><td>OpenSearch 1.x</td><td>7.13</td></tr>
</tbody>
</table>
<p>If you are receiving a <span class="inline-code warning">Elasticsearch::UnsupportedProductError</span>, then you'll need to ensure you're using a supported version of the Elasticsearch Ruby client.</p>
</div>
<div class="callout-note">
<h3>Note</h3>
<p>In this example, we are going to connect to Elasticsearch using the Searchkick gem. There are also the official Elasticsearch gems for Rails, which are covered in another set of documentation.</p>
</div>
Make sure that there is a Bonsai Elasticsearch cluster ready for your app to interact with. This needs to be set up first so you know which version of the gems you need to install; Bonsai supports a large number of Elasticsearch versions, and the gems need to correspond to the version of Elasticsearch you’re running.
Bonsai clusters can be created in a few different ways, and the documentation for each path varies. If you need help creating your cluster, check out the link that pertains to your situation:
When you have successfully created your cluster, it will be given a semi-random URL called the Elasticsearch Access URL. You can find this in the Cluster Overview, in the Credentials tab:
Heroku users will also have a <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span> environment variable created when Bonsai is added to the application. This variable will contain the fully-qualified URL to the cluster.
When you have a Bonsai Elasticsearch cluster, there are a few ways to check the version that it is running. These are outlined below:
The easiest is to simply get it from the Cluster Dashboard. When you view your cluster overview in Bonsai UI, you will see some details which include the version of Elasticsearch the cluster is running:
You can also use the Interactive Console. In the Cluster Dashboard, click on the Console tab. It will load a default view, which includes the version of Elasticsearch. The version of Elasticsearch is called “number” in the JSON response:
You can copy/paste your cluster URL into a browser or into a tool like <span class="inline-code"><pre><code>curl</code></pre></span>. Either way, you will get a response like so:
<div class="code-snippet-container">
<a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">curl https://abcd123:efg456@my-cluster-123456.us-west-2.bonsaisearch.net:443
{
"name" : "ip-172-31-14-16",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "jVJrINr5R5GVVXHGcRhMdA",
"version" : {
"number" : "7.2.0",
"build_flavor" : "oss",
"build_type" : "tar",
"build_hash" : "508c38a",
"build_date" : "2019-06-20T15:54:18.811730Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}</code></pre>
</div>
</div>
The version of Elasticsearch is called “number” in the JSON response.
To install Searchkick, you will need the searchkick gem. Add the following to your Gemfile outside of any blocks:
<div class="code-snippet-container">
<a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript">gem 'searchkick'
</code></pre>
</div>
</div>
This will install the gem for the latest major version of Elasticsearch. If you have an older version of Elasticsearch, then you should follow this table:
<table>
<thead>
<tr><th>Elasticsearch Version</th><th>Searchkick Version</th></tr>
</thead>
<tbody>
<tr><td>1.x</td><td>-> 1.5.1</td></tr>
<tr><td>2.x</td><td>-> 2.5.0</td></tr>
<tr><td>5.x</td><td>-> 3.1.3 (additional notes)</td></tr>
<tr><td>6.x and up</td><td>-> 4.0 and up</td></tr>
</tbody>
</table>
If you need a specific version of Searchkick to accommodate your Elasticsearch cluster, you can specify it in your Gemfile like so:
<div class="code-snippet-container">
<a fs-copyclip-element="click-4" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-4" class="hljs language-javascript">gem 'searchkick', '3.1.3' # For Elasticsearch 5.x
</code></pre>
</div>
</div>
Once the gem has been added to your Gemfile, run <span class="inline-code"><pre><code>bundle install</code></pre></span>.
Any model that you will want to be searchable with Elasticsearch will need to be configured to do so by adding the <span class="inline-code"><pre><code>searchkick</code></pre></span> keyword to it.
For example, our demo app has a User model that looks something like this:
<div class="code-snippet-container">
<a fs-copyclip-element="click-5" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-5" class="hljs language-javascript">class User < ApplicationRecord
searchkick
end</code></pre>
</div>
</div>
Adding the searchkick keyword makes our User model searchable with Searchkick.
Searchkick provides a number of reasonable settings out of the box, but you can also pass in a hash of settings if you want to override the defaults. The hash keys generally correspond to the official Create Indices API. For example, this will allow you to create an index with 0 replicas:
<div class="code-snippet-container">
<a fs-copyclip-element="click-6" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-6" class="hljs language-javascript">class User < ApplicationRecord
searchkick settings: { number_of_replicas: 0 }
end</code></pre>
</div>
</div>
If you have questions about shards and how many is enough, check out our Shard Primer and our documentation on Capacity Planning.
You will need to set up a route to handle searching. The easiest way to do this with Searchkick is to have a search route per model. This involves updating your models’ corresponding controller, and defining routes in <span class="inline-code"><pre><code>config/routes.rb</code></pre></span>. You’ll also need to have some views that handle rendering the results, and a form that posts data to the controller(s). Take a look at how we implemented this in our demo app for some examples of how this is done:
In our example Rails app, we have one model, <span class="inline-code"><pre><code>User</code></pre></span>, with <span class="inline-code"><pre><code>searchkick</code></pre></span>. It looks something like this:
<div class="code-snippet-container">
<a fs-copyclip-element="click-7" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-7" class="hljs language-javascript">class User < ApplicationRecord
searchkick
end</code></pre>
</div>
</div>
To implement search, we updated the file <span class="inline-code"><pre><code>app/controllers/users_controllers.rb</code></pre></span> and added this code:
<div class="code-snippet-container">
<a fs-copyclip-element="click-8" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-8" class="hljs language-javascript">class UsersController< ApplicationController
#... a bunch of controller actions, removed for brevity
def search
@results = User.search(params[:q])
end
#... more controller actions removed for brevity
end</code></pre>
</div>
</div>
We then created a route in the <span class="inline-code"><pre><code>config/routes.rb</code></pre></span> file:
<div class="code-snippet-container">
<a fs-copyclip-element="click-9" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-9" class="hljs language-javascript">Rails.application.routes.draw do
resources :users do
collection do
post :search # creates a route called users_search
end
end
end</code></pre>
</div>
</div>
Next, we need to have some views to render the data we get back from Elasticsearch. The <span class="inline-code"><pre><code>search</code></pre></span> controller action will be rendered by creating a file called <span class="inline-code"><pre><code>app/views/users/search.html.erb</code></pre></span> and adding:
<div class="code-snippet-container">
<a fs-copyclip-element="click-10" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-10" class="hljs language-javascript">Search Results
<% if @results.present? %>
<%= render partial: 'search_result', collection: @results, as: :result %>
<% else %>
Nothing here, chief!
<% end %>
</code></pre>
</div>
</div>
This way if there are no results to show, we simply put a banner indicating as such. If there are results to display, we will iterate over the collection (assigning each one to a local variable called <span class="inline-code"><pre><code>result</code></pre></span>), and passing it off to a partial. We also created a file for a partial called <span class="inline-code"><pre><code>app/views/users/_search_result.html.erb</code></pre></span> and added:
<div class="code-snippet-container">
<a fs-copyclip-element="click-11" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-11" class="hljs language-javascript">
<%= link_to "#{result.first_name} #{result.last_name} <#{result.email}>", user_path(result) %>
<%= result.company %>
<%= result.company_description %>
</code></pre>
</div>
</div>
This partial simply renders a search result using some of the data of the matching ActiveRecord objects.
At this point, the <span class="inline-code"><pre><code>User</code></pre></span> model is configured for searching in Elasticsearch, and has routes for sending a query to Elasticsearch. The next step is to render a form so that a user can actually use this feature. This is possible with a basic <span class="inline-code"><pre><code>form_with</code></pre></span> helper.
In this demo app, we added this to the navigation bar:
<div class="code-snippet-container">
<a fs-copyclip-element="click-12" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-12" class="hljs language-javascript"><%= form_with(url: "/search", method: "post", class: 'form-inline my-2 my-lg-0', local: true) do %>
<%= text_field_tag(:q, nil, class: "form-control mr-sm-2", placeholder: "Search") %>
<%= button_tag("Search", class: "btn btn-outline-info my-2 my-sm-0", name: nil) %>
<% end %></code></pre>
</div>
</div>
This code renders a form that looks like this:
Please note that these classes use Bootstrap, which may not be in use with your application. The ERB scaffold should be easily adapted to your purposes.
We’re close to finishing up. We just need to tell the app where the Bonsai cluster is located, then push our data into that cluster.
Searchkick looks for an environment variable called <span class="inline-code"><pre><code>ELASTICSEARCH_URL</code></pre></span>, and if it doesn’t find it, it uses <span class="inline-code"><pre><code>localhost:9200</code></pre></span>. This is a problem because your Bonsai cluster is not running on a localhost. We need to make sure Searchkick is pointed to the correct URL.
Bonsai does offer a gem, bonsai-searchkick, which populates the necessary environment variable automatically. If you're using this gem, then all you need to do is ensure that there is an environment variable called <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span> set in your application environment that points at your Bonsai cluster URL.
Heroku users will have this already, and can skip to the next step. Other users will need to make sure this environment variable is manually set in their application environment. If you have access to the host, you can run this command in your command line:
<div class="code-snippet-container">
<a fs-copyclip-element="click-13" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-13" class="hljs language-javascript"># Substitute with your cluster URL, obviously:
export BONSAI_URL="https://abcd123:efg456@my-cluster-123456.us-west-2.bonsaisearch.net:443"</code></pre>
</div>
</div>
You will only need to write an initializer if:
If you need to do this, then you can create a file called <span class="inline-code"><pre><code>config/initializers/elasticsearch.rb</code></pre></span>. Inside this file, you will want to put something like this:
<div class="code-snippet-container">
<a fs-copyclip-element="click-14" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-14" class="hljs language-javascript"># Assuming you can set the BONSAI_URL variable:
ENV["ELASTICSEARCH_URL"] = ENV['BONSAI_URL']</code></pre>
</div>
</div>
If you’re one of the few who can’t set the <span class="inline-code"><pre><code>BONSAI_URL</code></pre></span> variable, then you’ll need to do something like this:
<div class="code-snippet-container">
<a fs-copyclip-element="click-15" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-15" class="hljs language-javascript"># Use your personal URL, not this made-up one:
ENV["ELASTICSEARCH_URL"] = "https://abcd123:efg456@my-cluster-123456.us-west-2.bonsaisearch.net:443"</code></pre>
</div>
</div>
If you’re wondering why we prefer to use an environment variable instead of the URL, it’s simply a best practice. The cluster URL is considered sensitive information, in that anyone with the fully-qualified URL is going to have full read/write access to the cluster.
So if you have it in an initializer and check it into source control, that creates an attack vector. Many people have been burned by committing sensitive URLs, keys, passwords, etc to git, and it’s best to avoid it.
Additionally, if you ever need to change your cluster URL, updating the initializer will require another pass through CI and a deployment. Whereas you could otherwise just change the environment variable and restart Rails. Environment variables are simply the better way to go.
Now that the app has everything it needs to query the cluster and render the results, we need to push data into the cluster. There are a few ways to do this.
One method is to open up a Rails console and run <span class="inline-code"><pre><code>.reindex</code></pre></span>. So if you want to reindex a model called <span class="inline-code"><pre><code>User</code></pre></span>, you would run <span class="inline-code"><pre><code>User.index</code></pre></span>.
Another method is to use Rake tasks from the command line. If you wanted to reindex that same <span class="inline-code"><pre><code>User</code></pre></span> model, you could run: <span class="inline-code"><pre><code>bundle exec rake searchkick:reindex CLASS=User</code></pre></span>. Alternatively, if you have multiple Searchkick-enabled models, you could run <span class="inline-code"><pre><code>rake searchkick:reindex:all</code></pre></span>.
Regardless of how you do it, Searchkick will create an index named after the ActiveRecord table of the model, the environment, and a timestamp. So reindexing the <span class="inline-code"><pre><code>User</code></pre></span> model in a development environment might result in an index called <span class="inline-code"><pre><code>users_development_20191029111649033</code></pre></span>. This allows Searchkick to provide zero-downtime updates to settings and mappings.
At this point you should have all of the pieces you need to search your data using Searchkick. In our demo app, we have this simple list of users:
This search box is rendered by a form that will pass the query to the <span class="inline-code"><pre><code>UsersController#search</code></pre></span> action, via the route set up in <span class="inline-code"><pre><code>config/routes.rb</code></pre></span>:
This query will reach the <span class="inline-code"><pre><code>UsersController#search</code></pre></span> action, where it will be passed to Searchkick, which queries Elasticsearch. Elasticsearch will search the <span class="inline-code"><pre><code>users_development_20191029111649033</code></pre></span> index, and return any hits to a class variable called <span class="inline-code"><pre><code>@results</code></pre></span>.
The UsersController will then ensure the appropriate views are rendered. Each result will be rendered by the partial <span class="inline-code"><pre><code>app/views/users/_search_result.html.erb</code></pre></span>. It looks something like this:
Congratulations! You have implemented Searchkick in Rails!
This documentation demonstrated how to quickly get Elasticsearch added to a basic Rails application. We installed the Searchkick gem, added it to a model, set up the search route, and created the views and partials needed to render the results. Then we set up the connection to Elasticsearch and pushed the data into the cluster. Finally, we were able to search that data through our app.
Hopefully this was enough to get you up and running with Searchkick. This documentation is not exhaustive, and there are a lot of really cool features that Searchkick offers. There are other additional changes and customizations that can be implemented to make search more accurate and resilient.
You can find information on additional subjects in the section below. And if you have any ideas or requests for additional content, please don’t hesitate to let us know!
Jekyll is a static site generator written in Ruby. Jekyll supports a plugin model that Searchyll uses to read your site’s content and then index it into an Elasticsearch cluster.
In this guide, we are going to use this feature to tell Jekyll to index all of the content into a configured instance of Elasticsearch.
In order to make use of this documentation, you will need Jekyll installed and configured on your system.
Jekyll’s configuration systems live in a file called `_config` by default. Add the following snippet to the file:
<div class="code-snippet-container">
<a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">elasticsearch:
url: "https://:@my-awesome-cluster-1234.us-east-1.bonsai.io"
index_name: jekyll-posts
plugins:
- searchyll</code></pre>
</div>
</div>
We also need to add the gem `searchyll`
<div class="code-snippet-container">
<a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript"># Manually add the Searchyll gem into your Gemfile
bundle install</code></pre>
</div>
</div>
To get the site's data into Elasticsearch, you can load it by simply running the Jekyll build command:
<div class="code-snippet-container">
<a fs-copyclip-element="click-4" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-4" class="hljs language-javascript">jekyll build
</code></pre>
</div>
</div>
You should now be able to see your data in the Elasticsearch cluster:
<div class="code-snippet-container">
<a fs-copyclip-element="click-5" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-5" class="hljs language-javascript">$ curl -XGET "https://:@my-awesome-cluster-1234.us-east-1.bonsai.io/_search"
{"took":1,"timed_out":false,"_shards":{"total":2,"successful":2,"failed":0},"hits":{"total":1,"max_score":1.0,"hits":[{"_index":"hugo","_type":"doc","_id":...</code></pre>
</div>
</div>
At Bonsai, security and privacy are always a top concern. We are constantly evaluating the service and our vendors for vulnerabilities and flaws, and we will immediately address anything that could put our customers at risk. This is how we keep your data secure:
Note: The following article applies to Business and Enterprise tiers with dedicated single tenant clusters. These watermark alerts do not apply to Sandbox, Staging, or Standard tiers on shared multitenant clusters; instead, these tiers will be notified once an overage is detected on at least one metered metric.
Elasticsearch protects itself from data loss with the disk-based shard allocator. This mechanism attempts to strike a balance between minimizing disk usage across all nodes while also minimizing the number of shard reallocation processes. This ensures that all nodes have as much disk headroom as possible, with minimal impact to cluster performance.
Elasticsearch is constantly checking on each node’s disk availability to make decisions about where to allocate and move shards. There are 4 important“stages” that help dictate this decision making process:
The low and high watermark stages are when Business and Enterprise tiers receive an emailed notification from the Bonsai Support team with suggestions for scaling disk capacity. Clusters that are over the 75% threshold can start to experience performance issues that may include but are not limited to: increased bulk queue times, higher CPU and/or load usage, or slower than normal searches.
Bonsai offers two main architecture classes: multitenant and single tenant. The multitenant class – sometimes called “shared” – is designed to allow clusters to share hardware resources while still being securely sandboxed from one another. This allows us to provide unparalleled performance per dollar at smaller scales. All Hobby and Standard plans use multitenant architecture.
The single tenant class – sometimes called “dedicated” – maps one cluster to a private set of hardware resources. Because these resources are not shared with any other cluster, single tenant configurations provide maximum performance, security and customization. All Business and Enterprise plans use dedicated architecture.
Bonsai’s multitenant class utilizes some sandboxing features built into Elasticsearch. This allows the service to run multiple clusters on a single instance of Elasticsearch per node. This approach saves substantial hardware and network resources, and allows for radical cost savings especially for students, hobbyists, startups, projects in development, small businesses, and so on.
A great benefit of this approach is that Bonsai is able to provide some really nice features out of the box, for no additional cost: all multitenant clusters – even the free ones – are running on 3 nodes. They also get industry standard SSL/TLS and HTTP Basic authentication (see Security for more information), which keeps your data safe. Plus, the Bonsai dashboard offers plenty of tools for monitoring, managing, and engaging with your cluster.
Because these clusters are running on a shared Elasticsearch instance, there are also a few limitations. For one, certain API endpoints and actions are unavailable for security and performance reasons. Snapshots and plugins are not manageable by users to avoid collisions and regressions. And usage is metered about how you’d expect for a free/low-cost SaaS.
Clusters on the multitenant class can often be identified by their plan name. “Hobby”, “Staging”, “Production”, and “Shared” are all terms used by plans running on a multitenant architecture.
Bonsai’s single tenant class has a fairly standard configuration. The cluster is simply one or more nodes (three by default), each running Elasticsearch. These nodes are physically isolated and on a different network than those running multitenant clusters. This means that all available IO on the nodes is always 100% allocated to your cluster.
This approach offers all of the same benefits as the multitenant class: you get the same industry standard SSL/TLS and HTTP Basic authentication (see Security for details), and the Bonsai dashboard. In addition, the isolated environment is suitable for encryption at rest and VPC Peering (for applications with stringent security requirements).
Furthermore, this class offer extremely flexible deployments and scaling. Need it in a region we don’t support on the multitenant class? No problem! Have a plugin or script that is vital to your operation? We can package it into our deployment! Let us know what you need, and we can provide a quote.
Also, because this class is highly secure and customizable, we are also able to support amendments to our terms of service and privacy policy, custom SLAs and more. Contact us for questions and quotes.
This article details the differences between the two architectures we offer. There are a number of trade offs between these classes, summarized below:
<table>
<thead>
<tr><th>Class</th><th>Pros</th><th>Cons</th></tr>
</thead>
<tbody>
<tr><td>Multi-tenant</td><td>- Extremely cost effective
- Great performance for the money
- Can scale up or down on demand</td><td>- Limits on usage (disk, memory, connections, etc)
- Can not install and run arbitrary plugins
- Noisy neighbors*
- VPC pairing and at-rest encryption not available
- Subject to general terms of service and privacy policy</td></tr>
<tr><td>Single Tenant</td><td>- Extremely powerful
- No metering on usage
- Can deploy arbitrary plugins
- No noisy neighbors*
- Can have at-rest encryption
- VPC Pairing
- Custom terms, SLAs, etc</td><td>- More expensive to operate
- Scaling can be more difficult</td></tr>
</tbody>
</table>
* The Noisy Neighbor Problem is a well-known issue that occurs in multitenant architectures. In this context, one or more clusters may inadvertently monopolize IO resources (CPU, network, memory, etc), which can adversely affect other users on the same nodes. Bonsai actively monitors and addresses these situations when they come up, although the issue is frequently transient and resolves itself within a few minutes. Single tenant architectures do not suffer from this issue.
A frequent question that comes up when talking about our various service architectures is why we don’t use container or virtualization technologies. A service that incorporates these technologies would offer some nice benefits, like allowing users to install their own plugins and manage their own snapshots.
The simple answer is “overhead.” Running containers – and especially VMs – requires system resources via some orchestration daemon or hypervisor. And simply running multiple instances of Elasticsearch on a node would require multiple JREs, which wastes resources through duplication.
Any resources that are allocated towards management of environments are therefore unavailable for Elasticsearch to use. In comparison to an architecture that doesn’t have this overhead, the provider must either offer less performance for the money, or charge more money for the same performance.
There is also a practical aspect to Bonsai eschewing containers and virtual machines. It is impossible to provide both great support and absolute customization. For example, users who install their own plugins can introduce a variety of regressions into Elasticsearch’s performance and behavior. When they open a support ticket, the agent must either spend time bug squashing or decline assistance altogether. Being opinionated allows our team to focus on depth of knowledge rather than breadth, which leads to faster, higher quality resolutions.
Finally, there is a philosophical motivation for how we built the service. We want to make Elasticsearch accessible to people at all stages of development; from the hobbyists and students, all the way up to the billion dollar unicorns. And we want to make sure that it’s the best possible experience. This means being opinionated about certain features, and taking a more active role in managing the infrastructure.
Bonsai supports a large number of Elasticsearch versions in regions all over the world. Multitenant class clusters usually have more limited options in terms of available regions and versions, while single tenant class clusters have far more options.
If you need a version or geographic region that is not listed here, reach out to our support and let us know what you need. We can get you a quote and timeline for getting up and running.
Bonsai operates a fleet of shared resource nodes in Oregon, Virginia, Ireland, Frankfurt, and Sydney. The Elasticsearch versions available on these nodes do not change often.
Sandbox clusters are limited to the most recent version of Elasticsearch and may be subject to automatic upgrades when a new version is released.
For all other multitenant plans, when Bonsai adds support for a new version, we will create a new server group rather than upgrading an existing group. The exception to this is a potential patch upgrade in response to some critical vulnerability. In other words, users will not be upgraded in place unless there is a vulnerability to address or if they are on a Sandbox plan. Read How Bonsai Handles Elasticsearch Releases for more information.
This table shows which versions of Elasticsearch are available for multitenant plans. Last updated: 2023-08-10.
<table>
<thead>
<tr><th>Plan</th><th>Elasticsearch Versions </th><th>OpenSearch Versions</th></tr>
</thead>
<tbody>
<tr><td>Sandbox</td><td>7.10.2</td><td>2.6.0</td></tr>
<tr><td>All other multitenant </td><td>5.6.16 / 6.8.21 / 7.10.2 </td><td>2.6.0</td></tr>
</tbody></table>
Multitenant plans are supported in a handful of popular regions, although the versions available to free plans is limited. Additional regions are available to Business and Enterprise subscriptions.
<table>
<thead>
<tr><th>Cloud</th><th>Region</th><th>Location</th><th>Elasticsearch 5.6.16</th><th>Elasticsearch 6.8.21</th><th>Elasticsearch 7.10.2</th><th>OpenSearch 2.6.0</th></tr>
</thead>
<tbody>
<tr><td>AWS</td><td>us-east-1</td><td>Virginia, USA</td><td>Standard Plans</td><td>Standard Plans</td><td>Standard, Free Plans</td><td>Standard Free Plans</td></tr>
<tr><td>AWS</td><td>us-west-2</td><td>Oregon, USA</td><td>Standard Plans</td><td>Standard Plans</td><td>Standard, Free Plans</td><td>Standard, FreePlans</td></tr>
<tr><td>AWS</td><td>eu-west-1</td><td>Ireland, EU</td><td>Standard Plans</td><td>Standard Plans</td><td>Standard, Free Plans</td><td>Standard, Free Plans</td></tr>
<tr><td>GCP</td><td>us-east1</td><td>Virginia, USA</td><td>Standard Plans</td><td>Standard Plans</td><td>Standard, Free Plans</td><td>Standard, Free Plans</td></tr>
</tbody>
</table>
Single tenant clusters can be deployed in Oregon, Virginia, Ireland, Frankfurt, Sydney, and Tokyo. We support a variety of Elasticsearch versions for these kinds of clusters and will default to the most recent minor version unless something else is specified.
This table shows which versions of Elasticsearch are available for single tenant plans. Last updated: 2023-08-10.
<table>
<thead>
<tr><th>Plan</th><th>Elasticsearch Versions </th><th>OpenSearch Versions</th></tr>
</thead>
<tbody>
<tr><td>All single tenant </td><td>2.4.0 to 7.10.2</td><td>1.2.4 to 2.6.0</td></tr>
</tbody></table>
Bonsai can deploy and manage whichever version of Elasticsearch or OpenSearch your use case needs. Please reach out to us at support@bonsai.io.
For major search engine versions behind Bonsai’s current primary supported version (OpenSearch 2.x and Elasticsearch 7.x), clusters will be charged an operational and maintenance fee to ensure that we can continue to support these older versions, and that we can give your organization the time it needs to decide when to upgrade to a new major version.
The fee will be assessed based on the following table:
Bonsai clusters support most Elasticsearch APIs out of the box, with a few exceptions. This article details those exceptions, along with a brief explanation of why they’re in place. Here is what we will cover :
While having many of the following endpoints can be helpful for power users, the majority of applications don’t directly need them. If, however, you find yourself stuck without one of these available, please email us and we’ll be happy to help.
Wildcard delete actions are usually for clusters with a large number of indices, and can be useful for completely wiping out a cluster and starting over. Wildcard and _all destructive actions were initially available on shared tier clusters. However, we received an increasingly large number of threads from distressed developers that accidentally deleted their entire production clusters.
After some internal discussion, we decided to disable wildcard actions. Removing the ability to sweepingly delete everything forces users to slow down and identify exactly what they’re deleting, reducing the risk of accidental and permanent data loss.
Examples of _all and wildcard destruction:
<div class="code-snippet w-richtext">
<pre><code fs-codehighlight-element="code" class="hljs language-javascript">DELETE /*
DELETE /_all</code>
</pre></div>
Elasticsearch provides an API for viewing information about current running tasks and the ability to cancel them on versions 5.x and up. This API is disabled for multi-tenant plans. It can be enabled by request on Business and Enterprise plans through an email to support.
<div class="code-snippet-container">
<a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">GET /_tasks
GET /_tasks/{task_id}
POST /_tasks/{task_id}/_cancel</code></pre>
</div>
</div>
A Java thread that uses lots of CPU and runs for an unusually long period of time is known as a hot thread. Elasticsearch provides an API to get the current hot threads on each node in the cluster. This information can be useful in forming a holistic picture of potential problems within the cluster.
Bonsai doesn’t support these endpoints on our shared tier to ensure user activity isn’t exposed to others. For a detailed explanation of why this is a concern, please see the article on Architecture Classes. Additionally, Bonsai is a managed service, so it’s really the responsibility of our Ops Team to investigate node-level issues.
If you think there is a problem with your cluster that you need help troubleshooting, please email support.
<div class="code-snippet-container">
<a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript">GET /_cluster/nodes/hotthreads
GET /_cluster/nodes/hot_threads
GET /_cluster/nodes/{node_id}/hotthreads
GET /_cluster/nodes/{node_id}/hot_threads
GET /_nodes/hotthreads
GET /_nodes/hot_threads
GET /_nodes/{node_id}/hotthreads
GET /_nodes/{node_id}/hot_threads</code></pre>
</div>
</div>
Elasticsearch provides an API for shutting down and restarting nodes. This functionality is unsupported across the platform. On our multitenant architecture, this prevents a user from shutting down a node or set of nodes that may be in use by another user. That action would have an adverse affect on other users which is why it is unsupported. This also prevents users from exacerbating whatever problem they’re trying to resolve.
If you think there is a problem with your cluster that you need help troubleshooting, please email support.
<div class="code-snippet-container">
<a fs-copyclip-element="click-4" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-4" class="hljs language-javascript">POST /_cluster/nodes/_restart
POST /_cluster/nodes/_shutdown
POST /_cluster/nodes/{node_id}/_restart
POST /_cluster/nodes/{node_id}/_shutdown
POST /_shutdown</code></pre>
</div>
</div>
The Snapshot API allows users to create and restore snapshots of indices and cluster data. It’s useful as a backup tool and for recovering from problems or data loss. On Bonsai, we’re