For people new to Elasticsearch, shards can be a bit of a mystery. Why is it possible to add or remove replica shards on demand, but not primary shards? What’s the difference? How many indices can fit on a shard?
This article explains what shards are, how they work, and how they can best be used.
First, a little bit of background: Elasticsearch is built on top of Lucene, which is a data storage and retrieval engine. What are called “shards” in Elasticsearch parlance are technically Lucene instances. Elasticsearch manages these different instances, spreading data across different instances, and automatically balancing those instances across different nodes.
Whenever an Elasticsearch index is created, that index will be composed of one or more shards. This means that an Elasticsearch index can spread data across several Lucene instances. This architecture is useful for both redundancy and parallelization purposes.
Shards play one of two roles: primary or replica. Primary shards are a logical partitioning of the data in the index, and are fixed at the time that the index is created. Primary shards are useful for parallelization; when a large amount of data is split across several primary shards, a node can run a query on several Lucene instances in parallel, reducing the overall time of the job.
From one of our more popular blog entries: “Elasticsearch uses a naive hashing algorithm to route documents to a given primary shard. This design choice allows documents to be randomly distributed in a reproducible way. This avoids “hot spots” that affect performance and overallocation. However, it has one major downside, which is that the number of primary shards can not be changed after an index has been created. Replicas can be added and removed at will, but the number of primary shards is basically written in stone.
In contrast, replica shards are simply extra copies of the data. They are useful for redundancy or to handle extra search traffic, and can be added and removed on demand.
You can specify how many primary shards and replicas are used when creating a new index.
<div class="code-snippet-container">
<a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy">
<img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt="">
<img class="copied-image" src="https://cdn.prod.website-files.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt="">
</a>
<div class="code-snippet">
<pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">PUT /my_index/_settings
{
"number_of_shards": 1,
"number_of_replicas": 2
}</code></pre>
</div>
</div>
Replica shards are not only capable of serving search traffic, but they also provide a level of protection against data loss. If a node hosting a primary shard is taken offline for some reason, Elasticsearch will promote its replica to a primary role, if a replica exists. However, if the index’s replication is set to 0, then it is not in a High-Availability configuration. In the event of a data loss incident, the data will simply be lost.
Replicas are a multiplier on the primary shards, and the total is calculated as primary * (1+replicas). In other words, if you create an index with 3 primary shards and 2 replicas, you will have 9 total shards, not 5 or 6.
Elasticsearch offers some API endpoints to explore the state of your indices and shards. The <span class="inline-code"><pre><code>_cat</code></pre></span> APIs are helpful for human interaction. You can view your index states by visiting <span class="inline-code"><pre><code>/_cat/indices</code></pre></span>, which will show index names, primary shards and replicas. You can also inspect individual shard states and statistics by visiting <span class="inline-code"><pre><code>/_cat/shards</code></pre></span>. See example output below:
<div class="code-snippet w-richtext">
<pre><code fs-codehighlight-element="code" class="hljs language-javascript">$ curl -s https://user:password@bonsai-12345.bonsai.io/_cat/indices?v
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open images 1 0 0 0 130b 130b
green open videos 1 0 0 0 130b 130b
green open notes 1 0 0 0 130b 130b
$ curl -s https://user:password@bonsai-12345.bonsai.io/_cat/shards?v
index shard pri rep state docs store ip node
images 0 p STARTED 0 130b XXX.XXX.XXX.XXX Sugar Man
notes 0 p STARTED 0 130b XXX.XXX.XXX.XXX Sugar Man
videos 0 p STARTED 0 130b XXX.XXX.XXX.XXX Sugar Man</code></pre>
</div>
We’ve also made this easy by creating a live interactive console for you. Just visit your cluster’s dashboard console, chose <span class="inline-code"><pre><code>GET</code></pre></span> from the dropdown, and run <span class="inline-code"><pre><code>/_cat/indices?v</code></pre></span> or <span class="inline-code"><pre><code>/_cat/shards?v</code></pre></span>.
Short Answer: An index can have many shards, but any given shard can only belong to one index.
Long Answer: It is possible to “fake” multiple indices per shard by using “type” fields and index aliases. There is a section of the Elasticsearch Guide dedicated to this approach. Essentially, if you have several data models, you could put two or more into a single index instead of separate indices, and use aliases to perform automated filtering of queries.
The downside to this approach is that it requires a fair amount of work. Most frameworks and content management systems with integrated Elasticsearch clients will not take this approach by default. Developers will need to create and manage the index settings manually instead of relying on automated tools.
Answer: It depends. Because replicas can be added/removed at will, the real question is how many primary shards are needed. Increasing the number of primary shards for an index is one way to improve performance because it allows the query to be processed in parallel. But there are a lot of reasons to not have a kajillion primary shards.
Generally, we recommend that if you don’t expect data to grow significantly, then:
If you anticipate lots of growth – orders of magnitude within a short time – then the problem is a little more complicated. Take a read through the Ideal Elasticsearch Index post, especially the benchmarking section. Or shoot an email to support@bonsai.io.
Answer: There is an article dealing with this very subject: Reducing Shard Usage.
