+++
title = "Reducing Shard Usage"
weight = 33
updated = 2023-07-07
aliases = ["/docs/reducing-shard-usage", "/docs/managed-search/elasticsearch-and-opensearch/cluster-management/resource-management/reducing-shard-usage/", "/docs/cluster-management/resource-management/reducing-shard-usage/"]

[extra]
weight = 51
+++

A shard is the basic unit of work in Elasticsearch. If you haven't read the
Shards section
in [Elasticsearch Core Concepts](@/docs/platform/core-concepts/index.md),
that would be a good place to
start. Bonsai meters on the total number of shards in a cluster. That means both
primary and replica shards count
towards the limit.

The relationship between shard scheme and your cluster's usage can sometimes not
be readily apparent. For example, if
you have an index with a 3x2 sharding scheme (3 primaries, 2 replicas), that's
not 5 or 6 shards, it's 9. If this is
confusing, read our [Shard Primer](@/docs/platform/shard-primer/index.md)
documentation for some nice illustrations.

Managing shards is a basic skill for any Elasticsearch user. Shards carry system
overhead and potentially stale data.
Keeping your cluster clean by pruning old shards can both improve performance
and reduce your server costs.

In this guide we cover a few different ways of reducing shard usage:

* What is a Shard Overage?
* Delete Unneeded Indices
* Use a Different Sharding Scheme
* Reduce replication
* Data Collocation
* Upgrade the Subscription

This guide will make frequent references to the Elasticsearch API using the
command line tool `curl`. Interacting with
your own cluster can also be done via `curl`, or via a web browser. You can also
use the
interactive [Bonsai Console](@/docs/features/console/index.md)
in
your [cluster dashboard](@/docs/features/cluster-overview/index.md).

## What is a Shard Overage?

A shard overage occurs when your cluster has more shards than the subscription
allows. In practice, this usually means
one of three things:

1. Extraneous indices.
2. Sharding scheme is not optimal.
3. Replication too high

It is also possible that the cluster and its data are already configured
according to best practices. In that case, you
may need to get creative with aliases and data collocation in order to remain on
your current subscription.

## Delete Unneeded Indices

There are some cases where one or more indices are created on a cluster for
testing purposes, and are not actually being
used for anything. These will count towards the shard limits; if you're getting
overage notifications, then you should
delete these indices.

There are also some clients that will use aliases to roll out changes to
mappings. This is a really nice feature that
allows for zero-downtime reindexing and immediate rollbacks if there's a
problem, but it can also result in lots of
stale indices and data accumulating in the cluster.

To determine if you have extraneous indices, use the `/_cat/indices` endpoint to
get a list of all the indices you have
in your cluster:

```
GET /_cat/indices
green open test1 1 1 0 0 318b 159b
green open test2 1 1 0 0 318b 159b
green open test3 1 1 0 0 318b 159b
green open test4 1 1 0 0 318b 159b
green open test5 1 1 0 0 318b 159b
green open test6 1 1 0 0 318b 159b
```

If you see indices that you don't need, you can simply delete them:

```
# Delete a single index:
DELETE /test1

# Delete a group of indices:
DELETE /test2,test3,test4,test5
```

## Use a Different Sharding Scheme

It's possible that for some reason one or more indices were created with far
more shards than necessary. For example, a
check of `/_cat/indices` shows something like this:

```
GET /_cat/indices
green open test1 5 2 0 0 318b 159b
green open test2 5 2 0 0 318b 159b
```

That 5x2 shard scheme results in 15 shards per index, or 30 total for this
cluster. This is probably really
overprovisioned for these indices. Choosing a more conservative shard scheme of
1x1 would reduce this cluster's usage
from 30 shards down to 4.

Unfortunately, the number of primary shards can not be changed once an index has
been created. To fix this, you will
need to manually create a new index with the desired shard scheme and reindex
the data. If you have not
read [The Ideal Elasticsearch Index](https://docs.bonsai.io/article/109-ideal-elasticsearch-index),
it has some really nice
information on capacity planning and sizing. Check out the sections
on Intelligent Sharding
and Benchmarking for some
tips on what scheme would make more sense
for your particular use case.

## Reduce replication

Multitenant subscription plans (Hobby and Standard tiers) are required to have
at least 1 replica shard in use per
index. Indices on these plans cannot have a replica count of 0.

For most use-cases, a single replica is perfectly sufficient for redundancy and
load capacity. If any of your indices
have been created with more than one replica, you can reduce it to free up
shards. An index with more than one replica
might look like this:

```
GET /_cat/indices?v
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open test1 5 2 0 0 318b 159b
green open test2 5 2 0 0 318b 159b
```

Note that the `rep` column has a 2? That means there are actually 3 copies of
the data: one primary shard, and its two
replicas. Replicas are a multiplier against primary shards, so if an index has a
5×2 configuration (5 primary shards
with 2 replicas), reducing replication to 1 will free up five shards, not just
one. See
the [Shard Primer](@/docs/platform/shard-primer/index.md) for more details.

Fortunately, reducing the replica count for the index is a small JSON body to
the `_settings` endpoint:

```
PUT /test1,test2/_settings -d '{"index":{"number_of_replicas":1}}'
{"acknowledged":true}

GET /_cat/indices
green open test1 5 1 0 0 318b 159b
green open test2 5 1 0 0 318b 159b
```

That simple request shaved 10 shards off of this cluster's usage.

### Replication == Availability and Redundancy

It might seem like a good money-saving idea to simply set all replicas to 0 so
as to fit as many indices into your
cluster as possible. However, this is not advisable. This means that your
primary data has no live backup. If a node in
your cluster goes offline, data loss is basically guaranteed.

Data can be restored from a snapshot, but this is messy and not a great failover
plan. The snapshot could be an hour or
more old, and any updates to your data since then either need to be reindexed or
are lost for good. Additionally, the
outage will last much longer.

Having replication of at least 1 mitigates against all these problems.

## Data Collocation

Another solution for reducing shard usage involves using aliases and custom
routing rules to collocate different data
models onto the same group of shards.

What is data collocation? Many Elasticsearch clients use an index per model
paradigm as the default for data
organization. This is analogous to, say, a Postgres database with a table for
each type of data being indexed.

Sharding this way makes sense most of the time, but in some rare cases users may
benefit from putting all of the data
into a single namespace. In the Postgres analogy, this would be like putting all
of the data into a single table instead
of a table for each model. An attribute (i.e., table column) is then used to
filter searches by class.

For example, you might have a cluster that has three indices: `videos`, `images`
and `notes`. If each of these has a
conservative 1x1 sharding scheme, it would require 6 shards. But this data could
potentially be compacted down into a
single index, `production`, where the mapping has a `type` field of some kind to
indicate whether the document belongs
to the `video`, `image` or `note` class.

The latter configuration with the same 1x1 scheme would only require two
shards (one primary, one replica) instead of
six:

{{ image(src="replication-overview.png", alt="Comparison of replication use",
automatically_determine_ratio=true) }}

There are several major downsides to this approach. One is that field name
collisions become an issue. For example, if
there is a field called `published` in two models, but one is defined as a
boolean and the other is a datetime, it will
create a conflict in the mapping. One will need to be renamed.

Another downside is that it is a pretty large refactor for most users, and may
be more trouble than simply upgrading the
plan. Overriding the default behavior in the application's Elasticsearch client
may require forking the code and relying
on other hacks/workarounds.

There are others. Data collocation is mentioned here as a possibility, and one
that only works for certain users. It is
by no means a recommendation.

## Upgrade the Subscription

If you find that you're unable to reduce shards through the options discussed
here, then you will need to upgrade to the
next plan.