The Case for Multi-Index Search

The Case for Multiple Indexes

One of Elasticsearch's biggest strengths is the ease of creating many indexes. It's just a simple HTTP POST, and you're ready to start indexing documents!

Example: One index per environment

curl -X POST http://index.bonsai.io/blog-development
curl -X POST http://index.bonsai.io/blog-staging
curl -X POST http://index.bonsai.io/blog-production

Using many indexes for your application is very useful, particularly for scaling. When you have a lot of data, you can use many kinds of natural partitioning points in order to flexibly create many small indexes, which can be easier to scale than a single monolithic index.Some examples of why an app may need many indexes:

• Multiple environment support for keeping your development, staging, production and experimental data isolated.
• Multi-tenancy, in which you create an index per customer in multi-tenant applications.
• Temporal sharding, for applications which process a high volume of updates, particularly when recent documents are more valuable than older documents.
• Logical partitioning between models, to scale one model separately from another. Or within models, to keep related activity closely grouped.
• Hot reindexing, when you need to update your index analysis and mappings settings. Just create a new index and reindex into that in parallel without altering your existing index.

Example: Basic three day rolling-window temporal sharding

curl -X POST http://index.bonsai.io/events-day0
curl -X POST http://index.bonsai.io/events-day1
curl -X POST http://index.bonsai.io/events-day2

Searching Across Indexes

Storing your data into multiple indexes is only as useful as getting your documents back out of those indexes. Toward that end, Elasticsearch has excellent support for multi-index querying. Present since its earliest versions, a multi-index search provides multiple index names in the search request, separated by commas:

Multiple-index search with curl

curl http://index.bonsai.io/authors,articles,comments/_search

This is a useful feature for, e.g., the Tire Ruby client, which partitions separate models into their own indexes by default:

Multiple index search with Tire

Tire.index('authors,articles,comments').search do
  # ...
end

Now Supported on Bonsai

Because of the design of our systems, we don't always support 100% of Elasticsearch's API (see our new FAQ for more reasons why). However, as of today, you can now query your indexes with basic multi-index search syntax.

Reserved for future implementation work is the more advanced wildcard and operator syntax, introduced in Elasticsearch 0.19.8. Both are being worked on with a careful eye on ensuring proper security and privacy of the indexes in our clusters.

If you need this, or any other Elasticsearch feature, let us know! We are a small, focused development team and prioritize our plans based on input from our customers.