Multimodal Search vs Vector Search

Multimodal search is a type of vector search, not a separate thing.

Vector search, as a baseline

When most people talk about vector search, they mean text-to-text search based on meaning. A user types a query in the form of a sentence, a question, or a description, and the engine finds content that's semantically close to it, even if the exact words don't match. The text is converted into a vector (a mathematical representation of meaning), and the engine finds other vectors nearby. That's the core idea.

What multimodal adds

Multimodal search extends that same concept across different types of content, not just text. Images, for example, can be converted into vectors too. That means you can search for an image using a text query, or use one image to find similar images, because the engine is comparing mathematical representations rather than raw pixels or words.

A useful way to think about it: imagine every piece of content, such as the product photo, description, and the user query, as a point on a map. Multimodal search is what happens when those points come from different kinds of content but still end up in the same space, close enough to compare.

Think of searching for a coffee shop by name. That text string gets converted into coordinates, and the search engine uses those coordinates to measure distance between where you are and where the café is. The engine doesn't care whether the input was a sentence or a photo. It only cares about where the vectors land.

What this means in practice

For most Bonsai customers, vector search means text-based semantic search, and that's where to start. Multimodal search becomes relevant when your application involves images or other non-text content that users need to search across. This may mean you have visual product catalogs, image libraries, or media-rich applications.

If you're just getting started with vector search, see What is Vector Search. If your use case involves searching across different content types, contact us. Multimodal configuration involves additional setup beyond a standard vector search implementation, and we’d love to hear about what you’re building and how we can be a part of it!