Introducing: A search & relevancy assessment from engineers, not theorists. Learn more
Dec 7, 2021
Bonsai
•
•
5
min read
Want to find a needle in a haystack? Just drop the haystack into a search engine, type “needle” into the query box, and click Search.
Voila: needle found.
For many people, search engines operate in a sort of mystical black box. We enjoy the outcome of search engines without having the slightest idea of how they operate under the hood. The interesting thing is, the principles that govern search have been around for a very long time and don’t have to be confusing.
In this article, we’ve compiled 5 principles of search. This isn’t an exhaustive list. But hopefully, these points help developers, business leaders, and web users, to better understand the technology helping us navigate our digital lives.
As long as we’ve had books, we’ve had systems to organize knowledge and this might be an index or glossary at the back of a book. Libraries and collections have catalogs to help us quickly find authors, titles, and genres. As information has increased over the centuries, our methods for storing and sorting that information have become more critical, which brings us to the time of digitization we live in today.
Information has exploded because of the web. Every person with a computing device is creating new data. That means search is more necessary than it has ever been.
But search isn’t limited to companies like Google. Most individual applications need search engines because even small applications rely on so much data that search engines are necessary for a simple user experience.
With that said, it’s not optional to have a good search. Search has calibrated people to behave a certain way around digital tools and to expect a certain level of speed and quality from search experiences. We have been trained to expect certain behavior from technology by using Amazon, Siri, Google, etc.
This means eCommerce stores, B2B software, blogs, social media, and our favorite entertainment sites all rely on search engines to help users quickly find what they’re looking for.
As you create your website, digital product, or app, ask yourself a few questions about the information your users need.
Could someone reasonably scroll through all the users in our CRM? If not, then you need search.
If you’re an e-commerce store, could someone reasonably scroll through all your products? If not, then you need search.
In the early days of the web, Yahoo! was a literal directory of websites. The internet was so sparse in the mid- to late-1990s that you could scroll web pages to find the exact websites you were looking for. Obviously, this quickly became unreasonable as the internet became more widely used.
Yahoo! had to become a search engine instead.
Scaling is a lot easier when you know what you’re building toward. You won’t know all the answers at the beginning of your business journey. But if you generally know where your product is going, you can implement principles early into the creation of your product that helps you scale the searchability of your site or app with ease.
The foundation of your planning and prep can be distilled to a mathematical term.
In computer science, “logarithmic” is the opposite of exponential. Instead of something being squared, logarithmic data is the square root. Building a scalable search experience starts by organizing your data logarithmically.
The benefit here is that a search experience that’s built using logarithmic principles will sort through a decreasing-sized dataset with each new piece of data you feed it.
This is a complex explanation for something that’s easy to understand with an example.
If my query starts with the letter “B”, a logarithmic-based search engine can exclude all the indexes that don't begin with the same letter. Any index containing information that doesn’t start with “B” is ignored.
What’s the business case here? The data behind your index can double without doubling the time it takes to surface data. Your traffic and data can double, but your speed and performance are impacted only incrementally.
The ability to create a logarithmic index starts with a little term we call “precomputation.”
Want to provide more value through your search engine? Build a better index.
The visible part of search is running a query. But search isn’t just running a query. It’s building the index and organizing information based on select features.
Going back to our earlier example about books: a good index at the back of a book is usually keyword-driven. Someone has done the work to build a system that’s organized in such a way that it quickly gives people the relevant information they’re looking for.
So: The invisible side of the search process is the work that’s been done in advance to make the system fast and frictionless. Think of the old writing adage: the simpler and easier something is to read, the more time and energy it took to write.
It’s not too controversial when you think about it. To say that an algorithm has bias is to say it’s been designed. For better or worse, intentionally or unintentionally, bias is part of every platform because humans were involved in the creation process. Some social media accounts use intentional bias to keep users glued to their platforms. For example, a social media company might prioritize posts containing angry language to garner more attention and shares.
The problem arises when people think algorithms can be unbiased. Many folks think they can create these machines of perfect reasoning, devices that think black and white and aren’t subject to human subjectivity. But even at the very core of this idea is a major error: whoever designs the “objective” algorithm is biased toward its own definition of justice, truth, and falsehood.
Why is this a core principle of search? Because understanding the inevitable nature of bias gives us a better lens through which we can solve search-related problems.
Building a search experience is a subjective process intended to solve a human need—which means it’s okay that humans are involved in the process. In fact, this suggests that more, not fewer, humans should be involved in creating and vetting search algorithms.
Beginning with the understanding that bias is inherent to the creation of search algorithms allows designers to see search problems through a more accurate lens (and keeps people from gaming the system).
Even Google, the most recognized search engine globally, has added more subjective human assessment to its search process over the years. As they learned in the early 2010s, the more “objective” a search experience becomes, the more easily people can game the system once they understand the rules.
Every good ranking algorithm lives at the intersection of user behavior and business value. Both involve human bias. And that’s a good thing.
Schedule a free consultation to see how we can create a customized plan to meet your search needs.
