The search renaissance is here (but the present is still medieval)

Google, the second most beloved brand in America, and the face not only of search but to a degree, all of the Internet, is looking more vulnerable than ever. Consumers are increasingly unhappy and tech industry leaders are increasingly agreeing – and funding alternatives. 

Search has become a hot topic. ML-powered search is pointing the way toward significantly more powerful search engines. Meanwhile, AI-based chat tools like ChatGPT (and even with recent hamstringing, Bing) present an orthogonal challenge, a search experience that resembles question and answer. 

And though Google faces the brunt of these challenges, search technologies of all kinds face them too. Search, often under-prioritized and suffering from backlogged improvements, is now receiving renewed attention. The question for developers is how best to use this attention and how to actually create the best search experience for their organizations and their users. 

Our humble suggestion is this: Even though the attention is partially the result of new, cutting-edge technologies, the most effective improvements actually lie with more foundational design decisions. Even if your primary goal is to pursue the potential of AI and ML, the best path toward the future is through the present. 

The Google paradigm is facing threats from multiple directions

Google search is a generational product, a success so enormous that all search technology lives in its shadow. Google is the homepage; “Google” is the verb; and if you look at most companies’ search bars and results, you’ll see echoes of Google's design. 

So, you need to be careful about predicting Google’s defeat. It’s not likely. But by studying the threats, whether Google succumbs to them or not, we can better see what users want out of search. 

Critical momentum against Google is building

Google search was a paradigm shift. But a consequence of paradigm shifts of this scale is that consumers and companies alike can normalize the paradigm and expect little beyond what’s already offered. 

Google has taken advantage of this normalization, adding more ads to search results over the years and, depending on who you ask, not iterating as much as they could. Parallel to this stagnation, companies with search functions have felt little pressure to improve. Most of these functions aren’t as good as Google but a general complacency has arisen, a sense that the status quo for search is “good enough.”

Closer research, however, reveals that criticism of Google search is more widespread than you might think. 

• In 2014, former engineer at Twitter Dan Luu wrote about how recent predictions that Lucene would soon surpass Google didn’t pan out.
• In 2020, co-founder of search engine Cue, Daniel Gross, wrote about new approaches to search that could defeat a dying Google.
• In 2022, Michael Seibel, partner at Y Combinator, tweeted about the need for new search tools (and Paul Graham followed on too).
• In 2022, independent search engine and AI researcher Dmitri Brereton wrote a post about the death of Google search.

Though many of these predictions didn’t pan out, they point to a groundswell of dissatisfaction that has culminated in the past year. 

Recent challenges to Google are significant, widespread, and orthogonal

Predictions and criticism don’t necessarily mean a lot. A big company like Google is always going to draw a lot of criticism. If anything, predictions of Google’s demise that prove false actually show Google’s strength. If everyone is unhappy but keeps using Google anyway, that demonstrates staying power. 

The difference with this wave of criticism is that it’s multidirectional and people have new options to choose from. You can break down this feedback into three categories:

• Significant. Many users are complaining about core search features, such as the ability to fulfill simple queries. In one tweet, a user complained about Google “excluding search words I've put in quotes” and Danny Sullivan, the Google search liaison, responded, writing that “We absolutely are not omitting quoted words from web searches. We're really really really not.” Whether Sullivan is right or not is beside the point – many users appear to have lost trust in Google’s basic functionality. 
• Widespread: Criticism ranges from end-users, as evidenced by viral tweets, and from respected experts (see the predictions in the previous section). This creates a feedback loop where experts might notice end-user dissatisfaction and end-users can feel validated by expert criticism. Eventually, consensus can arise and it can become a shared assumption that Google is no longer effective.
• Orthogonal: Many challenges are orthogonal to Google search. Google research (ironically) shows that nearly half of gen-z are using Tik Tok and Instagram to search instead of Google. Meanwhile, Bing, the biggest Google challenger, has already incorporated AI features, including a chat interface that users could find more effective than Google’s “ten blue links” approach.

Speculation is fun but there’s a big opportunity here for developers responsible for their own search features: You have more publicly available feedback than ever before and with it, developers can advocate for the improvements their search features have needed for a long time. 

The foundation is yet to be laid

Underneath the surface, search is a nest of contradictions. 

On the one hand, virtually all users expect search and expect search to be good; similarly, most apps that would benefit from search have it and display it prominently. But on the other hand, despite the prominence of search, improvements to it are often treated like tech debt – deprioritized fine-tunings that can be passed down the backlog. 

The result of these unresolved contradictions is a feature that is often merely “good enough,” meaning there’s a lot of latent potential that companies have yet to realize. With the renewed attention around search, developers might be able to allocate resources to search improvement but only if they know how it went wrong in the first place.

Teams that contribute to search are siloed

Silos are a notorious problem both because they can cause a lot of damage and because, despite recognition of the problem, silos can still emerge. 

Search is a great example. Many teams contribute to search but there are rarely processes in place to coordinate and direct that collaboration. 

• UI and UX designers are largely mimicking search designs from Google, Airbnb, and similar companies with robust search features.
• Operations engineers are often focused primarily on cost and resource optimization.
• Data scientists tend to be reading about cutting-edge search technologies but not all of that knowledge is applicable or even applied.

With silos like these, a lot of effort can be expended and little progress can be made, which can further disincentivize teams from prioritizing search. 

A counterexample is Twitter, which does have a dedicated search infrastructure team. In an interview on the Software Engineering Daily podcast, Twitter engineer Nico Tonozzi shared that every time they release a search improvement, they measure latency – both the averages and the high percentiles. By measuring latency, he says, they can find times when “one in a thousand queries takes three times as long.” Those bugs, according to him, are “really hard to track down but are important to do for cost saving and for user benefits.”

Companies needn’t be the size of Twitter and search needn’t be as important a feature to take the lesson that measuring changes can lead to significant benefits. 

Teams haven’t nailed the basics yet

Search features are often stalled because they reach a state of “good enough” without the involved teams knowing either how to improve search or knowing how search is failing. Core components of effective search are often missing and the metrics that would indicate how important it is to make those improvements are often missing too.

Sometimes, this is an issue of inheritance and prioritization. Teams will inherit an aging search feature and will only be encouraged to maintain it instead of improving it. If teams do decide to improve it, they can quickly become overwhelmed. Without the basics in place, even seemingly simple problems can cause a cascade of other issues. 

Jason Bosco, now co-founder and CEO at Typesense Search, had an experience like this while working at Dollar Shave Club. On the Changelog podcast, he shared that when he was initially developing the site’s search feature, it ended up being much more complex than he predicted. “We were just quite frustrated with how complicated it was,” Bosco said. The hardest parts, he said, were “scaling it and fine-tuning it.” 

The kinds of problems Bosco’s team ran into aren’t impossible to solve, of course, but if there aren’t metrics in place, there might not be a strong enough incentive to take them on.

No one owns search and few leaders prioritize it

More often than not, no single person is in charge of search. This is a result of the aforementioned silos, which make it hard to determine an owner, and a result of the absent basics, which make the process of finding an owner less appealing. 

A lack of ownership is a strong signal code quality is currently bad and may worsen. Microsoft research shows, for example, that code ownership correlates with software quality. Without an owner, code quality can suffer but worse, without an owner, there is no one to advocate for improvement. The connection between search and revenue is obscure, so without advocacy, the feature can languish.

The future is marginal (for now)

The cutting-edge of search technology – best exemplified by ML-powered search and AI chat tools – poses potentially significant improvements to search but until these technologies mature, they won’t be worth the investment. Further, until the teams in charge of search mature too, new technologies will remain out of reach even as they advance. 

Search improvements follow the pareto principle

The pareto principle states that (roughly) 80% of the benefits from a given effort will come from 20% of the work done during that effort. The proportion is directional and not literal, the lesson pointing to the idea that there’s an unequal relationship between inputs and outputs. 

The pareto principle applies to search because a disproportionate amount of benefits will come from improving the basics, whereas focusing on the more advanced ideas will likely earn much less payoff. Once you prioritize search and establish metrics that indicate what’s working and what’s not, you’ll likely find more things needed improvement than you realize and that fixing those things will have a greater payoff than you might predict. 

For example, a developer wrote into Scott Hanselman’s blog a few years ago, sharing that demonstrates the power of the basics. After a release caused a substantial slowdown for a product that uses ElasticSearch to process millions of documents and queries, the solution was as simple as “switching the word ‘bool’ to the word ‘or’.” 

The developer chalked this success up to taking the time to research ElasticSearch instead of optimizing what they were already comfortable with. In other words, they focused on the basics of search and in so doing, lived out the pareto principle with major benefits coming from relatively little effort.

The best part of focusing on the foundation, though, is that you can have your cake and eat it too. If you lay the foundation to better search first, you can set yourself up for successfully adopting and integrating new search technologies as they mature and emerge. 

There’s more potential than you might realize

As you work on the basics, you might also discover that there’s a surprisingly large amount of latent potential waiting to be realized. 

Google is one of the most generic products ever, performing more than one trillion searches per year to people across the world, across demographic, across industries, and across use cases. 

You aren’t Google. If you measure the quality of your search feature and study how your users feel about your search experience, you might find a multitude of ways to tune your search feature to their needs. 

One study, for example, focusing on how software developers use search, showed that “code related searching often requires more effort (e.g., time, result clicks, and query modifications) than general noncode search, which indicates code search performance with a general search engine is less effective.” Even if your end-users aren’t developers, a similar dynamic likely applies with a generic search feature requiring more effort than a finely tuned one. 

An instructive counterexample to the stagnancy we’ve talked about so far is digital asset management (DAM). In this industry, search is clearly associated with revenue because end-users are accessing the primary value of a given DAM tool via search. When we worked with Widen, for instance, we were able to reduce search downtime from 58 hours to zero. Other companies can achieve similar results if they realize the potential of improving search. 

You have to draw the Vitruvian Man before you can paint the Mona Lisa

It really feels like we’re on the precipice of entering a renaissance in search technology. The technology itself is advancing and people – from consumers to leaders – are both excited about the possibilities and dissatisfied with the status quo. 

But there’s a lesson to take from the renaissance itself: Leonardo DaVinci, the emblem of the renaissance and the exemplar of the “renaissance man,” painted the Mona Lisa in 1503. But before he could paint the Mona Lisa, he had to draw the Vitruvian Man – a diagram analyzing the proportions of the human body – in 1490. 

Just as DaVinci, and all artists, need to focus on anatomy and on the fundamentals before making great art, so do developers need to focus on the foundational elements of search before incorporating ML and AI. The basics come first not only because they’re most important but because once the basics are done, and only when they’re done, will you be ready for the future.