The Challenge: Postgres couldn’t handle search
ActionSprout connects nonprofits and advocacy groups to tens of millions of people each day. ActionSprout customers must be able to sort through millions of records in near real-time. Search is a mission-critical part of their app, so the ActionSprout team found it imperative to find a good search partner to manage it.
“We had this situation where our data’s storage was fine, but when it came time to search it, filter it, and sort it, we really ran into trouble,” Nathan Carnes, ActionSprout’s Chief Product Officer, says. “Our application stores a lot of information about people — a significant amount of people. We’d been having a lot of challenges making sure that our customers could access that data as quickly and as reliably as they needed to.” At the time, ActionSprout was relying on a Postgres database to deliver their results. But with so much data, returning results could take as long as 15 seconds. Nathan knew that a better search engine was necessary. “I’ve used Elasticsearch in the past and it’s certainly the best thing on the market in terms of open-source search offerings,” Nathan explains. “We knew it could make our data searchable, sortable, and filterable very, very quickly at a very high scale.”
There was just one problem: managing an Elasticsearch cluster in-house wasn’t doable. With around ten employees, ActionSprout couldn’t afford to bring in an expert and host their own Elasticsearch cluster. ActionSprout needed a hosting solution that would not only handle their massive data storage requirements and the complexities of running an Elasticsearch cluster, but also help scale that cluster as their databases continued to grow. “We really wanted to identify a well-managed provider for that,” Nathan says. “One that could also help us out with guidance and advice.” That’s when Nathan decided to try Bonsai.
The Solution: Scalability paired with expert advice
Originally, Nathan found Bonsai through Heroku’s add-on marketplace. While there’s no shortage of Elasticsearch hosting providers, none of them had the reputation of Bonsai. “The decision to work with them initially came from their reputation, from asking around a little bit,” Nathan says. “Their pricing was really competitive and they also had a nice management interface.” Instead of jumping into a full Enterprise plan for Elasticsearch, Nathan opted to try Bonsai’s shared plan first.
After rebuilding their app to work with Elasticsearch, the ActionSprout team had their cluster had their cluster up and running on Bonsai almost immediately. “The entire process was so smooth from our end,” Nathan beams. “It was actually a big project for us, but having set all that up, tying into Bonsai’s platform was super easy.” Once the team saw Elasticsearch in action, they knew it was time to double down.
“When we started with them, we were treating them as a commodity provider,” Nathan explains. “But as it became clear that Elasticsearch was going to become a core part of our infrastructure and something that we needed to take seriously, that’s when we reached out to Bonsai and moved to a dedicated cluster. That’s when we began leaning on them not only for the servers, but also advice.”
Over the course of a year, Bonsai’s experts helped ActionSprout navigate through two major version upgrades. Meanwhile, ActionSprout’s data continued to grow to over double the original size, and Bonsai worked proactively to scale their clusters three times to fit the evolving demand.
Bonsai has been able to manage that scale flawlessly so far, and that’s a huge credit to them. That’s important for us. And to be able to go to them for advice and questions, they’ve been a huge asset there too.
But Bonsai wasn’t able to just provide advice, either. Nathan recalls one troublesome evening when their Elasticsearch cluster was being hammered by heavy traffic. “I reached out to them at 8:30 PST at night, and their support staff was able to add a new node to our cluster in about a half hour to help get us out of that situation,” Nathan says. “That was really incredible and it helped us out when we were in a tough spot after hours.”
Despite being such a complex technology, Nathan trusts that Bonsai has his back.
I do not worry about our very large, very high load Elasticsearch instance on a day to day basis. I have faith that it’s always going to work. And if there is an issue, Bonsai is going to help us work through it quickly. That peace of mind is so important.
The Results: Search as a strength, instead of a liability
Now, when searching through ActionSprout’s databases, clients are getting results at an average rate of less than 50 milliseconds. Before, with Postgres, that same search might take 15 seconds. Nathan has been saved the immense expense and headache of hiring an in-house expert to manage a self-hosted Elasticsearch cluster. “The real cost would have been over time as we’re maintaining those servers and trying to scale them,” Nathan explains. “Maintaining your own Elasticsearch cluster on an ongoing basis at the scale that we are at is a really substantial time commitment. The fact that we’re not worrying about that is saving us a lot of time each week.” What’s more, Bonsai has proven a capable and motivated ally that will continue growing to meet ActionSprout’s ever-evolving needs, letting Nathan focus his efforts elsewhere.
“I don’t spend very much time thinking about Elasticsearch, and that’s perfect. It’s less about time savings, and more about having confidence in that portion of our infrastructure and that Bonsai going to handle what we need it to. That’s incredibly valuable. ”
Nathan Carnes, Chief Product Officer, ActionSprout