In the dynamic world of digital commerce, where consumer expectations evolve at lightning speed, the humble search bar has become a battleground for user engagement and loyalty.
For Ibotta, a pioneer in the cashback rewards space, ensuring that its millions of “Savers” could effortlessly discover and activate relevant offers was not just a feature – it was the core of its mission to “Make Every Purchase Rewarding.”
Yet, even a company built on innovation found its homegrown search system struggling to keep pace with the nuanced demands of modern users.
The challenge was stark: traditional keyword matching was no longer enough.
Savers were increasingly using natural language, conversational phrases, and even misspellings.
They expected an intuitive experience that understood intent, not just exact terms.
Ibotta’s legacy system, while functional, presented clear limitations in semantic relevance, contextual understanding, flexibility to integrate new offers, and the ability to rapidly iterate on improvements.
The opportunity, as the company saw it, was to transform search from a mere utility into a powerful discovery engine, enhancing value for both Savers and the brands they served.
The catalyst for this ambitious overhaul emerged not from a top-down mandate, but from the fertile ground of an internal hackathon.
A cross-functional team, inspired by insights from the Databricks Data + AI Summit, recognized the potential of Databricks Vector Search.
In a remarkable three-day sprint, they conjured a working proof-of-concept, demonstrating semantically relevant results by leveraging Ibotta’s vast offer catalog and multiple embedding models.
This swift success not only clinched first place in the hackathon but ignited crucial internal buy-in, propelling the prototype towards full-scale production.
The journey from proof-of-concept to a robust production system, however, was far from a straight line.
Initial optimism was met with a dose of reality.
Early production tests, involving a small percentage of Savers, revealed a disheartening truth: engagement metrics, including clicks, unlocks, and activations, actually decreased, particularly among the most active users.
It was a critical juncture.
While the underlying technical performance of the Vector Search solution was promising – offering faster response times, a simpler mental model, and greater flexibility – the initial user-facing results were a clear signal that raw technological prowess alone wasn’t enough.
This setback forced Ibotta to confront the limitations of traditional A/B testing for such a complex system.
The sheer number of variables – from embedding models and text combinations to hybrid search settings and reranking options – made a purely iterative A/B approach inefficient and impractical.
The solution was an ingenious one: a bespoke semantic evaluation framework.
This sophisticated environment tracked over 50 online and offline metrics, marrying standard information retrieval measures like Mean Reciprocal Rank with real-world engagement signals.
Crucially, Ibotta implemented an “LLM-as-a-judge” mechanism, allowing for rapid data labeling and quality scoring of query-result pairs.
This innovative approach transformed their development process, enabling rapid, data-driven decisions and high-confidence validation of improvements before exposing them to users.
With this powerful framework in place, Ibotta embarked on an exhaustive exploration of off-the-shelf models.
While some showed technical promise and yielded modest improvements, they ultimately proved insufficient.
The inability to train models on Ibotta’s unique, proprietary offer catalog, the difficulty in evolving models alongside business changes, and concerns over long-term API availability and performance from external providers were significant drawbacks.
It became clear that to truly optimize the search experience, a custom solution was paramount.
This realization led to the pivotal decision to fine-tune an embedding model specifically tailored to Ibotta’s data and Saver behavior.
Leveraging millions of labeled search interactions collected through their LLM-as-a-judge process, Ibotta embarked on a methodical fine-tuning journey.
They meticulously engineered training datasets, including synthetic data and “hard negatives” (human-verified inappropriate matches), and optimized hyperparameters within a serverless AI Runtime environment.
The results were astounding: their top-performing fine-tuned model outperformed the best third-party baseline by a remarkable 20% in synthetic evaluations.
The technical rigor and iterative process paid off handsomely.
In the final production test, deploying their custom fine-tuned model, Ibotta witnessed a dramatic transformation of their search experience.
Offer unlocks from search results soared by 14.8%, a leading indicator of downstream redemptions and revenue.
Engaged users increased by 6%, signifying greater value and action within the search experience.
Engagement on high-value bonus offers jumped by 15%, directly benefiting brand and retail partners.
Perhaps most impactful for user satisfaction, searches returning zero results plummeted by 72.6%, and the number of users encountering such frustrating experiences dropped by 60.9%.
Beyond user-facing gains, the system itself became leaner, boasting 60% lower latency.
Building on this powerful foundation, Ibotta further innovated with enhancements like “Query Transformation“, dynamically enriching vague or niche queries, and “Multi-Search“, fanning out generic terms into parallel, related searches.
These intelligent additions ensure a comprehensive, lightning-fast, and highly relevant search experience.
Ibotta’s journey stands as a powerful testament to the strategic importance of data and AI in modern business.
Their story underscores several critical lessons: the value of a quick proof-of-concept, the necessity of a tailored, robust evaluation framework, the wisdom of exhausting off-the-shelf options before committing to fine-tuning, the irreplaceable value of collecting high-quality proprietary data, and the accelerating power of deep collaboration.
As Ibotta now looks to scale this refined search solution to its Ibotta Performance Network, bringing improved offer discovery to millions more shoppers across its publisher network, its experience serves as a compelling blueprint for any enterprise seeking to truly “make every search rewarding.”
