Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms

Summary

Shopify’s Help Center serves more than 2 million users monthly, making search relevance important across setup, marketing, and troubleshooting queries. The post presents a three-step framework for evaluating new search algorithms against an existing system, combining data collection, offline metrics, and online A/B testing. It contrasts Vanilla Pagerank, which boosts articles by total views across searches, with Query-specific Pagerank, which uses click frequencies from historic searches containing the search term. Kafka event streams are modeled into search facts for near-real-time monitoring and experiment assignment, while Support-team annotations provide relevance labels for offline evaluation, with dataset staleness requiring recurring updates. In the reported A/B test, Query-specific Pagerank users clicked more, relied less on follow-up searches and later pages, and reached a lower average clicked rank; the algorithm replaced Vanilla Pagerank.

Context

The Help Center contains thousands of articles, and broad popularity signals from Vanilla Pagerank can surface non-relevant results. Because algorithm changes may improve some queries while worsening others, the team needed a data-based way to evaluate ranking effectiveness before and after production changes.

Approach / What changed

The framework collects user interactions through Kafka event streams and relevance judgments through Support-team annotation. It evaluates algorithm variants offline with curated labels and metrics including Mean Average Precision and Normalized Discounted Cumulative Gain, then compares production behavior through online metrics and A/B tests.

Takeaways

• Vanilla Pagerank boosts articles by total views across searches, while Query-specific Pagerank uses click frequencies from historic searches containing the search term.
• Kafka-based search facts support near-real-time metric evaluation, monitoring of product changes, real-time A/B-test assignment, and streaming feedback.
• The Query-specific Pagerank A/B-test group clicked more results, used fewer follow-up searches, and was less likely to go beyond the first results page; the algorithm replaced Vanilla Pagerank.

Related reading