Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms

Summary

Shopify’s Help Center serves more than 2 million users monthly, making search relevance important across setup, marketing, and troubleshooting queries. The post presents a three-step framework for evaluating new search algorithms against an existing system, combining data collection, offline metrics, and online A/B testing. It contrasts Vanilla Pagerank, which boosts articles by total views across searches, with Query-specific Pagerank, which uses click frequencies from historic searches containing the search term. Kafka event streams are modeled into search facts for near-real-time monitoring and experiment assignment, while Support-team annotations provide relevance labels for offline evaluation, with dataset staleness requiring recurring updates. In the reported A/B test, Query-specific Pagerank users clicked more, relied less on follow-up searches and later pages, and reached a lower average clicked rank; the algorithm replaced Vanilla Pagerank.

Takeaways

1. 1

   Vanilla Pagerank boosts articles by total views across searches, while Query-specific Pagerank uses click frequencies from historic searches containing the search term.

2. 2

   Kafka-based search facts support near-real-time metric evaluation, monitoring of product changes, real-time A/B-test assignment, and streaming feedback.

3. 3

   The Query-specific Pagerank A/B-test group clicked more results, used fewer follow-up searches, and was less likely to go beyond the first results page; the algorithm replaced Vanilla Pagerank.