---
title: "Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms"
description: "Shopify’s Help Center serves more than 2 million users monthly, making search relevance important across setup, marketing, and troubleshooting queries. The post presents a three-step framework for evaluating new search algorithms against an existing system, combining data collection, offline metrics, and online A/B testing. It contrasts Vanilla Pagerank, which boosts articles by total views across searches, with Query-specific Pagerank, which uses click frequencies from historic searches containing the search term. Kafka event streams are modeled into search facts for near-real-time monitoring and experiment assignment, while Support-team annotations provide relevance labels for offline evaluation, with dataset staleness requiring recurring updates. In the reported A/B test, Query-specific Pagerank users clicked more, relied less on follow-up searches and later pages, and reached a lower average clicked rank; the algorithm replaced Vanilla Pagerank."
---

# Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Apr 1, 2021

**Type:** Tutorial

## Summary

Shopify’s Help Center serves more than 2 million users monthly, making search relevance important across setup, marketing, and troubleshooting queries. The post presents a three-step framework for evaluating new search algorithms against an existing system, combining data collection, offline metrics, and online A/B testing. It contrasts Vanilla Pagerank, which boosts articles by total views across searches, with Query-specific Pagerank, which uses click frequencies from historic searches containing the search term. Kafka event streams are modeled into search facts for near-real-time monitoring and experiment assignment, while Support-team annotations provide relevance labels for offline evaluation, with dataset staleness requiring recurring updates. In the reported A/B test, Query-specific Pagerank users clicked more, relied less on follow-up searches and later pages, and reached a lower average clicked rank; the algorithm replaced Vanilla Pagerank.

## Context

The Help Center contains thousands of articles, and broad popularity signals from Vanilla Pagerank can surface non-relevant results. Because algorithm changes may improve some queries while worsening others, the team needed a data-based way to evaluate ranking effectiveness before and after production changes.

## Approach / What changed

The framework collects user interactions through Kafka event streams and relevance judgments through Support-team annotation. It evaluates algorithm variants offline with curated labels and metrics including Mean Average Precision and Normalized Discounted Cumulative Gain, then compares production behavior through online metrics and A/B tests.

## Takeaways

- Vanilla Pagerank boosts articles by total views across searches, while Query-specific Pagerank uses click frequencies from historic searches containing the search term.
- Kafka-based search facts support near-real-time metric evaluation, monitoring of product changes, real-time A/B-test assignment, and streaming feedback.
- The Query-specific Pagerank A/B-test group clicked more results, used fewer follow-up searches, and was less likely to go beyond the first results page; the algorithm replaced Vanilla Pagerank.

**Tags:** [Kafka](https://yomu.fyi/topic/kafka), [Search](https://yomu.fyi/topic/search), [Testing](https://yomu.fyi/topic/testing)

- Source: [Shopify](https://shopify.engineering/evaluating-search-algorithms)
- Source URL: https://shopify.engineering/evaluating-search-algorithms
- Ingested by Yomu: 2026-08-31T01:09:54.894Z

[Read original post](https://shopify.engineering/evaluating-search-algorithms)