# The Data Canary: How Netflix Validates Catalog Metadata

[Netflix](https://yomu.fyi/company/netflix) · Netflix Technology Blog · Jun 19, 2026

**Type:** Problem & solution

## Summary

A manual mitigation action during an incident corrupted a data feed for a subset of titles, causing playback issues and catalog service failures that existing code canary systems failed to catch. To protect streaming reliability, Netflix built an automated data canary system that validates transformed catalog metadata prior to publication. The architecture utilizes a dedicated orchestrator alongside permanent baseline and canary service clusters to coordinate validation using real production traffic. By leveraging custom chaos experiment thresholds, sticky session affinity, and Starts Per Second playback metrics, the system detects regressions in under ten minutes and blocks publication automatically. Controlled failure injection experiments routing approximately 0.2% of global traffic confirmed that issues could be identified in 2.5 to 4 minutes.

## Context

A manual mitigation during an incident corrupted a catalog data feed by rendering it empty for a subset of titles. Missing metadata broke manifest generation and playback, but existing code canaries could not detect the failure because no code or configuration had changed.

## Approach / What changed

Netflix implemented a dedicated Data Canary Orchestrator pattern with continuous baseline and canary clusters in a canary region. The system routes roughly 0.2% of production traffic using sticky canaries, runs chaos experiments using Starts Per Second (SPS) as the primary behavioral metric, tunes custom detection thresholds, and blocks bad data from publishing within a 10-minute cycle.

## Takeaways

- Starts Per Second (SPS) proved more reliable than latency or error rates for detecting catalog corruption because data errors do not always trigger application errors.
- Shadow traffic was insufficient because it could not simulate the complete playback lifecycle across multiple services and domains.
- Sticky canaries apply session affinity to keep user traffic on either the baseline or canary cluster, avoiding cross-contamination from concurrent chaos experiments.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Observability](https://yomu.fyi/topic/observability), [Reliability](https://yomu.fyi/topic/reliability), [Streaming](https://yomu.fyi/topic/streaming), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://netflixtechblog.com/the-data-canary-how-netflix-validates-catalog-metadata-18b699d58e36)