Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

Summary

Engineers at Netflix required a unified, real-time view of service dependencies to navigate distributed architecture and improve incident troubleshooting. Traditional batch systems introduced stale data, so the team created a streaming-first platform backed by reactive streams and backpressure handling to ingest flow records from multi-region Kafka streams and Server-Sent Events without data loss. The architecture partitions data into physically separate graph and columnar storage layers covering eBPF network flows, IPC metrics, and distributed traces. Network flow ingestion relies on a three-stage distributed aggregation pipeline using consistent hashing to resolve network intermediaries into logical application connections. The resulting production system serves time-travel and topology queries with sub-second latency while continuously updating dependency views.

Takeaways

1. 1

   Reactive streams with backpressure propagation allow the ingestion pipeline to slow upstream stages and Kafka consumers during load spikes without crashing instances or dropping records.

2. 2

   Physical storage isolation across network flow graphs, IPC metric graphs, and Parquet trace storage enables independent optimization for differing throughput and query patterns.

3. 3

   Multi-stage aggregation with redistribution at each stage prevents single-instance bottlenecks caused by power-law traffic distributions.