# Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

[Netflix](https://yomu.fyi/company/netflix) · Netflix Technology Blog · Jul 13, 2026

**Type:** Problem & solution

## Summary

Engineers at Netflix required a unified, real-time view of service dependencies to navigate distributed architecture and improve incident troubleshooting. Traditional batch systems introduced stale data, so the team created a streaming-first platform backed by reactive streams and backpressure handling to ingest flow records from multi-region Kafka streams and Server-Sent Events without data loss. The architecture partitions data into physically separate graph and columnar storage layers covering eBPF network flows, IPC metrics, and distributed traces. Network flow ingestion relies on a three-stage distributed aggregation pipeline using consistent hashing to resolve network intermediaries into logical application connections. The resulting production system serves time-travel and topology queries with sub-second latency while continuously updating dependency views.

## Context

Engineers at Netflix needed a unified, real-time view of service dependencies across distributed architecture to troubleshoot incidents faster and understand blast radius, whereas traditional batch-processing approaches yielded hours-old or day-old dependency maps.

## Approach / What changed

Netflix implemented a streaming-first architecture with backpressure handling across separate storage layers for eBPF network flows, IPC metrics, and distributed traces. A three-stage distributed aggregation pipeline ingests multi-region Kafka streams, batches flow logs into 5-minute windows, and resolves intermediate hops into logical service dependencies.

## Takeaways

- Reactive streams with backpressure propagation allow the ingestion pipeline to slow upstream stages and Kafka consumers during load spikes without crashing instances or dropping records.
- Physical storage isolation across network flow graphs, IPC metric graphs, and Parquet trace storage enables independent optimization for differing throughput and query patterns.
- Multi-stage aggregation with redistribution at each stage prevents single-instance bottlenecks caused by power-law traffic distributions.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Kafka](https://yomu.fyi/topic/kafka), [Microservices](https://yomu.fyi/topic/microservices), [Observability](https://yomu.fyi/topic/observability), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://netflixtechblog.com/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-f4b792f3f0d8)