How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC…

Summary

Netflix designed a query execution serving layer for its Real-Time Distributed Graph to power sub-100ms responses across diverse graph traversal workloads spanning billions of nodes and edges. To prevent compounding network delays in multi-hop queries, the engine uses a breadth-first traversal model that batches entity lookups across entire frontier levels rather than tracing paths depth-first. The entire serving architecture relies on asynchronous composition across small thread pools of 16 to 24 threads, ensuring no thread blocks while waiting on remote storage or enrichment I/O. Selective caching via EVCache targets stable properties with volatility-matched TTLs, producing 70 to 80 percent cache hit rates and reducing backend storage calls by three to four times. A layered filtering hierarchy pushes depth and edge limits directly to the storage tier, eliminating bespoke code changes while isolating system resources against excessive fan-out.

Context

Netflix required a fast, flexible serving layer for its Real-Time Distributed Graph (RDG) to deliver sub-100ms query responses over billions of evolving nodes and edges. The system needed to support diverse workloads, balancing shallow, wide queries that cause massive I/O fan-out with deep, narrow multi-hop queries where sequential network hops create significant latency overhead.

Approach / What changed

The team designed the query layer around breadth-first graph traversal, batching network lookups across frontier levels rather than following sequential depth-first paths. They implemented an async-first execution pipeline across small dedicated thread pools of 16 to 24 threads total, integrated selective caching via EVCache with TTLs aligned to data volatility, and introduced a layered override system for early filtering and per-hop limits.

Takeaways

• Asynchronous composition enabled the serving layer to handle thousands of concurrent requests across only 16 to 24 threads without blocking on I/O, dramatically lowering infrastructure costs.
• Selective caching with EVCache matched TTLs to property volatility and skipped near-expiry nodes, achieving a 70 to 80 percent hit rate and reducing storage calls by 3 to 4 times.
• Breadth-first traversal batches lookups across frontier levels, replacing sequential network call chains with bounded parallel rounds to maintain low multi-hop query latency.

Related reading

• Netflix · Jul 13, 2026

  Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

  Engineers at Netflix required a unified, real-time view of service dependencies to navigate distributed architecture and improve incident troubleshooting. Traditional batch systems introduced stale data, so the team created a streaming-first platform backed by reactive streams and backpressure handling to ingest flow records from multi-region Kafka streams and Server-Sent Events without data loss. The architecture partitions data into physically separate graph and columnar storage layers covering eBPF network flows, IPC metrics, and distributed traces. Network flow ingestion relies on a three-stage distributed aggregation pipeline using consistent hashing to resolve network intermediaries into logical application connections. The resulting production system serves time-travel and topology queries with sub-second latency while continuously updating dependency views.

  Netflix Technology Blog

• Meta · Jul 1, 2026

  Meta’s AI Storage Blueprint at Scale

  Meta redesigned its BLOB-storage architecture on top of the foundational Tectonic block layer to eliminate GPU stalls and accelerate AI training workflows. The legacy storage system suffered from multi-layered metadata lookups, cross-region latency, and dataplane proxy bottlenecks that conflicted with the millisecond access requirements of flash-based AI clusters. To resolve these issues, the team collapsed disparate metadata into a single flat schema backed by ZippyDB for O(1) path lookups and introduced a fat client SDK capable of streaming data directly from storage servers. The updated stack also leverages spare GPU host memory as a distributed data cache alongside distributed read-plan caches to handle severe traffic spikes during checkpointing and model loading. These protocol and architectural changes deliver predictable latencies, reduce power consumption, and maintain high throughput during distributed training.

  By Sidharth Bajaj, Venkatraghavan Srinivasan

• Slack · May 28, 2026

  Slack AI: The Path to Multi-Cloud

  Slack evolved its Slack AI serving infrastructure across multiple phases to handle enterprise LLM workloads reliably and securely. The initial deployment on AWS SageMaker provided zero-knowledge escrow VPC isolation and FedRAMP compliance, but engineers faced scaling latency, GPU scarcity, and significant operational overhead. Slack then migrated live traffic to Amazon Bedrock to leverage managed Model Units and eliminate model release lag without customer-facing incidents. However, fixed Provisioned Throughput commitments and regional peak traffic variations created persistent underutilization challenges. Consequently, Slack expanded into a multi-cloud orchestration architecture that normalizes disparate provider APIs, integrates unified cross-cloud telemetry, and routes traffic dynamically around latency spikes and outages.

  Shaurya Kethireddy

• Github · May 14, 2026

  From latency to instant: Modernizing GitHub Issues navigation performance

  GitHub Issues addressed navigation latency by shifting workloads to the client using a local-first, stale-while-revalidate architecture. To evaluate perceived delays, the team tracked user transitions through Highest Priority Content thresholds, aiming for instant rendering in under 200 milliseconds. The core implementation added a persistent client-side cache using IndexedDB, paired with a synchronous in-memory tier to serve hot issue payloads without asynchronous overhead. A selective preheating mechanism resolves missing cache entries for high-intent links without overburdening backend capacity with redundant requests. Following broad rollout, the proportion of instant React soft navigations increased from 4% to approximately 22%, yielding an overall cache-hit ratio of around 33%.

  Natalie Guevara