---
title: "Large-Scale Automated Dependency Analysis Across Uber's Service Mesh"
description: "Uber describes an automated method for cataloging dependency behavior across its service mesh, where nested asynchronous calls make failure attribution difficult. Because distributed traces are sampled and erroneous requests are rare, trace-based correlation can take too long; the system instead instruments yarpc middleware to observe every request and correlate inbound failures with outbound failures at service-and-endpoint granularity. An in-memory request record links each inbound request to its outbound calls, while emitted metrics capture caller and callee nodes and whether each errored. The resulting edge probability, calculated as caller failures divided by requests where the callee failed, classifies dependencies as fail-close at Pc >= 0.8, fail-open at Pc <= 0.2, or unknown between those thresholds. Uber says the initiative cataloged dependencies of critical APIs and indexed most fail-close anomalies, supporting reliability work and regression guardrails for Failover Architecture 2.0."
---

# Large-Scale Automated Dependency Analysis Across Uber's Service Mesh

[Uber](https://yomu.fyi/company/uber) · Sep 15, 2026

**Type:** Explainer

## Summary

Uber describes an automated method for cataloging dependency behavior across its service mesh, where nested asynchronous calls make failure attribution difficult. Because distributed traces are sampled and erroneous requests are rare, trace-based correlation can take too long; the system instead instruments yarpc middleware to observe every request and correlate inbound failures with outbound failures at service-and-endpoint granularity. An in-memory request record links each inbound request to its outbound calls, while emitted metrics capture caller and callee nodes and whether each errored. The resulting edge probability, calculated as caller failures divided by requests where the callee failed, classifies dependencies as fail-close at Pc >= 0.8, fail-open at Pc <= 0.2, or unknown between those thresholds. Uber says the initiative cataloged dependencies of critical APIs and indexed most fail-close anomalies, supporting reliability work and regression guardrails for Failover Architecture 2.0.

## Context

In Uber’s microservice architecture, a service API may have hundreds of direct and indirect dependencies. A dependency is modeled as a service-and-endpoint tuple or datastore, and classified as fail-close when its failure consistently causes the caller to fail, or fail-open otherwise. Sampling distributed traces is too slow for rapidly cataloging these relationships at service-mesh scale.

## Approach / What changed

The implementation uses yarpc inbound and outbound middleware. Inbound middleware assigns each request a unique ID and stores an in-memory record; outbound middleware records downstream calls and their errors. After application processing completes, inbound middleware emits metrics for each call containing caller and callee endpoints plus caller and callee error states. Aggregated metrics produce an edge-failure probability, with thresholds of 0.8 and 0.2 determining fail-close, fail-open, or unknown classifications.

## Takeaways

- The system correlates every served request’s inbound outcome with the outcomes of its outbound calls, avoiding the slow convergence of sparse distributed-trace sampling.
- An edge is classified as fail-close when Pc is at least 0.8, fail-open when Pc is at most 0.2, and unknown when Pc falls between those thresholds.
- Retries are excluded from the dependency-analysis observation: application-layer retries are prohibited, and the retry middleware runs after dependency-analysis outbound middleware.

## FAQ

### How does the system decide whether a dependency is fail-close?

It calculates Pc as the number of requests where both the callee and caller failed divided by all requests where the callee failed. Pc values of at least 0.8 are fail-close, values of at most 0.2 are fail-open, and intermediate values are unknown.

### Why use middleware and metrics instead of relying only on distributed traces?

Distributed traces are sampled because processing every request is prohibitively expensive, but errors are rare enough that sampled failures can take too long to observe. yarpc middleware can inspect inbound and outbound requests and emit correlation metrics for every request served.

### How are retries treated during dependency analysis?

The implementation ignores retries originating before the outbound middleware. Application-layer retries are not used, and a separate chained outbound middleware performs retries after dependency analysis has observed the outbound request status.

**Tags:** [Distributed Systems](https://yomu.fyi/topic/distributed-systems), [Microservices](https://yomu.fyi/topic/microservices), [Observability](https://yomu.fyi/topic/observability), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Uber](https://www.uber.com/us/en/blog/automated-dependency-analysis)
- Source URL: https://www.uber.com/us/en/blog/automated-dependency-analysis
- Ingested by Yomu: 2026-09-16T12:00:52.233Z

[Read original post](https://www.uber.com/us/en/blog/automated-dependency-analysis)