# Context Deadlines and How to Set Them

[Grab](https://yomu.fyi/company/grab) · Michael Cartmell · Mar 11, 2019

**Type:** Explainer

## Summary

Microservice architectures operating under heavy network traffic require robust timeout handling to prevent slow or failing dependencies from causing cascading failures across services. Naive, static timeout configurations across a call chain often cause downstream components to waste compute effort on requests that upstream callers have already abandoned. To establish predictable timeout thresholds, engineers can align limits with service-level latency percentiles, pairing P99 limits with median latency retry allowances. Go's context package improves upon static network timeouts by propagating request-scoped deadlines and cancellation signals across service boundaries. This distributed context ensures downstream servers recognize remaining time budgets and terminate unneeded processing immediately when parent deadlines expire or callers manually cancel requests.

## Context

Grab's microservice architecture handles high network traffic where network issues cause API calls to fail or slow down, and poor timeout configurations risk cascading failure across services.

## Approach / What changed

Configuring timeouts using service-level agreement latency percentiles with retries, and using Go's context package to propagate deadlines and cancellation signals across API boundaries.

## Takeaways

- Setting timeout durations using latency percentiles like P99 combined with median latency retries establishes predictable request limits while reducing the failure rate.
- Go context propagation acts as a distributed timeout, allowing downstream services to calculate remaining processing time and avoid wasting compute on abandoned calls.
- A 'context cancelled' error originates from an upstream client's manual cancellation, whereas a 'context deadline exceeded' error indicates that a local or propagated deadline expired.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Go](https://yomu.fyi/topic/go), [Microservices](https://yomu.fyi/topic/microservices), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://engineering.grab.com/context-deadlines-and-how-to-set-them)