# Debugging High Latency Due to Context Leaks

[Grab](https://yomu.fyi/company/grab) · Sourabh Suman · Jun 30, 2021

**Type:** Incident / postmortem

## Summary

Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.

## Context

Market-Store, a real-time ML feature store at Grab, suffered latency spikes up to 2 seconds—violating its 200ms p99 SLA—due to unexplained memory growth during high-QPS periods.

## Approach / What changed

Engineers used Go's PPROF heap profiler to trace memory allocations back to child contexts generated by the Async Library and identified an uncancelled task context reference introduced in a merged merge request.

## Takeaways

- Child contexts created from non-empty parent contexts retain parent references and will not be garbage collected until explicitly cancelled.
- PPROF heap profiling in Go pinpoints in-use memory allocations and reveals leak sources that standard metrics and logs may miss.
- Calling CancelFunc removes parent references from child contexts, stops associated timers, and prevents context-driven memory leaks.

**Tags:** [Go](https://yomu.fyi/topic/go), [Observability](https://yomu.fyi/topic/observability), [Performance](https://yomu.fyi/topic/performance), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://engineering.grab.com/debugging-high-latency-market-store)