Debugging High Latency Due to Context Leaks

Summary

Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.

Takeaways

1. 1

   Child contexts created from non-empty parent contexts retain parent references and will not be garbage collected until explicitly cancelled.

2. 2

   PPROF heap profiling in Go pinpoints in-use memory allocations and reveals leak sources that standard metrics and logs may miss.

3. 3

   Calling CancelFunc removes parent references from child contexts, stops associated timers, and prevents context-driven memory leaks.