From Python3.8 to Python3.10: Our Journey Through a Memory Leak

Summary

During an initiative to upgrade Python services from version 3.8 to 3.10, Lyft engineers encountered severe latency spikes and timeouts in one service within their test environment. Profiling DynamoDB queries revealed that gevent thread joins were taking up to 30 seconds while pod memory consumption climbed steadily. To isolate the issue, engineers used an internal tracemalloc-based profiler triggered via SIGUSR2 signals, temporarily disabling Gunicorn preload to prevent workers from terminating prematurely on signal receipt. Memory traces pointed to botocore and an incompatibility between weakref.finalize and gevent monkey patching in urllib3 version 1.26.16, which prevented connections from returning to the pool. Downgrading urllib3 to version 1.26.15 immediately resolved both the timeouts and the memory leak before a permanent fix arrived in gevent and urllib3 updates.

Context

Lyft upgraded Python services from version 3.8 to 3.10 due to version 3.8 reaching end-of-life, but one service experienced severe latency spikes, downstream timeouts, and growing memory usage across all pods.

Approach / What changed

Engineers analyzed latency traces, disabled Gunicorn preload so worker processes could handle SIGUSR2 signals, used an internal tracemalloc-based memory profiler, identified a deadlock in urllib3 1.26.16 under gevent, and downgraded urllib3 to 1.26.15 before later updating both gevent and urllib3.

Takeaways

• Running Gunicorn with preload enabled can prevent worker processes from registering custom signal handlers established during application initialization.
• An incompatibility between weakref.finalize and gevent monkey patching in urllib3 1.26.16 caused unreturned pool connections, deadlocks, timeouts, and memory leaks.
• Downgrading urllib3 to 1.26.15 immediately eliminated the memory leak and timeouts, while long-term stability was achieved with gevent v25.4.1 and urllib3 1.26.16+.

Related reading

• Grab · Jun 30, 2021

  Debugging High Latency Due to Context Leaks

  Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.

  Sourabh Suman

• Lyft · Jul 9, 2026

  From Day 1 to Production: Building Lyft’s Analytics & Rides Intelligence Assistant as Onboarding…

  Lyft developed Aria, an AI-powered assistant allowing approved employees to query ride analytics via natural language through a backend with LangGraph orchestration and SQL generation. However, Aria's user interface was limited to a Streamlit prototype that lacked standard service framework integration, Lyft authentication, and multi-user scalability. To resolve these limitations, a new engineer built a production-grade web client from scratch using Lyft's internal Node.js framework and Next.js as a three-week onboarding assignment. The implementation involved configuring Envoy routing, resolving authentication plugin incompatibilities, debugging service connectivity with Grafana, and managing interface state with an XState state machine. Aria was successfully hardened and launched into production, expanding natural language analytics access across the company.