# From Python3.8 to Python3.10: Our Journey Through a Memory Leak

[Lyft](https://yomu.fyi/company/lyft) · Jay Patel · Dec 15, 2025

**Type:** Incident / postmortem

## Summary

During an initiative to upgrade Python services from version 3.8 to 3.10, Lyft engineers encountered severe latency spikes and timeouts in one service within their test environment. Profiling DynamoDB queries revealed that gevent thread joins were taking up to 30 seconds while pod memory consumption climbed steadily. To isolate the issue, engineers used an internal tracemalloc-based profiler triggered via SIGUSR2 signals, temporarily disabling Gunicorn preload to prevent workers from terminating prematurely on signal receipt. Memory traces pointed to botocore and an incompatibility between weakref.finalize and gevent monkey patching in urllib3 version 1.26.16, which prevented connections from returning to the pool. Downgrading urllib3 to version 1.26.15 immediately resolved both the timeouts and the memory leak before a permanent fix arrived in gevent and urllib3 updates.

## Context

Lyft upgraded Python services from version 3.8 to 3.10 due to version 3.8 reaching end-of-life, but one service experienced severe latency spikes, downstream timeouts, and growing memory usage across all pods.

## Approach / What changed

Engineers analyzed latency traces, disabled Gunicorn preload so worker processes could handle SIGUSR2 signals, used an internal tracemalloc-based memory profiler, identified a deadlock in urllib3 1.26.16 under gevent, and downgraded urllib3 to 1.26.15 before later updating both gevent and urllib3.

## Takeaways

- Running Gunicorn with preload enabled can prevent worker processes from registering custom signal handlers established during application initialization.
- An incompatibility between weakref.finalize and gevent monkey patching in urllib3 1.26.16 caused unreturned pool connections, deadlocks, timeouts, and memory leaks.
- Downgrading urllib3 to 1.26.15 immediately eliminated the memory leak and timeouts, while long-term stability was achieved with gevent v25.4.1 and urllib3 1.26.16+.

**Tags:** [Incident Response](https://yomu.fyi/topic/incident-response), [Migrations](https://yomu.fyi/topic/migration), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://eng.lyft.com/from-python3-8-to-python3-10-our-journey-through-a-memory-leak-1fd9b43cc01e)