From Python3.8 to Python3.10: Our Journey Through a Memory Leak

Summary

During an initiative to upgrade Python services from version 3.8 to 3.10, Lyft engineers encountered severe latency spikes and timeouts in one service within their test environment. Profiling DynamoDB queries revealed that gevent thread joins were taking up to 30 seconds while pod memory consumption climbed steadily. To isolate the issue, engineers used an internal tracemalloc-based profiler triggered via SIGUSR2 signals, temporarily disabling Gunicorn preload to prevent workers from terminating prematurely on signal receipt. Memory traces pointed to botocore and an incompatibility between weakref.finalize and gevent monkey patching in urllib3 version 1.26.16, which prevented connections from returning to the pool. Downgrading urllib3 to version 1.26.15 immediately resolved both the timeouts and the memory leak before a permanent fix arrived in gevent and urllib3 updates.

Takeaways

1. 1

   Running Gunicorn with preload enabled can prevent worker processes from registering custom signal handlers established during application initialization.

2. 2

   An incompatibility between weakref.finalize and gevent monkey patching in urllib3 1.26.16 caused unreturned pool connections, deadlocks, timeouts, and memory leaks.

3. 3

   Downgrading urllib3 to 1.26.15 immediately eliminated the memory leak and timeouts, while long-term stability was achieved with gevent v25.4.1 and urllib3 1.26.16+.