How to make Rust leak memory (also: how to make it stop)

Summary

fly-proxy, a Rust-based proxy, leaked memory because its obsolete custom HoneycombLayer retained child spans under a long-lived parent span. The post follows a request from browser input through DNS, TCP, TLS, HTTP/2, and asynchronous Rust components including an aya-built eBPF program, tokio, hyper, and tracing, before explaining how the leak surfaced. In the custom layer, every closed child span was added to its parent’s collection, so each request consumed a small amount of memory while a high-level async task remained alive. Removing that legacy code fixed the primary problem: memory usage barely exceeded 50MB during a load test, while remaining growth came from queues and latency data retained for processing rather than the identified leak.

Takeaways

1. 1

   A custom HoneycombLayer added each closed child span to its parent, causing a small memory leak for every request whose high-level parent span remained open.

2. 2

   Removing the legacy tracer implementation reduced memory usage to barely above 50MB during the load test.

3. 3

   Some post-fix allocation growth came from queues and latency data retained for queueing and percentile calculations; the source describes this as acceptable when bounded.