---
title: "Taming A Voracious Rust Proxy"
description: "Fly edge proxies began reporting elevated HTTP errors and sharply higher CPU utilization on several IAD hosts, with restarting fly-proxy providing only temporary relief. Profiling an affected process showed Rust tracing and repeated Future polls dominating samples while little time was spent in the kernel, pointing to an async busy-loop. Investigation narrowed the fully qualified polling type to a stack containing the proxy’s Duplex and Rustls’s TlsStream, whose mishandled Waker behavior after a TLS CloseNotify could spin when buffered data remained underneath. Tigris Data’s load test, involving tens of thousands of connections that sent small HTTP bodies and terminated early, triggered the condition; stopping the test and deploying the fix ended the spin-outs. The incident prompted plans to instrument spurious wakeups and reinforced that dependency updates require deliberate tracking, testing, and prioritization."
---

# Taming A Voracious Rust Proxy

[Fly.io](https://yomu.fyi/company/fly-io) · Feb 26, 2025

**Type:** Incident / postmortem

## Summary

Fly edge proxies began reporting elevated HTTP errors and sharply higher CPU utilization on several IAD hosts, with restarting fly-proxy providing only temporary relief. Profiling an affected process showed Rust tracing and repeated Future polls dominating samples while little time was spent in the kernel, pointing to an async busy-loop. Investigation narrowed the fully qualified polling type to a stack containing the proxy’s Duplex and Rustls’s TlsStream, whose mishandled Waker behavior after a TLS CloseNotify could spin when buffered data remained underneath. Tigris Data’s load test, involving tens of thousands of connections that sent small HTTP bodies and terminated early, triggered the condition; stopping the test and deploying the fix ended the spin-outs. The incident prompted plans to instrument spurious wakeups and reinforced that dependency updates require deliberate tracking, testing, and prioritization.

## Context

Several edge hosts in IAD triggered alerts for elevated fly-proxy HTTP errors and sharply increased CPU utilization. Restarting the proxy temporarily cleared the problem, but affected proxies repeatedly returned to the same state. The incident was ultimately associated with Tigris Data’s load testing and a Rustls TLS state-machine issue.

## Approach / What changed

Responders profiled an affected proxy, traced the busy-loop through the Future type and its I/O wrappers, and narrowed the behavior to Rustls’s TlsStream. The identified Waker-handling bug after TLS CloseNotify was fixed and deployed. The team also planned instrumentation to detect spurious wakeups directly.

## Takeaways

- Async Rust can busy-loop when an AsyncRead returns Ready without advancing its underlying state machine, or when a Pending poll incorrectly triggers a Waker.
- Rustls’s TlsStream mishandled its Waker when a TLS session received CloseNotify while buffered data remained on the underlying connection.
- Tigris Data’s test sent small HTTP bodies and terminated connections early, triggering the condition; after the fix was deployed, testing resumed without further spin-outs.

**Tags:** [Incident Response](https://yomu.fyi/topic/incident-response), [Rust](https://yomu.fyi/topic/rust), [TLS](https://yomu.fyi/topic/tls)

- Source: [Fly.io](https://fly.io/blog/taming-rust-proxy)
- Source URL: https://fly.io/blog/taming-rust-proxy
- Ingested by Yomu: 2026-09-03T01:26:47.190Z

[Read original post](https://fly.io/blog/taming-rust-proxy)