---
title: "parking_lot: ffffffffffffffff..."
description: "Fly.io’s fly-proxy, a Rust-based Anycast router, suffered lockups while managing rapidly changing routing state across thousands of servers. An earlier outage came from an if let expression that kept a read lock through both branches, while later regionalization and lazy loading increased write contention in the Catalog. The investigation traced the new failures to parking_lot’s try_write_for wakeup path: after a waiting writer timed out, it attempted to clear WRITER and WRITER_PARKED together even though one bit had already been cleared, and wrapping arithmetic produced the corrupted lock value 0xFFFFFFFFFFFFFFFE. The parking_lot team fixed the defect, after which the lockups did not recur; the team also audited Catalog locking, removed if let patterns and RAII lock guards, and added lock-timing metrics, labeled slow-write logs, and holder tracking."
---

# parking\_lot: ffffffffffffffff...

[Fly.io](https://yomu.fyi/company/fly-io) · May 28, 2025

**Type:** Incident / postmortem

## Summary

Fly.io’s fly-proxy, a Rust-based Anycast router, suffered lockups while managing rapidly changing routing state across thousands of servers. An earlier outage came from an if let expression that kept a read lock through both branches, while later regionalization and lazy loading increased write contention in the Catalog. The investigation traced the new failures to parking\_lot’s try\_write\_for wakeup path: after a waiting writer timed out, it attempted to clear WRITER and WRITER\_PARKED together even though one bit had already been cleared, and wrapping arithmetic produced the corrupted lock value 0xFFFFFFFFFFFFFFFE. The parking\_lot team fixed the defect, after which the lockups did not recur; the team also audited Catalog locking, removed if let patterns and RAII lock guards, and added lock-timing metrics, labeled slow-write logs, and holder tracking.

## Context

fly-proxy distributes and uses routing state for Fly Machines across thousands of servers. Its Catalog is an in-memory aggregation of Corrosion data, accessed concurrently by many readers and writers. Regionalization and lazy loading changed those access patterns, introducing more writer contention and exposing failures that appeared as lockups.

## Approach / What changed

The investigation used a minimal reproduction with a forked parking\_lot that confirmed and logged the faulty condition. The lock implementation was fixed so the writer bit is cleared separately in the same wakeup queue as the reader. Catalog locking was also audited, with lock timing, labeled slow-write logs, and write-lock holder tracking added for future diagnosis.

## Takeaways

- The Catalog’s read-write locking became more contended as regionalization expanded lazy loading, making the failures appear to be deadlocks even though the underlying issue was a lock implementation defect.
- parking\_lot’s try\_write\_for path could clear WRITER and WRITER\_PARKED together after WRITER\_PARKED had already been unset. The resulting wrapping arithmetic corrupted the lock word to 0xFFFFFFFFFFFFFFFE.
- After the fix, lockups did not recur. The team also stopped relying on if let lifetimes and RAII lock guards, enabling lock timing and contextual write-holder diagnostics.

**Tags:** [Anycast](https://yomu.fyi/topic/anycast), [Distributed Systems](https://yomu.fyi/topic/distributed-systems), [parking\_lot](https://yomu.fyi/topic/parking-lot), [Rust](https://yomu.fyi/topic/rust)

- Source: [Fly.io](https://fly.io/blog/parking-lot-ffffffffffffffff)
- Source URL: https://fly.io/blog/parking-lot-ffffffffffffffff
- Ingested by Yomu: 2026-09-03T01:24:19.081Z

[Read original post](https://fly.io/blog/parking-lot-ffffffffffffffff)