parking_lot: ffffffffffffffff...

Summary

Fly.io’s fly-proxy, a Rust-based Anycast router, suffered lockups while managing rapidly changing routing state across thousands of servers. An earlier outage came from an if let expression that kept a read lock through both branches, while later regionalization and lazy loading increased write contention in the Catalog. The investigation traced the new failures to parking_lot’s try_write_for wakeup path: after a waiting writer timed out, it attempted to clear WRITER and WRITER_PARKED together even though one bit had already been cleared, and wrapping arithmetic produced the corrupted lock value 0xFFFFFFFFFFFFFFFE. The parking_lot team fixed the defect, after which the lockups did not recur; the team also audited Catalog locking, removed if let patterns and RAII lock guards, and added lock-timing metrics, labeled slow-write logs, and holder tracking.

Takeaways

1. 1

   The Catalog’s read-write locking became more contended as regionalization expanded lazy loading, making the failures appear to be deadlocks even though the underlying issue was a lock implementation defect.

2. 2

   parking_lot’s try_write_for path could clear WRITER and WRITER_PARKED together after WRITER_PARKED had already been unset. The resulting wrapping arithmetic corrupted the lock word to 0xFFFFFFFFFFFFFFFE.

3. 3

   After the fix, lockups did not recur. The team also stopped relying on if let lifetimes and RAII lock guards, enabling lock timing and contextual write-holder diagnostics.