Operationalizing Macaroons

Summary

Operationalizing Macaroons at Fly.io requires treating them as online-stateful bearer tokens whose nonces must be checked against database state, while keeping root secrets away from the primary API cluster. The team built tkdb, a roughly 5,000-line Go service backed by encrypted SQLite managed with LiteFS and Litestream, deployed on isolated hardware across the US, Europe, and Australia. Its HTTP/Noise interface supports verification and restricted signing, while FlyCast routing, retries, and client-side caching reduce the impact of unreliable global connectivity; verification caching exceeds 98%. Revocation uses blacklist state, and third-party caveats let services such as flyd obtain narrowly scoped secret-reading authority through tkdb rather than holding broad credentials. OpenTelemetry, standard metrics, and a retained OpenSearch audit trail provide visibility, while the post concludes that Macaroons’ internal infrastructure benefits outweigh limited user adoption.

Takeaways

1. 1

   tkdb is approximately 5,000 lines of Go and uses encrypted SQLite managed by LiteFS and Litestream; the database is only a couple dozen megabytes after several years of Macaroon use.

2. 2

   Macaroon verification caching exceeds 98% because a verified token contains enough information to verify more-specific descendants produced through the chaining HMAC construction.

3. 3

   flyd receives a secret-reading Macaroon with a third-party caveat, requiring tkdb to verify organization permissions instead of granting every worker broad access to user secrets.