Loading…
Topic
Litestream
4 posts about Litestream. Every summary links to the original.
- 4 posts
- 1 company
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Litestream Writable VFS
Litestream’s VFS is being extended to support Sprites, whose storage stack needs SQLite-backed block maps to serve writes immediately during cold starts, before a full database or block map is available. The VFS indexes each database page by file, offset, and size in LTX files, serves reads from object storage, and now optionally buffers writes locally before syncing them roughly every second or on clean shutdown. Write mode assumes a single writer, disables polling, and offers only the same eventual durability already provided by Sprite storage. A separate hydration mode serves remote reads while reconstructing a complete database file in the background, then switches reads to that file; the temporary file is discarded on exit. These features target Sprite-like workloads, while ordinary Litestream sidecar deployments do not require them.
Litestream VFS
Litestream VFS lets SQLite query a Litestream-backed database directly from object storage, including historical states selected with SQL pragmas, without first downloading the complete database. It is a loadable SQLite VFS plugin: Litestream v0.5 uses the LTX data-shipping format, whose compaction removes superseded page versions, while index trailers map database pages to remote file offsets. The VFS fetches only requested page ranges through S3, uses an LRU cache, and can poll second-level L0 backups to maintain a near-realtime replica. Writes remain handled by the regular Litestream Unix program, and the existing SQLite library is unchanged. The result is fast querying and instantaneous point-in-time recovery from arbitrary relative or absolute timestamps, while the optional VFS preserves Litestream’s conventional backup and restore workflow.
Operationalizing Macaroons
Operationalizing Macaroons at Fly.io requires treating them as online-stateful bearer tokens whose nonces must be checked against database state, while keeping root secrets away from the primary API cluster. The team built tkdb, a roughly 5,000-line Go service backed by encrypted SQLite managed with LiteFS and Litestream, deployed on isolated hardware across the US, Europe, and Australia. Its HTTP/Noise interface supports verification and restricted signing, while FlyCast routing, retries, and client-side caching reduce the impact of unreliable global connectivity; verification caching exceeds 98%. Revocation uses blacklist state, and third-party caveats let services such as flyd obtain narrowly scoped secret-reading authority through tkdb rather than holding broad credentials. OpenTelemetry, standard metrics, and a retained OpenSearch audit trail provide visibility, while the post concludes that Macaroons’ internal infrastructure benefits outweigh limited user adoption.
I'm All-In on Server-Side SQLite
Ben Johnson argues that SQLite should be considered as the core data and persistence layer for many production full-stack applications, rather than being relegated to unit tests in an n-tier architecture. He attributes this case to cheaper, faster hardware and the fact that most successful applications will never approach terabyte-scale data, making distributed database infrastructure excessive for some workloads. Litestream works alongside standard SQLite by taking control of WAL-mode journaling, capturing WAL updates, replicating them to storage such as S3, and handling checkpoints itself. This preserves ordinary SQLite libraries and transactions while providing a path to restore copies and improve resilience to storage failure; upcoming direct database-to-database replication is also mentioned. The resulting model places data in the application process, where the post reports 10–20 microsecond per-query latency, but it remains aimed at single-node applications and has restore, deployment, and replication-guarantee limitations.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.