Litestream: Revamped

Summary

Litestream’s redesign aims to make SQLite suitable for more deployment scenarios while preserving its embedded, application-oblivious model. The original tool continuously streamed WAL pages to S3-compatible object storage, but restores could be slow because they replayed every change since the last snapshot. The new design adopts LiteFS’s transaction-aware LTX format, sorted changesets, and compaction, enabling faster point-in-time restores with fewer duplicate pages; conditional writes provide time-based leases that prevent overlapping instances from confusing a destination. Planned additions include a VFS-based read-replica layer that fetches and caches pages from object storage, plus replication across hundreds or thousands of database files. The remote read path may be less efficient than local SQLite, but caching and prefetching are expected to provide strong performance for suitable use cases.

Takeaways

1. 1

   LTX stores sorted, transaction-associated page ranges, allowing multiple time ranges to be compacted into a changeset containing only the latest version of each page.

2. 2

   Conditional writes from modern object stores can provide Litestream with a time-based lease, avoiding a Consul dependency while preventing overlapping instances from confusing one destination.

3. 3

   The planned VFS read-replica layer trades local SQLite efficiency for remote reads from Tigris or S3, with caching and prefetching intended to improve performance for suitable use cases.