Introducing LiteFS

Summary

LiteFS is an open-source distributed file system for SQLite that addresses the durability, replication, availability, and latency limitations of deploying SQLite on a single server. LiteFS interposes a FUSE pass-through filesystem between an application and its database, identifies transactions from SQLite’s journal files, packages page changes into LTX files, and ships them to replicas without requiring application-code changes. A Consul-backed, time-based lease elects a primary writer, while local replicas serve reads; checksum state embedded in each LTX file helps detect out-of-order application and preserve byte-for-byte consistency. Because replication is asynchronous, the design trades some durability guarantees for performance and operational simplicity, while defensive checksum checks address potential split-brain failure modes. The project is described as beta-ready, with WAL-mode support, write forwarding, S3 replication, and encryption listed as planned features.

Takeaways

1. 1

   In SQLite’s default rollback-journal mode, LiteFS identifies a write transaction between creation and deletion of the -journal file, then retrieves new page data from the main database file.

2. 2

   Consul can grant a time-based lease to a candidate node, making it the sole writer; releasing that lease allows primary status to move to another node during deployment or failure.

3. 3

   LiteFS maintains database checksums incrementally with XOR operations over page changes and stores them in LTX files, allowing startup validation and rejection of out-of-order state.