Loading post…
Fly.ioAnnouncement
WAL Mode in LiteFS
Summary
LiteFS v0.3.0 adds support for SQLite’s write-ahead log (WAL) alongside the previously supported rollback journal, allowing its distributed file system to replicate transactions from either journaling mode. The post explains how rollback journals and WAL differ, then shows how LiteFS observes filesystem calls, identifies changed pages, and emits a common LTX change-set format; in WAL mode, the final frame’s commit field marks transaction completion. Replica nodes validate LTX files, acquire SQLite-compatible locks, and apply the changes, while the new litefs import command transactionally replaces a database for bootstrapping or snapshot reversion. Version 0.3.0 also replaces previous-page checksum tracking with a per-page checksum map, costing about 3 MB of RAM per gigabyte and roughly 5 ms of CPU time per gigabyte per commit, and adds a trace log whose enabled use can double disk I/O.
Context
Before v0.3.0, LiteFS supported only SQLite’s rollback journal mode, despite WAL’s concurrency and write-performance benefits. Converting an existing application to LiteFS required creating a SQL dump and importing it with the sqlite3 command line. LiteFS also found it difficult to calculate previous page checksums correctly because page data was spread across the database, journal, and WAL files.
Approach / What changed
LiteFS observes filesystem operations to detect transaction boundaries and changed pages, then writes those page change sets to a common LTX format regardless of SQLite’s journaling mode. In WAL mode, it reads changed pages and uses the final WAL frame’s commit field to identify transaction completion. Replicas validate LTX files, acquire SQLite-compatible locks, and apply transactions. Version 0.3.0 also adds litefs import, a per-page checksum map, and a trace log.
Takeaways
WAL lets SQLite readers continue while another process writes because the original database data is not changed during the transaction; it also tends to improve write performance.
The revised checksum map uses about 12 bytes per page, or approximately 3 MB of RAM per gigabyte of SQLite data, and adds about 5 ms of processing per gigabyte after each commit in testing.
The trace log translates filesystem activity into SQLite-related events with page and checksum details, but enabling it can double disk I/O; it is disabled by default and retains ten 64 MB rolled, gzipped logs.
Source and freshness
- Source
- fly.io
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.