Copy-and-Replay vs Branch-at-a-Point: Two Backup Recovery Paths
A side-by-side comparison of copy-and-replay recovery and branch-at-a-point restores, showing why restore time scales with data size in one model and not the other.
Loading…
Topic
Replication
6 posts about Replication. Every summary links to the original.
- 6 posts
- 3 companies
- 1 explainer
See the idea
Replication explainers
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Data liberation: Apache Kafka's native cluster mirroring
KIP-1279 introduces native cluster mirroring, embedding cross-cluster replication directly into the Kafka broker. This eliminates external tooling, offset translation tables, and recompression, as destination brokers fetch committed records using the standard fetch protocol and append them byte for byte. The approach preserves offsets, compression, and consumer group state, simplifying failover to stopping the mirror and redirecting clients. The architecture uses mirror fetcher threads inside the broker, extending AbstractFetcherThread, and leverages existing client quota mechanisms. Migration involves creating a new KRaft cluster, setting up a mirror, and starting mirroring, with future work including synchronous mirroring and tiered storage integration.
Postgres on NVMe: performance and the convergence of transactions and analytics
The post examines how local NVMe storage changes Postgres performance, particularly when working sets exceed memory and disk latency becomes the bottleneck. It argues that while faster storage accelerates transactional workloads, it does not alter the row-store layout, so analytical queries still require different approaches. Benchmark results show 9.2× more throughput on NVMe, with a median of 16,030 TPS versus 1,734 on EBS. The post introduces WalShadow, an open-source replication engine that reads the physical Postgres WAL to enable sub-second replication to ClickHouse without logical slots. It concludes that failover is solvable with tools like Patroni, and that storage latency is only part of the performance problem.
Litestream: Revamped
Litestream’s redesign aims to make SQLite suitable for more deployment scenarios while preserving its embedded, application-oblivious model. The original tool continuously streamed WAL pages to S3-compatible object storage, but restores could be slow because they replayed every change since the last snapshot. The new design adopts LiteFS’s transaction-aware LTX format, sorted changesets, and compaction, enabling faster point-in-time restores with fewer duplicate pages; conditional writes provide time-based leases that prevent overlapping instances from confusing a destination. Planned additions include a VFS-based read-replica layer that fetches and caches pages from object storage, plus replication across hundreds or thousands of database files. The remote read path may be less efficient than local SQLite, but caching and prefetching are expected to provide strong performance for suitable use cases.
LiteFS Cloud: Distributed SQLite with Managed Backups
LiteFS Cloud, launched in preview, adds managed backups and restores to LiteFS, an open-source distributed SQLite database that replicates data across application-cluster nodes. Existing LiteFS users had to create snapshots, choose retention, and accept restoration points limited by snapshot frequency; the service instead provides an equivalent snapshot every five minutes and restores within the last 30 days at five-minute granularity. Setup requires LiteFS 0.5.1 or later, a dashboard-created cluster, and an auth token, after which backups are automatic and dashboard restores use the latest snapshot before a selected timestamp. Its LTX format stores changed database pages, while multi-level compactions combine daily snapshots, hourly changes, and five-minute intervals so a restore can stream-merge a minimal set of files. To improve async replication’s durability trade-offs, the primary batches changes, LiteFS Cloud persists them in S3, and a high water mark prevents local transaction files from being removed before storage confirmation.
I'm All-In on Server-Side SQLite
Ben Johnson argues that SQLite should be considered as the core data and persistence layer for many production full-stack applications, rather than being relegated to unit tests in an n-tier architecture. He attributes this case to cheaper, faster hardware and the fact that most successful applications will never approach terabyte-scale data, making distributed database infrastructure excessive for some workloads. Litestream works alongside standard SQLite by taking control of WAL-mode journaling, capturing WAL updates, replicating them to storage such as S3, and handling checkpoints itself. This preserves ordinary SQLite libraries and transactions while providing a path to restore copies and improve resilience to storage failure; upcoming direct database-to-database replication is also mentioned. The resulting model places data in the application process, where the post reports 10–20 microsecond per-query latency, but it remains aimed at single-node applications and has restore, deployment, and replication-guarantee limitations.
Last Mile Redis
Last Mile Redis presents a globally distributed caching design aimed at keeping backend responses under the 100-millisecond threshold by placing Redis near users. It argues that regional datasets overlap less than expected, making one Redis server per region a practical alternative to centralized sharding. The design combines standalone regional caches with a Dallas primary Redis and writable replicas configured with replica-read-only no, allowing local writes while replication distributes global changes. A DEL issued at the primary can therefore purge bad cache data across regions, while timestamped keys and expiration handle ordinary freshness; the same mechanism can also broadcast content or fan out queue messages.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.