Active Storage Is a Boundary Decision
See how Active Storage balances local simplicity, durable object storage, and preview-processing dependencies so you can choose the right boundary for a Rails file workflow.
Loading…
Topic
Object Storage
15 posts about Object Storage. Every summary links to the original.
- 15 posts
- 4 companies
- 1 explainer
See the idea
Object Storage explainers
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Lakebase Postgres branch-based restores for fast recovery at scale
Traditional Postgres point-in-time recovery is slow because compute and storage are coupled, requiring provisioning a new instance, restoring a snapshot from object storage to disk, and replaying WAL, leading to hours of downtime for large databases. In a survey of developers running 1TB+ production Postgres, 59% had a critical production failure and 30% were down for over three hours. Lakebase Postgres decouples compute and storage using safekeepers, a pageserver, and object storage, storing database history as an immutable timeline. A restore becomes a branch at a point in history, a metadata operation that does not copy data, making it near-instant regardless of database size. This eliminates data copies and WAL replay, so restoring a 100 TB database is as fast as restoring 10 GB, and agents can use restores as ordinary tool calls for versioning and undo.
chDB Durable Layer for agent memory
Agent memory built on chDB, an embedded SQL OLAP engine, accumulated state such as project rules, user preferences, and recall traces on a single laptop. Moving that memory to CI, sandboxes, or another laptop broke the local-disk assumption because the MergeTree directory existed on only one disk. Adding a ClickHouse server backend fixed portability but turned every recall into a network round trip. The chDB Durable Layer fills the gap by keeping the local analytical hot path while publishing authoritative state to object storage (S3, GCS, or Azure Blob). Applications write to a local working copy and use flush() to publish writes and checkpoint() to compact the log, avoiding a sidecar or database server. The contract supports Python, Node, Go, and Rust bindings with cross-binding recoverability.
How agentic software development will change databases
AI agents are changing software development from a slow, linear process into rapid evolutionary experimentation, creating many small, ephemeral applications and databases. The post identifies three resulting database requirements: near-zero marginal cost with elastic growth, native branching for rapid variants, and open ecosystems that agents can use reliably. Lakebase addresses branching through O(1) metadata copy-on-write at the storage layer, so agents can branch production or test state without physical data copying and pay only for experiment compute. Its decoupled, serverless compute can scale in sub-second intervals and down to zero, while standard open Postgres page formats in cloud object storage keep data accessible to external engines and tools. Lakebase telemetry reports about 10 branches per project on average, some nested beyond 500 iterations, and roughly four times as many databases created by agents as by humans.
Running Cost-Efficient Export Workloads at Uber
Uber’s offline export workloads often retrieve a small set of records from large, date-partitioned Apache Hudi tables, but narrow historical predicates can still trigger full-table scans. Repeatedly touching files, metadata, or Parquet footers keeps Google Cloud Storage objects hot, undermining Auto-Class tiering and increasing storage, retrieval, metadata-operation, egress, and latency costs. The approach combines Hudi column stats with sorting on predicate columns: metadata-table statistics prune files without opening them, while clustered records tighten file-level min/max ranges and make pruning more selective. A Hudi secondary index was evaluated but did not reduce file scans enough because records for a subject were spread across most files. The conclusion is that a one-time rewrite and ongoing Hudi table services can turn broad historical scans into lower-touch lookups, while future work targets multi-column layouts and automated ROI-based optimization.
Litestream Writable VFS
Litestream’s VFS is being extended to support Sprites, whose storage stack needs SQLite-backed block maps to serve writes immediately during cold starts, before a full database or block map is available. The VFS indexes each database page by file, offset, and size in LTX files, serves reads from object storage, and now optionally buffers writes locally before syncing them roughly every second or on clean shutdown. Write mode assumes a single writer, disables polling, and offers only the same eventual durability already provided by Sprite storage. A separate hydration mode serves remote reads while reconstructing a complete database file in the background, then switches reads to that file; the temporary file is discarded on exit. These features target Sprite-like workloads, while ordinary Litestream sidecar deployments do not require them.
The Design & Implementation of Sprites
Sprites are Linux virtual machines designed as fast, disposable computers with root access, persistent 100GB durable filesystems, automatic sleeping, and low inactive cost. To make creation feel instantaneous, the platform removes user-facing OCI container images and keeps pools of preloaded standard Fly Machines, so a new Sprite mainly performs startup work rather than pulling and unpacking an image. Its storage stack follows a hacked-up JuiceFS model, splitting immutable data chunks from metadata, keeping metadata durable with Litestream, and using a sparse NVMe volume as a dm-cache-like read-through cache for object storage; checkpoints and restores primarily shuffle metadata. This design supports migration and recovery without depending on local storage, while low-cost sleeping and object-backed disks suit interactive work more than warm, containerized production workloads, and billing applies only to storage blocks actually written.
Litestream VFS
Litestream VFS lets SQLite query a Litestream-backed database directly from object storage, including historical states selected with SQL pragmas, without first downloading the complete database. It is a loadable SQLite VFS plugin: Litestream v0.5 uses the LTX data-shipping format, whose compaction removes superseded page versions, while index trailers map database pages to remote file offsets. The VFS fetches only requested page ranges through S3, uses an LRU cache, and can poll second-level L0 backups to maintain a near-realtime replica. Writes remain handled by the regular Litestream Unix program, and the existing SQLite library is unchanged. The result is fast querying and instantaneous point-in-time recovery from arbitrary relative or absolute timestamps, while the optional VFS preserves Litestream’s conventional backup and restore workflow.
Litestream v0.5.0 is Here
Litestream v0.5.0 updates the SQLite backup and restore tool with faster operation and efficient point-in-time recovery (PITR). It replaces WAL segment generations with LTX files, an interchange format for ordered page ranges that supports compaction and monotonically incrementing transaction IDs. Hierarchical compaction combines changes across 30-second, five-minute, and hourly levels, allowing restoration to any point in time using only about a dozen files on average and performance limited by I/O throughput. The release keeps old v0.3.x WAL files intact and preserves configuration compatibility, but permits only one replica destination per database and renames litestream wal to litestream ltx. It also adds per-page compression and indexing, removes CGO through modernc.org/sqlite, supports NATS JetStream replicas, updates cloud-storage clients, and previews a VFS for read replicas that can fetch pages from S3 while hydration continues.
Litestream: Revamped
Litestream’s redesign aims to make SQLite suitable for more deployment scenarios while preserving its embedded, application-oblivious model. The original tool continuously streamed WAL pages to S3-compatible object storage, but restores could be slow because they replayed every change since the last snapshot. The new design adopts LiteFS’s transaction-aware LTX format, sorted changesets, and compaction, enabling faster point-in-time restores with fewer duplicate pages; conditional writes provide time-based leases that prevent overlapping instances from confusing a destination. Planned additions include a VFS-based read-replica layer that fetches and caches pages from object storage, plus replication across hundreds or thousands of database files. The remote read path may be less efficient than local SQLite, but caching and prefetching are expected to provide strong performance for suitable use cases.
We're Cutting L40S Prices In Half
Fly.io lowered the hourly price of NVIDIA L40S GPUs to $1.25, making them available at the same price as A10s after customer data showed that A10s—not the more powerful cards—were its most popular GPU. The post argues that most users adding GPUs to fast networks are running inference rather than training, making networking, memory, object storage, and request latency more important than access to an H100. The L40S is described as an AI-optimized L40 with AI compute performance comparable to an A100, while the L40 retains graphics hardware and is designed for rack use with more memory and lower energy consumption than gaming GPUs. The intended combination places model parameters, datasets, and compute near one another on Fly.io’s Anycast network and Tigris object storage. Examples include Llama 3.1 70B, Flux, Whisper, SegAlign, and DOOM Eternal.
Globally Distributed Object Storage with Tigris
Globally distributed object storage is presented as a way to avoid the latency and application complexity that arise when modern apps route uploads and downloads through a single-region S3 bucket. Tigris addresses this with redundant FoundationDB clusters for object tracking, Fly.io NVMe volumes as a first-level raw-byte cache, and a queueing system modeled on Apple’s QuiCK paper that sends data to replicas, in-demand regions, and third-party stores such as S3. Objects smaller than about 128 kilobytes are instantly global by default, while the platform also provides inter-region routing, consistency guarantees, instant purge, and relay regions. Its S3-compatible API means existing S3 libraries can be reused, and fly storage create injects the required AWS configuration and secrets. Fly.io says the partnership delivers object storage under its billing system without requiring a separate account.
LiteFS Cloud: Distributed SQLite with Managed Backups
LiteFS Cloud, launched in preview, adds managed backups and restores to LiteFS, an open-source distributed SQLite database that replicates data across application-cluster nodes. Existing LiteFS users had to create snapshots, choose retention, and accept restoration points limited by snapshot frequency; the service instead provides an equivalent snapshot every five minutes and restores within the last 30 days at five-minute granularity. Setup requires LiteFS 0.5.1 or later, a dashboard-created cluster, and an auth token, after which backups are automatic and dashboard restores use the latest snapshot before a selected timestamp. Its LTX format stores changed database pages, while multi-level compactions combine daily snapshots, hourly changes, and five-minute intervals so a restore can stream-merge a minimal set of files. To improve async replication’s durability trade-offs, the primary batches changes, LiteFS Cloud persists them in S3, and a high water mark prevents local transaction files from being removed before storage confirmation.
Logbook: November 7 to November 14, 2022
The November 7–14, 2022 logbook presents updates intended to make Elixir applications easier to deploy and monitor in production, alongside documentation and storage guidance. Mark’s contribution explains how to set up Github Actions for Elixir CI, including the work needed to automate deployment workflows. A separate guide uses OpenTelemetry to monitor a production Elixir deployment and trace an N+1 query performance problem, while the Django section reports community progress toward more comprehensive Fly documentation. The logbook also points to a MinIO deployment guide for self-hosted, S3-compatible object storage, noting that colocating storage in the same datacenter as an application can matter although using an S3 host such as AWS S3 is described as easier in some cases.
Volume Expansion and Snapshot Restores
Fly.io introduces two improvements to its persistent storage volumes: extending an existing volume and restoring a regular volume from a snapshot. Previously, increasing capacity required provisioning an empty volume and copying data, whereas the new fly volume extend command resizes it after the attached app restarts; regular apps restart automatically, while Machines VMs require manual restarts. Individual restores are available from flyctl v0.0.363 by supplying --snapshot-id when creating a new volume, after listing snapshots and selecting the desired ID. Snapshots are taken daily, stored in object storage for five days, and can restore data only to a volume at least as large as the source.
Appkata - Example Apps on Fly
Appkata introduces four example backend apps for Fly, intended to show how to deploy common services while learning the platform’s deployment patterns. The examples cover Redis as a customizable, volume-backed key/value database; MinIO as a single-node, S3-compatible object store exposed through Fly’s TLS edge connections; Gogs as a private Git server using SQLite3 and Fly volumes; and Mosquitto as an MQTT broker with TLS, username/password authentication, and volume-backed message persistence. The examples emphasize adapting existing Docker images, attaching persistent storage, and exposing services securely. The post presents these as an initial set that will expand over the following weeks, alongside a Fly Community site for questions and guidance.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.