Copy-and-Replay vs Branch-at-a-Point: Two Backup Recovery Paths
A side-by-side comparison of copy-and-replay recovery and branch-at-a-point restores, showing why restore time scales with data size in one model and not the other.
Loading…
Topic
Postgres
301 posts about Postgres. Every summary links to the original.
- 301 posts
- 15 companies
- 2 explainers
See the idea
Postgres explainers
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
September 2026 newsletter
The September 2026 ClickHouse newsletter gathers product updates, community contributions, technical articles, and upcoming events. Its main release coverage describes ClickHouse 26.8, including custom HTTP handlers, dynamic query filtering, pipelined SQL through the |> operator, current-user query logging, a URL database engine, and Japanese and Chinese text-index tokenizers. Other featured developments include On-Demand Compute in private preview with a cost-based optimizer and distributed query execution, PromQL and a TimeSeries engine in private preview, AI Functions, and Postgres integrations such as WalShadow and Managed Postgres on Google Cloud. The issue also reports CostBench’s first end-to-end performance-per-dollar results, explains replication-queue diagnosis, notes 57 breaking changes since 26.3, and lists training, webinars, roadshows, and other events.
Managed Postgres: What Lakebase Actually Takes Off Your Plate
Managed Postgres varies from OS patching alone to operating scaling, failover, backups, recovery, and related database tooling. Lakebase Postgres applies this model to serverless infrastructure, offering automatic scaling, scale-to-zero, PostgreSQL compatibility, automatic snapshots, point-in-time recovery, branching, built-in PgBouncer, and support for extensions including pgvector and PostGIS. Within a region, patching, scaling, failover, and backups run automatically, while encryption, access control, audit visibility, connection pooling, and lakehouse integrations are also addressed. The discussion also stresses checking PostgreSQL versions, configuration, extensions, and migration methods before moving an existing database. Cross-region disaster recovery remains the main exception: it is in Private Preview for AWS, with periodic replication, manual failover, and customer-managed recovery procedures.
Introducing WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL
ClickHouse announces WalShadow, an open-source engine that replicates Postgres data to ClickHouse directly from physical WAL, targeting sub-second analytical availability without logical replication. It consumes the physical WAL stream outside the source database, tracks schema in a shadow Postgres instance, decodes heap records through parallel Rust-based decoders, builds ClickHouse-native blocks, and inserts them concurrently. Because parallel blocks can arrive out of order, each row carries the source WAL position (_lsn), while barriers preserve ordering for schema changes and truncations. In benchmarks using modest c8i.2xlarge instances, WalShadow reached approximately 200 ms commit-to-visible latency and 289,000 rows per second, compared with PeerDB’s roughly 10 seconds and 120,000 rows per second; it supports initial load, continuous replication, schema evolution, restart recovery, and planned switchovers.
Three principles for building a vector platform at Thumbtack
Thumbtack describes a vector platform that lets ML engineers stand up production vector search without separately negotiating database access, building ETL, or writing a query service. The design uses an existing Aurora Postgres cluster with pgvector, while Airflow jobs generate embeddings into BigQuery and managed reverse-ETL syncs them into vector tables; a query API validates inputs and returns ranked results. Three principles guide the layers: reuse trusted infrastructure, treat embeddings as ordinary data flowing through established pipelines, and lower adoption costs for subsequent teams. Production examples include semantic customer-support escalation using cosine similarity and an internal knowledge assistant combining vector retrieval, often with keyword search, before a language model responds; planned directions include self-serve provisioning, vector-backed feature retrieval, and framework-neutral agent memory.
Improving Lakebase Postgres compute cache
Lakebase Postgres uses disaggregated storage backed by object storage, so compute-side caching must deliver low latency and high throughput while avoiding inefficient memory use. Its original design paired a conservatively sized PostgreSQL shared buffer with a local file cache (LFC), but the roughly 1 GB shared-buffer cap pushed many hits to slower NVMe storage and could create double buffering. For fixed-size computes with CU >= 80, the new configuration disables the LFC and sets shared buffers to 75% of DRAM, using explicit 2 MB HugeTLB pages to control page-table and TLB overhead. Benchmark tests found huge pages reduced tail read latency by up to ~40% and CPU use by up to ~30%, while production examples reported doubled throughput, 5× fewer storage reads, or 5× lower CPU use. Work is underway to dynamically resize shared buffers and huge pages for autoscaling computes.
Announcing ClickHouse Managed Postgres on Google Cloud
ClickHouse Managed Postgres is entering Private Preview on Google Cloud, extending the fully managed, NVMe-backed Postgres service previously launched on AWS. The GCP deployment targets teams wanting Postgres for OLTP and ClickHouse for OLAP in one cloud, avoiding cross-cloud replication, egress costs, and separate networking and compliance management. It uses GCP Local SSD storage, native change data capture through ClickPipes, in-region networking, and pg_clickhouse, which pushes filters, joins, and aggregations to ClickHouse behind a single Postgres connection. Preview capabilities include zonal high availability with two standbys and quorum replication, point-in-time recovery, read replicas, PgBouncer, more than 90 Postgres extensions, monitoring, Query Insights, scheduled upgrades, and infrastructure-as-code support through OpenAPI, Terraform, and clickhousectl. Access is invitation-only via a waitlist, with availability in every GCP region where ClickHouse Cloud runs and hands-on migration support from Cloud SQL, AlloyDB, RDS, Aurora, Neon, or self-managed Postgres.
Introducing Consort: Test-driven development on a branching database
Consort is introduced as an open-source agentic development framework for test-driven development on branching Lakebase Postgres databases. It addresses the limitations of mocks, shared staging databases, and risky schema-change ceremonies by bringing integration testing into the inner loop. Its build lane runs red/green/refactor cycles against isolated copy-on-write branches of real data, while versioned migrations keep code and schema changes together and pull-request checks test migrations against fresh branches. Consort organizes work through role-based agents and a deterministic conductor, preserving feature artifacts in dedicated directories and supplying scoped context packages. A VS Code plugin pairs Git and Lakebase branches, while an observability dashboard exposes agent turns and human approval gates; the project is community-supported rather than a bundled platform feature with an SLA.
Build durable agents with Temporal and Lakebase
Long-running personal-loan underwriting agents must preserve completed work, resume after worker failures, tolerate repeated tool calls, wait for reviewers, expose operational state, and retain an audit trail. The reference implementation combines Temporal durable Workflows and Activities with Lakebase Postgres, using Temporal Event History for replay and Lakebase schemas for run status, evidence, review state, messages, and metrics. Deterministic identifiers, constraints, guarded updates, and Postgres upserts make repeated Activity attempts target the same logical records, while Signals carry human decisions and long waits do not occupy workers. A Unity Catalog synced table supplies governed underwriting policy, and optional Change Data Feed publishes operational changes to Delta history tables. The implementation includes a React/FastAPI interface, mocked deterministic providers, OAuth credential refresh, and 21 passing tests, but does not validate production lending, compliance, security, scale, or an observed end-to-end feed.
Introducing chdb Postgres extension: High-performance imports from cloud storage
The new chdb Postgres extension uses the chDB library, an in-process ClickHouse engine, to import and export data between Postgres and files on cloud storage. In benchmarks using the one-million-row NYC Taxi dataset, chdb ran on four-vCPU, 32-GB ClickHouse Managed Postgres instances and showed the most consistent performance; pg_duckdb and pg_lake took about two to three times longer for CSV, JSON, and Parquet, while aws_s3 supported fewer formats. Its chdb_query() function executes chDB queries, while chdb_hook integrates with COPY for AWS S3, Google Cloud Storage, Azure Blob Storage, local files, and HTTP URLs, with wildcard expansion for multiple files. The extension launches a separate helper application on demand and exchanges ClickHouse Native results through file descriptors, limiting resource use and isolating the PostgreSQL cluster.
The 40-year-old database rule agents just broke: How LTAP unifies OLTP and OLAP workloads
LTAP (Lake Transactional/Analytical Processing) addresses the long-standing separation between operational row-oriented databases and analytical column-oriented systems, which traditionally required pipelines and copied data. It uses Lakebase’s stateless, ephemeral compute decoupled from durable lake storage to expose live operational data for analytical queries without moving it or loading the transaction-serving system. The storage layer preserves Postgres data types and physical representation while writing data into Parquet, with a hot row-format tier for operational access and a cooler columnar tier for analytical reads; Apache Spark and SQL can read that representation directly. By separating serverless operational and analytical compute, LTAP allows each workload to use specialized engines and independent capacity, while a unified catalog applies governance across both. The stated motivation is agentic workloads such as fraud detection, where stale copies miss fast events and uncontrolled analytical queries can degrade transactions.
Introducing Scheduled Upgrades in ClickHouse Managed Postgres
ClickHouse Managed Postgres now supports scheduled maintenance windows, addressing the problem of routine platform upgrades landing at the same time for every customer, potentially during heavy workloads. Users can choose a two-hour UTC window, while Enterprise organizations can also specify the days of the week when maintenance is least disruptive. The setting covers ClickHouse-initiated Postgres minor-version upgrades, Managed Postgres feature updates, and patches to operating-system and system components, but not user-triggered changes or urgent storage scaling above 90% disk usage. Read replicas follow the schedule of their primary, and the feature is available without opt-in to Scale and Enterprise organizations; Basic users can see the setting and required plan.
From Neon Postgres to ClickHouse Managed Postgres
This user story describes three teams that moved production Postgres workloads from Neon to ClickHouse Managed Postgres after encountering reliability, performance, cost, and network-transfer problems as usage grew. ClickPipes handled schema migration, a parallel initial load, and change data capture, allowing teams to test synchronized databases before cutover. Reported outcomes included zero downtime or connection restarts, faster and more predictable queries, lower CPU or transfer costs, and p99 latency improving from seconds to single-digit milliseconds in one case. The post attributes these gains to local NVMe storage and eliminating transfer fees when analytics already runs on ClickHouse, while noting differences from Neon such as slower branch creation and manual application of several DDL changes.
Collaboration makes us all stronger
An external researcher, Mehmet Ince, reported a memory-safety flaw in PostGIS’s address_standardizer extension, where caller-controlled grammar data indexed a fixed-size array without bounds checking. Because the extension was available to ordinary tenant roles on managed Postgres services, Databricks treated the issue as a platform exposure rather than a third-party problem, while noting that its microVM architecture meant the exploit did not produce cross-customer impact on Lakebase Postgres and Neon. Detection alerted the Neon team during testing, enabling rapid validation and downstream patching while a durable fix was developed and deployed to protect tenants without customer action. The upstream fix initially missed cases; Mehmet supplied the remaining pieces, and the complete correction was returned to PostGIS, with his bounty donated and matched for the project.
New system views in PostgreSQL 19
PostgreSQL 19 adds four system views for lock contention, recovery, autovacuum priorities, and dynamic shared memory allocations. The article presents pg_stat_lock as cumulative, cluster-wide statistics with one row per lock type, reporting waits, total wait time, fastpath_exceeded, and stats_reset; waits and wait_time count only successfully acquired locks that exceeded deadlock_timeout, while fast-path overflow attempts are counted separately. Examples reset statistics, reproduce a 2.5-second table lock, and scan 140 partitions, producing 1 wait, 2201.783 milliseconds, and 422 fast-path overflows. The post describes recovery and autovacuum views, noting that autovacuum scores suggest likely priorities rather than guarantees because scoring and worker scheduling can differ. DSM registry entries expose name, type, and size; NULL means initialization failed.
Autoscaling Lakebase Postgres
Lakebase Postgres addresses the waste and uncertainty of choosing a fixed database instance size before workload demand is known. Its decoupled architecture separates stateless compute from durable storage, allowing compute to start, stop, move, or resize without moving the database. The autoscaling algorithm combines CPU load, memory pressure, and a time-aware estimate of the Postgres working set, using modified HyperLogLog registers that store timestamps and estimate windows from one to 60 minutes. Scaling occurs live through autoscaler-agent, vm-monitor, Kubernetes scheduling, and NeonVM, with checks preventing unsafe downscales and scheduler approval for upscales. The system scales in both directions without dropping connections and operates across 100-millisecond, five-second, and 20-second timescales, including more than 32,000 resizes per month.
A new getting started experience for ClickHouse Managed Postgres
ClickHouse Managed Postgres introduces a four-step onboarding experience designed to move users from provisioning a service to working with operational data and ClickHouse analytics. Existing onboarding placed new users on an empty Overview page, while the documentation quickstart required users to track progress outside the product. The new flow guides users through creating a service, migrating or ingesting data, running a query in the SQL Console, and configuring Postgres CDC to ClickHouse. Data can arrive through a managed Postgres-to-Postgres ClickPipe, a language client, a sample dataset, or a table created from scratch; pg_clickhouse provides a PostgreSQL query layer over ClickHouse foreign tables. The article reports less than a minute of replication lag and benchmarks showing five aggregations falling from 555 ms to 164 ms and a JOIN with aggregations from 1,246 ms to 170 ms on one million-row data set.
Building for the AI Era: Lakebase, Streaming, and Lakehouse Innovations at VLDB 2026
Databricks previews its VLDB 2026 keynote, four accepted papers, a demo, and a sponsor talk covering database architecture, streaming, and Lakehouse optimization. Reynold Xin will introduce Lakebase, which separates serverless PostgreSQL compute from storage, and LTAP, a transactional-analytical model intended for AI-agent workloads; Lakebase stores data and write-ahead logs in cloud object storage and supports sub-second cold starts and copy-on-write branching. The papers describe a decade of Spark Structured Streaming evolution, including microbatch pipelining that improved throughput by up to 3x, stateful APIs, and fine-grained access control, alongside AutoLiquid’s CLUSTER BY AUTO and Ultron’s history-based query optimization. AutoLiquid outperformed customer-selected clustering keys on over 95% of evaluated workloads, while Ultron improved median join latency by 25% in production workloads. The program also includes an Enzyme demonstration of incremental materialized-view maintenance and a sponsor presentation on LakehouseRT and the Reyden engine.
Vertical Advantage: Transforming Industries with Lakebase and Agentic AI
The post presents partner-built, industry-specific solutions on Databricks Lakebase for financial services, manufacturing and energy, retail and consumer sectors, travel and hospitality, healthcare and life sciences, communications and media, and the public sector. Lakebase is described as a serverless Postgres transactional database on the Databricks Data and AI Platform, combining operational and analytical workloads under unified governance and providing operational serving, agent memory, audit trails, and transactional runtime capabilities. Examples include regulation-change monitoring, AI-native claims operations, KYC and personalization accelerators, near-real-time fraud detection, telecom customer-lifecycle decisioning, and public-sector emergency Situation Reports, using components such as Unity Catalog, Lakeflow, Genie, Mosaic AI agents, and Agent Bricks. The stated outcome is production-ready acceleration from insight to action, including faster compliance response, claims adjudication in minutes, low-latency decisions, and reduced manual effort, with partners positioned to help scope pilots.
Object Storage + WAL: Lakebase Postgres for the agentic era
The post presents Lakebase Postgres as a design for agent workloads that need isolated copies, restores, replicas, historical queries, and many short-lived databases without repeatedly moving large datasets. It contrasts conventional data-centric OLTP with a transaction-centric model in which PostgreSQL’s write-ahead log (WAL) is the source of truth, with LSNs making database history addressable. Lakebase splits compute, which runs standard Postgres and uses RAM and local NVMe caches, from storage, where safekeepers replicate WAL through quorum acknowledgements, a pageserver materializes pages, and object storage retains immutable history. Commits wait for replicated WAL acknowledgement, not page uploads, while reads request pages at an LSN. The design supports point-in-time queries, cheap branch or read-only compute, scale-to-zero suspension, and shared transactional and analytical access, while still relying on RAM and NVMe for performance.
A universal interface: How QuintoAndar made ClickHouse plug-and-play with managed Postgres
QuintoAndar’s customer data platform was split between a Postgres-based hot layer for low-latency lookups and a Databricks-based cold layer for analytical processing, creating duplicate code and expensive MERGE INTO operations. The team consolidated the pipeline by having its event gateway write directly to ClickHouse Cloud, which also serves API requests and analytical queries while remaining accessible to Trino, Spark, and Superset. Hightouch’s Customer Studio did not support ClickHouse, so QuintoAndar used ClickHouse Managed Postgres and the pg_clickhouse extension to expose ClickHouse tables as Postgres tables. Queries are pushed down to ClickHouse, while Hightouch’s planner and audit tables remain in NVMe-backed Postgres, allowing the existing activation workflow to continue with negligible round-trip impact for that use case.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.