Copy-and-Replay vs Branch-at-a-Point: Two Backup Recovery Paths
A side-by-side comparison of copy-and-replay recovery and branch-at-a-point restores, showing why restore time scales with data size in one model and not the other.
Loading…
Topic
Postgres
298 posts about Postgres. Every summary links to the original.
- 298 posts
- 15 companies
- 2 explainers
See the idea
Postgres explainers
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
We're Transferring the Stripe Sync Engine to Stripe
The Stripe Sync Engine repository is moving from supabase/stripe-sync-engine to stripe/sync-engine after growing from an open-source tool for syncing Stripe data into a Stripe schema in Postgres. The transfer follows collaboration that added cursor-based incremental sync, JSONB storage with generated columns, multi-account support for Stripe Connect platforms, event ordering in a new CLI, and automatic incremental backfilling using Supabase Cron and Queue. The latest release adds coupon synchronization, branching support, immediate syncing on installation, one-click upgrades, SSL enforcement, and tighter installation controls, along with clearer errors and dashboard visibility for Edge Function source code. The project remains open source under Apache 2.0, existing installations continue to work, and the Supabase dashboard integration is unchanged. With Stripe data in Postgres, users can join it with application data for subscription checks, feature gating, MRR, and churn analysis.
AI Agents Know About Supabase. They Don't Always Use It Right.
Supabase is releasing Supabase Agent Skills, an open-source instruction set for AI coding agents working with Supabase through Claude Code, Codex, GitHub Copilot, Cursor, and compatible Agent Skills implementations. The skill addresses recurring errors such as omitted RLS policies, hallucinated CLI commands, views missing security_invoker = true, and reliance on outdated training data by requiring documentation checks and embedding critical security guidance in SKILL.md. It also guides CLI and MCP workflows and recommends direct schema changes on local or staging databases, followed by database advisors and a migration after the schema stabilizes. Early evaluations across six scenarios found MCP + Skill scores of 67% for Claude Opus 4.6, 71% for Claude Sonnet 4.6, 88% for Codex GPT-5.4, and 71% for GPT-5.4 Mini, though the sample was small.
How Respan is scaling LLM observability with ClickHouse Cloud
Respan scaled its LLM gateway and observability platform from hundreds of daily requests to 50 million daily events, causing its original Django and Postgres architecture to stall under transaction contention at 50 to 100 requests per second. To overcome Postgres write-ahead logging bottlenecks, the team migrated the ingestion and analytics pipelines to ClickHouse Cloud. Engineers minimized row sizes by truncating large prompt text and writing structured metrics to MergeTree tables, then built incremental minute- and hour-level materialized views alongside scheduled refreshable views for user metadata. Distributed tracing spans are ingested asynchronously with analytical parent-child resolution, utilizing functions like argMax rather than the FINAL modifier to prevent costly table merges. This architectural shift allows Respan to achieve sub-second search across tens of millions of records and support customer dashboards with predictable query performance.
100,000 GitHub stars
Supabase reports reaching 100,000 GitHub stars and says eight million developers are building with the platform. Founded in 2020 by two people around a Postgres database, the project says it optimized for developers having a good time rather than for star counts, treating the milestone as evidence of a community that contributed, stayed involved, and built unexpected projects. Its open-source approach centers on supporting existing tools and communities: projects run unmodified Postgres, PostgREST handles billions of API requests daily, pgvector stores embeddings, Deno underpins Edge Functions, imgproxy handles image transformations, and Phoenix supports Realtime at almost 1.5 billion messages per day. The team says it aims to help make Postgres the default database for new internet projects, including by pursuing petabyte scale and pluggable storage.
PostgresBench: A Reproducible Benchmark for Postgres Services
PostgresBench is a public, reproducible benchmarking suite created to compare transactional performance across managed PostgreSQL providers. The benchmark uses the standard pgbench tool with a TPC-B-like write-intensive workload executing across 256 clients and 16 threads for ten-minute durations. Evaluators tested five services—including Postgres managed by ClickHouse, AWS Aurora, AWS RDS, Neon, and Crunchy Bridge—under 100 GB and 500 GB scale factors using default configurations in the us-east-2 region. Across tested configurations, Postgres managed by ClickHouse delivered higher transactions per second and lower P99 latencies due to physically co-located NVMe primary storage. These results demonstrate that remote network storage layers, such as EBS volumes, introduce significant latency overhead during sustained write and fsync operations.
Supabase Joins the Stripe Projects Developer Preview
Supabase has joined the Stripe Projects developer preview, a Stripe CLI workflow for provisioning real services and obtaining working credentials with a single command. The integration provisions a complete Supabase project—including Postgres, Auth, Storage, Edge Functions, and Realtime—without requiring dashboard interaction, while supporting both developers and AI agents through deterministic, repeatable steps. After installing the Stripe CLI and adding supabase/project, stripe projects env --sync writes credentials to a local .env file and stores them in the Stripe Project for agent access; existing Supabase users can link their account. Projects remain in the user’s Supabase account with dashboard, connection-string, and data access, and database credentials can be rotated with stripe projects rotate supabase/project. The feature is a developer preview, and Stripe and Supabase are requesting feedback through Discord or GitHub.
March 2026 newsletter
The March 2026 ClickHouse newsletter details major platform updates, community deployments, and architectural practices for high-throughput analytics. Recent product announcements include ClickHouse 26.2 featuring time-based block flushing for streaming feeds, general availability of native inverted indexes for full-text search, and the release of chDB 4 with a Pandas-compatible DataStore API. Community implementations show Cloudflare managing 1.61 quadrillion daily events, while Trigger.dev isolated multi-tenant SQL execution using a domain-specific query language that compiles safe ClickHouse queries. In data optimization techniques, Hookdeck accelerated webhook payload search from over 30 seconds to under 400 milliseconds by partitioning payload hashes into typed bucket columns and scanning iterative time windows. Additional operational guidance emphasizes setting the ttl_only_drop_parts parameter and aligning table partitioning with time units so background merges drop entire data parts cleanly without triggering mutation rewrites.
How Socialpruf built a faster, more reliable data stack by replacing Neon with Postgres managed by ClickHouse
Socialpruf, a social media performance tracking platform ingesting hundreds of posts per second, experienced analytics latency, connection dropouts, and high network transfer costs while running Postgres on Neon alongside ClickHouse Cloud. To resolve these issues, the team migrated their transactional system of record to NVMe-backed Postgres managed by ClickHouse, colocating transactional and analytical workloads. Although standard Postgres logical replication failed on their 0.5 TB dataset, they achieved continuous synchronization using PeerDB before conducting a production cutover. The migration eliminated connection dropouts, reduced network transfer fees, and improved overall Postgres query performance by approximately 30 percent. To manage thousands of concurrent connections generated by ingestion workers, Socialpruf implemented ClickHouse's bundled, horizontally scalable PgBouncer instances.
AI is redrawing the database market
Emerging artificial intelligence workloads are fundamentally transforming data platform requirements across real-time analytics, data warehousing, and observability. Conversational analytics and autonomous agents replace periodic batch queries with rapid bursts of concurrent, low-latency requests, exposing the architectural limits of traditional cloud data warehouses and standalone transactional databases. To resolve these constraints, architectures are converging on engine-level integrations between Postgres for transactions and ClickHouse for analytical scale, while unifying operational observability and business event storage. ClickHouse also incorporates turnkey agent interfaces through its acquisition of LibreChat alongside dedicated execution tracing from tools like Langfuse. These combined capabilities ensure organizations can reliably support interactive agentic workflows without incurring severe latency penalties or compounding infrastructure costs.
Log Drains: Now available on Pro
Supabase has launched Log Drains for its Pro tier, allowing users to send Supabase logs to their own logging backend and investigate them alongside the rest of their stack. The feature addresses the separation between Supabase logs and external observability tools by exporting events from Postgres, API Gateway, Auth, Storage, Edge Functions, and Realtime. Logs are sent over HTTP in batches of up to 250 entries or every second, whichever comes first, with Gzip compression when supported by the destination. Supported destinations include Sentry, Grafana Loki, Datadog, AWS S3, Axiom, and generic HTTP endpoints; users configure drains in the dashboard and can create multiple drains, with pricing of $60 per drain per project, $0.20 per million events, and $0.09 per GB of egress.
Supabase Storage: major performance, security, and reliability updates
Supabase Storage has shipped updates live in all regions for projects using Storage, targeting performance at scale, security vulnerabilities, and reliability issues. As projects reached tens of millions of objects, concurrent uploads exposed write bottlenecks, certain SQL deletes could orphan files, and a crafted path could escape the file backend’s storage root. The object-listing rewrite removes the prefixes table, six triggers, and twelve helper functions, derives folder structure from the objects table with a hybrid skip-scan algorithm, and uses cursor-based pagination; deep pagination on 60 million rows is up to 14.8 times faster without a write penalty. Direct SQL deletes are blocked by default through a statement-level trigger, while a configurable 30-second timeout supports cancellation of disconnected clients via native Postgres protocol. Idempotent migrations, TUS lock-race prevention, orphan-scanner fixes, OpenTelemetry metrics, and adapter and API bug fixes are included, with no code changes required.
Innovating with MongoDB | Customer Successes, February 2026
Modern artificial intelligence startups frequently face operational bottlenecks when deploying autonomous agents and dynamic interfaces on rigid relational schemas or restricted database engines. To address these limitations, platforms including Modelence, Thesys, Emergent Labs, and Heidi integrated MongoDB Atlas as their operational data foundation. Emergent Labs eliminated recursive schema migration failures previously encountered with PostgreSQL by adopting document structures that match native JSON generation, supporting millions of deployed applications. Meanwhile, Heidi migrated from Amazon DocumentDB to utilize MongoDB Vector Search for retrieval-augmented generation pipelines under a single unified API without requiring external search databases. These architectural transitions streamlined continuous agent iterations, eliminated operational scaling downtime, and accelerated deployment cycles across diverse production environments.
Welcome to Notion, Sequin team!
Notion welcomes the Sequin team—Eric Goldman, Anthony Accomazzo, and Carter Pedersen—following discussions about the companies’ shared view of fragmented knowledge work. Sequin built a data-syncing product that brought information from systems including Salesforce, ServiceNow, and Stripe into Postgres, giving developers flexible building blocks instead of requiring them to recreate integrations. The founders say they were drawn to Notion’s enterprise search, work on AI and agents, ambitious direction, and strong user community, while Notion describes Sequin’s experience as aligned with its plans. The announcement frames the broader problem as information scattered across documents, conversations, and tools, and points to AI and agents as a way to help teams find information, automate repetitive work, and prepare for AI-native work.
The evolution of Grab's machine learning feature store
Grab describes how it evolved its machine learning feature store as increasingly complex models began requiring high-cardinality keys and high-dimensional data. It replaced its initial Amphawa approach with feature tables: data-lake tables whose contents are atomically ingested into serving storage, with emphasis on consistency and reproducibility. The workflow stores feature tables and version-control metadata as Parquet objects in Amazon S3, validates schemas, loads data through reverse ETL into Aurora Postgres, and warms new tables through shadow reads before switching. The redesign supports contextual retrieval, atomic collection-wide updates, and isolation between reading and writing capacity; its SDK remains a thin database-driver wrapper that translates queries to SQL. Grab also chose decentralised deployments for version flexibility, rollout strategies, noisy-neighbor mitigation, and improved caching, concluding that Amazon Aurora for Postgres provides a robust, scalable, cost-effective foundation.
Heroku Postgres Upgrade Guide: Simplify Your Move to Version 17
Heroku Postgres 17 is now available, and the platform introduces a simplified, self-service path for upgrading managed databases. The new heroku pg:upgrade:* commands cover preparation, manual execution, cancellation, dry runs, and progress tracking, while email notifications report major stages and detected issues. Essential-tier databases can be upgraded directly with pg:upgrade:run; Standard-tier and higher databases can use a dry run, schedule an upgrade for the next maintenance window, or run it manually, with followers upgraded automatically. The in-place workflow uses an internal replica, leaves DATABASE_URL and other config vars unchanged, and typically reduces downtime to about five to ten minutes. The legacy heroku pg:upgrade command will be deprecated soon, although the traditional follower approach remains available.
Scaling Nextdoor’s Datastores: Part 5
Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.
Scaling Nextdoor’s Datastores: Part 4
Look-aside caching systems can become inconsistent when concurrent database updates execute cache writes out of order, allowing stale data to overwrite newer modifications. To prevent these racing writes, Nextdoor introduced a unique, monotonic db_version column to Postgres tables using database triggers that initialize version numbers on insert and increment them on update. Application updates retrieve this new version inside a transaction block and attach it as a metadata header to serialized cache values. Redis then executes custom Lua scripts, specifically set_if_version and del_if_version, to perform atomic conditional updates that reject any incoming payload with a version lower than or equal to the stored version. This serializable check ensures that out-of-order writes are dropped and the cache remains strictly aligned with the latest database state.
Scaling Nextdoor’s Datastores: Part 1
Nextdoor relies on a Django backend connected to monolithic PostgreSQL databases and Redis look-aside caches. Migrating to distributed SQL datastores proved impractical because legacy business logic depends heavily on multi-table joins that could not be rewritten. Intermediate mitigations, including read replicas and data partitioning by severing foreign keys, extended infrastructure runways but left primary databases vulnerable to load bottlenecks. Furthermore, look-aside caching and replica usage introduced data staleness risks, atomicity losses across partitioned databases, and inconsistent cache-population behaviors. To address these limitations, Nextdoor initiated an architecture redesign focused on dynamic query routing to replicas, replica-driven cache hydration, time-bounded eventual consistency, and schema-resilient cache serialization.
Optimizing Data Reliability: Heroku Connect & Drift Detection
Heroku Connect syncs Salesforce data with Heroku Postgres, but primary polling can miss changes when long-running Salesforce automations commit after the polling window advances, creating drift. The service uses secondary polls over the same bounds as primary polls to find and repair missed changes, with Bulk API or SOAP API selected according to dataset size and retries for failed polls. Recent improvements change Bulk API drift handling by creating one secondary bulk job for each primary bulk job, then adding corresponding batches instead of creating many jobs. Secondary tasks are now scheduled and executed with limited concurrency, reducing simultaneous database connections consumed while Postgres is updated and preserving capacity for applications. The post says these changes reduce load on Salesforce and improve operational reliability during heavy synchronization, while noting that Heroku Connect remains asynchronous rather than truly real-time.
Building a GPT Backed by a Heroku-Deployed API
The walkthrough shows how to connect an OpenAI custom GPT to a Node.js backend that provides natural-language access to an employee directory. It builds an Express API with a POST /search endpoint, where OpenAI’s Chat Completions API converts a user’s question into a single SQL query, the database returns rows, and a second OpenAI call turns those rows into a natural-language response. The project uses Heroku Postgres, environment configuration, bearer-key authentication, Swagger UI, and an OpenAPI specification imported into the GPT action. The deployment flow covers seeding dummy records, deploying from a GitHub-connected repository, and configuring the action to call the hosted API, while noting that data leaves Heroku’s trust boundary for OpenAI services.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.