Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
Loading…
Topic
Databases
38 posts about Databases. Every summary links to the original.
- 38 posts
- 14 companies
- 5 explainers
See the idea
Databases explainers
Apache Iceberg vs. Hive: The Table Format Trade-off
Apache Iceberg replaces Hive's partition-based metadata with a versioned catalog layer, trading operational simplicity for ACID guarantees, schema evolution, and cross-engine compatibility.
How Apache Arrow Moves Data Without Copies
Apache Arrow is a columnar in-memory format that lets databases, analytics engines, and Python libraries share data without serialization overhead or format conversion, cutting memory peaks and speeding up common workflows.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Lakebase Postgres branch-based restores for fast recovery at scale
Traditional Postgres point-in-time recovery is slow because compute and storage are coupled, requiring provisioning a new instance, restoring a snapshot from object storage to disk, and replaying WAL, leading to hours of downtime for large databases. In a survey of developers running 1TB+ production Postgres, 59% had a critical production failure and 30% were down for over three hours. Lakebase Postgres decouples compute and storage using safekeepers, a pageserver, and object storage, storing database history as an immutable timeline. A restore becomes a branch at a point in history, a metadata operation that does not copy data, making it near-instant regardless of database size. This eliminates data copies and WAL replay, so restoring a 100 TB database is as fast as restoring 10 GB, and agents can use restores as ordinary tool calls for versioning and undo.
Your Database Just Became Your AI Control Point
Agentic AI transforms the database from a passive record store into an active control point that serves real-time context and enforces policy at the moment an agent acts. According to IDC’s April 2026 survey of 715 organizations, 30% already run agentic AI in production and another 50% have established spending plans. Real-time data delivery is now baseline: 64.6% of enterprise use cases require real-time or near-real-time delivery, and 91.2% of streaming scenarios need end-to-end latency under 200 milliseconds. IDC identifies four requirements for agentic AI: a real-time operational foundation for unstructured data, elastic scale for bursty agent traffic, accurate always-current retrieval via integrated vector databases, and a governed runtime with durable memory and identity-bound policy. By 2029, IDC predicts 60% of enterprise data platforms will unify transactional and analytical workloads in hybrid processing architectures to power agentic AI.
What is a Relational Data Model?
The relational data model organizes structured data in tables of rows and columns, with SQL used to access and manage it. Introduced by Edgar F. Codd in 1970, it uses schemas, domains, keys, constraints, and relationships to define and connect data. A relation schema specifies a table's structure, while a relation instance is the data held at a particular moment; primary, candidate, super, foreign, and composite keys support identification and referential integrity, and one-to-one, one-to-many, and many-to-many relationships model associations between tables. Functional dependencies inform normalization, whose 1NF, 2NF, and 3NF stages reduce redundancy and insertion, update, and deletion anomalies, though heavier normalization can require more joins. The model is used in banking, e-commerce, healthcare, education, and enterprise systems, where constraints and ACID transactions help preserve accuracy and reliability.
What is a Cloud
A cloud-based database management system stores, manages, and queries data on third-party cloud infrastructure instead of local or on-premises servers. It retains capabilities such as query optimization, security, transactions, and support for relational SQL, NoSQL, data warehouse, and in-memory workloads, while operating across distributed nodes. Organizations can self-manage database software on cloud virtual machines or use DBaaS, where the provider manages the software and infrastructure; connections typically use SQL, APIs, or web consoles. Virtualized resources can scale, and providers may handle patching, backups, maintenance, and failover, while replication across availability zones supports recovery and availability. The comparison with on-premises systems covers ownership, scalability, cost models, maintenance, accessibility, control, compliance, total cost, and emerging capabilities including AI integration, vector search, autonomous operation, multicloud, hybrid, and serverless architectures.
Database Branching in Postgres: Git
Shared staging databases often drift as schemas diverge, test data accumulates, and developers avoid creating isolated environments because traditional pg_dump copies are slow and expensive. Databricks Lakebase Postgres addresses this with database branching, which creates fully isolated environments from a parent’s exact schema and data at a point in time. Branches use copy-on-write: they share underlying versioned storage until a branch changes, then record only new data, while each branch has independent compute and a separate connection string. Idle branches scale compute to zero, and storage cost tracks changes rather than the full database size. This supports per-developer, per-pull-request, per-test-run, and recovery workflows, including point-in-time branching within the restore window, with the source describing branch creation through the console, CLI, or API in under a minute.
Move and Scale Faster with FGA Permissions Index
Auth0 FGA Permissions Index is announced in Early Access as a way for B2B SaaS applications to enforce permissions while handling expanding AI-agent activity and complex relationship graphs. Customers select relations used for search and filtering, typically high-cardinality relations such as can_view, and the index precomputes the access those relations imply. It streams the resulting data into the customer’s database alongside application data, allowing indexed permission checks through SQL JOINs instead of repeated authorization API calls or runtime graph traversal. Computation moves to relationship-write time, while relations outside the index continue to be evaluated by FGA in real time. The announcement positions this design as a path to consistently low-latency filtering at enterprise scale, with permission decisions described as precomputed, traceable, and compliant.
Managed Postgres: What Lakebase Actually Takes Off Your Plate
Managed Postgres varies from OS patching alone to operating scaling, failover, backups, recovery, and related database tooling. Lakebase Postgres applies this model to serverless infrastructure, offering automatic scaling, scale-to-zero, PostgreSQL compatibility, automatic snapshots, point-in-time recovery, branching, built-in PgBouncer, and support for extensions including pgvector and PostGIS. Within a region, patching, scaling, failover, and backups run automatically, while encryption, access control, audit visibility, connection pooling, and lakehouse integrations are also addressed. The discussion also stresses checking PostgreSQL versions, configuration, extensions, and migration methods before moving an existing database. Cross-region disaster recovery remains the main exception: it is in Private Preview for AWS, with periodic replication, manual failover, and customer-managed recovery procedures.
SQL Data Types: Reference and Best Practices
SQL data types define what values columns accept, how those values are stored, and how databases process them, making type selection central to schema design. The reference covers numeric, character, date/time, and specialized types, noting that MySQL, PostgreSQL, SQL Server, and Oracle differ in naming, precision, limits, and storage requirements. It recommends choosing the smallest type that safely accommodates expected values, checking range, precision, NULL requirements, string length, and time-zone needs before deployment. Examples contrast CHAR with VARCHAR, DECIMAL with FLOAT, DATE with TIMESTAMP, and integer sizes such as TINYINT, INT, and BIGINT; indexes and distributed engines also benefit from right-sized data. Sample data, execution-plan analysis, storage measurements, schema audits, and documentation help validate choices, identify waste or overflow risks, and preserve reasoning behind edge cases.
MySQL CDC connector for ClickPipes is now Generally Available
MySQL CDC connector for ClickPipes is now Generally Available, enabling replication from MySQL and MariaDB into ClickHouse Cloud for analytical workloads. The release adds automatic parallel snapshotting: ClickPipes selects a logical partition key from a table’s primary key, including string and UUID types, and distributed processing reduced a benchmarked 1TB snapshot from over a day on one thread to about an hour across 32 workers. Production safeguards now require GTID-based replication and at least 72 hours of binlog retention, while new CPU, memory, network, and source, destination, and end-to-end lag metrics improve troubleshooting. MySQL CDC ClickPipes also support OpenAPI and Terraform 3.14.0+, are available across all ClickHouse Cloud service tiers, and began billing on September 1, 2026.
From Chaos to Control: Addressing Shard Distribution Challenges in M3DB with Subclusters
M3DB’s sharded placement algorithm lets replicas spread across nodes subject primarily to isolation-group constraints, but large clusters experienced broad failure and maintenance impacts. The subclustered placement algorithm partitions nodes into fixed-size, self-contained groups, each owning a distinct shard slice while preserving replica separation across isolation groups. It requires equal instance weights, scaling in multiples of the subcluster size, a replica factor that cannot change through AddReplica, and instancesPerSubcluster divisible by RF. During scaling, a greedy donation strategy simulates removing each candidate shard, ranks the resulting per-node skew, and chooses the shard that leaves the donor most balanced, avoiding a second rebalancing move. The resulting model contains recovery within subclusters, enables parallel maintenance, and reduces arbitrary shard sharing from O(cluster) to O(subcluster), while the original algorithm remains suitable for smaller clusters or impractical scaling steps.
The 9 best tools for your early-stage startup tech stack in 2026
Early-stage startups face a crowded software market and limited budgets, making technology selection a question of leverage rather than completeness. The guide proposes three selection principles: choose tools that scale with usage, integrate with the existing stack, and solve a clearly defined problem. It then recommends Stripe for payments, Slack for collaboration, Granola for meeting notes, Fin and Intercom for customer service, Linear for planning, Datadog for observability, MongoDB for data, Rippling for workforce administration, and Apollo for outbound sales and CRM. The examples emphasize operational coverage, from Stripe’s subscriptions and MongoDB’s flexible schema to Fin’s stated ability to resolve up to 76% of support conversations. The Fin Startup Pack is presented as a way to reduce adoption costs through partner credits and discounts, including more than $500,000 in credits and specified offers for each recommended tool.
The Architecture of Schemaless, Uber Engineering's Trip Datastore Using MySQL
Schemaless is Uber's scalable, fault-tolerant datastore whose immutable cells are identified by row key, column name, and ref key, while JSON content remains schemaless and supports eventually consistent secondary indexes. Its worker nodes accept HTTP requests, route them to storage nodes, and aggregate results, while storage data is partitioned into typically 4096 shards, each replicated across one master and two minions in multiple data centers. Writes go to the master and replicate asynchronously, with idempotent retries, circuit breakers, failover, and configurable reads from masters or minions. To reduce loss during master failure, buffered writes persist each request to a primary and secondary cluster, though buffered data may not be immediately readable. Each shard maps to a MySQL database, where an auto-increment added_id primary key supports linear insertion and trigger processing, and MessagePack with ZLib compresses cell JSON blobs.
Migrating Counter Service storage: Design choices and learnings
Grab’s Counter Service supports time-windowed fraud queries at tens of thousands of QPS and about a billion requests per day, but an infrastructure review prompted migration from a wide-column database to Aerospike. The team first separated Rust business logic from storage, then added an enum-dispatched facade with Single, WithShadow, and WithSplit modes so traffic could be compared and shifted without downtime. They also redesigned the writer path around Aerospike, including a map-based schema and a single atomic increment, while testing index placement and reverting from an NVMe-backed primary index after hot-key p99 spikes. Production p99 read latency improved by roughly 50%, disk usage fell from about 3 TB to 1 TB, and per-node cost dropped 45–50%; the rollout reported zero downtime and no data-integrity issues.
Simplifying Data and Product Integrations with a Data Abstraction Layer
Uber built a domain-agnostic Data Abstraction Layer (DAL) to decouple data consumers from changing physical datasets and simplify advertiser reporting. The DAL is an RPC service that accepts a logical table, schema, time range, and filters; logical tables define the interface, while configuration describes locations, relationships, availability windows, and rollup transforms. It generates database-specific queries, runs them in parallel across systems such as an internal OLAP database, Docstore, and Apache Hive, and assembles responses with aggregation and rollup behavior determined by the request. Removing a dimension can produce an account-level response with metrics automatically rolled up, while mixed realtime and daily tables can cover different portions of a requested time range. For advertiser reporting, the approach reduced new report delivery from several weeks or months to under two days and enabled dynamic filters, report builders, and third-party API integrations.
Enabling Evolutionary Database Development: database branching with Lakebase
Evolutionary database development treats migrations as first-class CI/CD artifacts, but shared development databases still limit per-developer isolation and fast, realistic feedback. The post follows Jen as she adds location, batch, and serial fields to inventory, coordinating application changes, production-data preservation, schema migration, and tests for storing, reading, and displaying the new values. It contrasts shared databases and simplified local substitutes with Databricks Lakebase copy-on-write branches, created in one second with zero storage at creation, including for a terabyte-scale production database. Jen pairs code and database branches, while CI creates a temporary branch, validates an idempotent and reversible migration, runs application and database tests, and posts a schema diff, enabling combined review and shifting DBA attention toward design and maintainability.
10 trillion samples a day: Scaling beyond traditional monitoring infra at Databricks
Databricks’ monitoring infrastructure now tracks 5 billion active timeseries in real time and ingests more than 10 trillion samples daily, exposing scalability, reliability, cost, and operability limits in its older stack. The replacement, Pantheon, is a fork of CNCF Thanos deployed across more than 160 instances in about 70 regions and three cloud providers; tiered storage, differentiated memory retention, isolated replicated Receive groups, multitenancy, and a custom control plane support automated scaling and recovery. Pantheon’s largest instance holds about 300 million in-memory timeseries and handles nearly 1,000 PromQL queries per second, while migration reduced annual cloud costs by millions and monitoring downtime by roughly five times. For high-cardinality troubleshooting, Hydra preserves raw metrics in Delta tables, exposes them through Grafana and SQL, and unifies metric semantics across aggregated and raw paths, with freshness improvements planned.
Database modeling: a practical guide to techniques and best practices
Database modeling defines the structure, relationships and constraints that keep systems coherent across transactional, analytical and flexible workloads. The guide presents a three-phase design process: conceptual modeling captures entities and relationships, logical modeling adds attributes, keys, cardinality and normalization, and physical design translates that model into tables, indexes, constraints, partitioning and other platform-specific choices. It compares relational, document/NoSQL, dimensional, hierarchical and network models, linking each to data shape, workload and consistency requirements. It recommends explicit keys, consistent naming, appropriate normalization, query validation and version-controlled DDL, while warning against premature physical design, missing foreign keys, unclear NULL handling and poorly balanced schemas. It concludes that Databricks Lakebase combines a transactional database engine with the Databricks Lakehouse Architecture so one model can support operational and analytical workloads.
Are LLM agents good at join order optimization?
Join ordering is difficult because relational optimizers choose among exponentially many plans and depend on potentially inaccurate cardinality estimates. Databricks and UPenn researchers built an LLM agent that uses structured outputs to generate valid join reorderings and executes candidates through a tool returning runtime and intermediate subplan sizes. It can test up to 50 iterations, while the evaluation tested 15 rollouts per query across 113 Join Order Benchmark queries after duplicating each dataset row ten times. With a frontier model, the best tested orders improved geometric-mean latency by 1.288 times and reduced P90 latency by 41% versus the standard Databricks optimizer. A five-way example links its gain to LIKE predicates that are exceptionally difficult for cardinality estimators.
Concurrency Control in DBMS: How Locking, MVCC and Optimistic Strategies Keep Data Consistent
The guide explains how database management systems coordinate simultaneous transactions to preserve consistency and serializability, preventing dirty reads, lost updates, non-repeatable reads and phantom reads. It compares lock-based control, including shared and exclusive locks and two-phase locking, with MVCC, which gives transactions consistent snapshots and lets readers and writers proceed without blocking. The discussion covers pessimistic and optimistic strategies, isolation levels, deadlocks, retries, lock-granularity trade-offs and MVCC’s storage and garbage-collection costs, including PostgreSQL’s VACUUM. For lakehouse workloads, Delta Lake uses a read-validate-commit process with optimistic concurrency, snapshot isolation and conflict detection for concurrent writes. The source presents Databricks Lakebase as a managed implementation offering optimistic concurrency control, snapshot isolation for reads and write-serializable isolation for writes without lock or deadlock management.
Take Control: Customer-Managed Keys for Lakebase Postgres
Lakebase Customer Managed Keys (CMK) gives regulated enterprises control of the root of trust through customer-owned keys from AWS KMS, Azure Key Vault, or Google Cloud KMS. Because Lakebase separates persistent storage from elastic Postgres compute that can scale to zero, CMK protects both layers and their caches rather than storage alone. Its hierarchical envelope-encryption model uses the CMK as the root, transient Key Encryption Keys (KEKs) to wrap Data Encryption Keys (DEKs), and unique DEKs for each data segment, while plaintext CMKs remain in the customer’s KMS. Account Admins bind a configured key to a workspace, after which new projects inherit the CMK; rotation needs no re-encryption or downtime, revocation terminates compute instances, and audit services record cryptographic operations.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.