---
title: "Delta Lake"
description: "28 posts about Delta Lake, summarised, each linking to the original."
---

# Delta Lake
> 28 posts about Delta Lake, summarised, each linking to the original.

## Articles

### [The Convergence of Open Table Formats and Open Catalogs: Catalog Commits is Generally Available](https://yomu.fyi/post/the-convergence-of-open-table-formats-and-open-catalogs-catalog-commit.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Benjamin Mathew, Michelle Leon, Lukas Rupprecht, Ryan Johnson
- Published: May 12, 2026

Catalog Commits is generally available for Unity Catalog managed Delta tables, aligning Delta with Iceberg’s catalog-oriented model and making the catalog responsible for table discovery, access, and latest table state. The change addresses three coordination problems: metadata “split brain” when engines write directly to storage, fragmented multi-engine governance, and the historical inability to coordinate atomic writes across multiple tables. With Catalog Commits enabled, Unity Catalog brokers table access through standardized APIs, keeping catalog and table state synchronized and enabling consistent authorization, holistic auditability, automated optimizations, and multi-statement, multi-table ACID transactions on Databricks. The release supports Databricks products and engines including Delta Spark, Delta Flink, Starburst Trino, DuckDB, and StreamNative, while Delta Kernel provides a shared path for connector support.


### [Unlocking the Archives: Turning Unstructured Documents into a Searchable Database for Groundwater Discovery](https://yomu.fyi/post/unlocking-the-archives-turning-unstructured-documents-into-a-searchabl.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Andres David Blandon Restrepo, Mofeed Nagib
- Published: May 11, 2026

MapAid and the Sudan Association for Archiving Knowledge (SUDAAK) needed to make nearly 700 scanned geological reports searchable and recover well data for MapAid’s WellMapr groundwater models. Databricks built a Databricks-only pipeline that renders pages as images, uses multimodal AI Functions for page classification and schema-constrained JSON, and applies targeted sampling to cut classification processing by more than 70%. Water-relevant documents receive page-by-page OCR through the Foundation Model API, with well and borehole identifiers linking information across pages before structured records capture locations, depths, static water levels, and yields. In its first full run, 654 documents and 5,570 pages were classified in under three hours; 95% received excellent or good evaluations, about half contained water data, and 299 well or borehole records were extracted.


### [Peril predicts: Precision payouts for a volatile world](https://yomu.fyi/post/peril-predicts-precision-payouts-for-a-volatile-world.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Anindita Mahapatra, Timo Roest, Justin Monaldo
- Published: May 5, 2026

Parametric insurance pays automatically when objective thresholds—such as wind speed, rainfall, or earthquake magnitude—are met, replacing lengthy loss assessments with event-based payouts. Modern catastrophe modeling combines geospatial data, weather observations, engineering insights, and historical loss records to estimate extreme-event probability and impact and define reliable triggers. Operationalizing these programs requires near-real-time processing of satellite imagery, weather feeds, exposure datasets, and model outputs. Databricks’ Geospatial Lakehouse unifies those sources on Delta Lake while Spark runs spatial joins and catastrophe modeling pipelines. When thresholds are crossed, the system identifies eligible policies, calculates tiered payouts, and surfaces results through dashboards, Lakehouse Apps, and Genie; aerial imagery and multimodal AI can support damage validation and fraud detection, while Unity Catalog governs access and Delta Sharing supports controlled data exchange.


### [Approximate answers, exact decisions: New sketch functions for analytics](https://yomu.fyi/post/approximate-answers-exact-decisions-new-sketch-functions-for-analytics.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Daniel Tenedorio, Kent Marten, Gengliang Wang, Chenhao Li
- Published: Apr 29, 2026

Databricks now supports four sketch function families, built on Apache DataSketches, for replacing expensive exact percentiles, distinct counts, set operations, top-K rankings, and metric aggregations with bounded-memory approximations. KLL sketches answer quantiles, Theta sketches support unions, intersections, and differences, approximate top-K sketches track frequent items, and Tuple sketches combine distinct counting with aggregated metrics such as customer revenue. The sketches can be built during ETL, stored as columns in Delta tables, and merged on read, allowing dashboards and streaming pipelines to avoid rescanning raw data; SQL, DataFrame, and Structured Streaming pipelines are supported. The stated trade-off is configurable 1–2% relative error, with the source citing a 1000x speedup for applicable workloads. Exact methods remain appropriate for financial auditing, compliance reporting, and other precision-required uses.


### [Operational databases: How they work and when to use them](https://yomu.fyi/post/operational-databases-how-they-work-and-when-to-use-them.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 24, 2026

Operational databases, or online transaction processing (OLTP) databases, support day-to-day operations by processing real-time CRUD transactions with low latency, high concurrency, and ACID guarantees. The guide contrasts them with data warehouses and OLAP systems, which favor historical analysis, complex aggregations, and read-heavy workloads, while describing ETL, CDC, and operational data stores in the OLTP-to-OLAP pipeline. It argues that traditional OLTP systems can struggle with real-time analytics and AI because of siloed data, structured-data constraints, rigid schemas, scaling limits, and missing governance features. Databricks Lakebase is presented as a hybrid solution using separate storage and compute, serverless Postgres, elastic scaling, branching, shared lakehouse storage, and open formats. The proposed workflow connects existing OLTP systems through CDC or streaming into Delta Lake for fresh SQL, BI, ML, and AI access.


### [Stop hand-coding change data capture pipelines](https://yomu.fyi/post/stop-hand-coding-change-data-capture-pipelines.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Matt Jones, Zoé Durand, Phoebe Weiser, Bilal Aslam, Ray Zhu
- Published: Apr 22, 2026

Hand-coded change data capture (CDC) and slowly changing dimension (SCD) pipelines become fragile as they handle out-of-order updates, duplicate events, deletes, late-arriving data, retries, backfills, and schema evolution. The post presents AutoCDC in Lakeflow Spark Declarative Pipelines as a declarative alternative: engineers specify keys, sequencing, delete conditions, and SCD Type 1 or Type 2 semantics while the platform manages ordering, state, deduplication, versioning, and incremental processing. Examples show a current-state table that keeps Mercedes’s latest city and removes deleted user 123, alongside a Type 2 table that preserves validity windows through \_\_START\_AT and \_\_END\_AT. The supplied comparison reports roughly 6–10 lines of declarative definition versus 40–200+ lines of custom logic, and says AutoCDC supports both change data feeds and snapshot-based CDC.


### [Real-Time Decisioning for AI Agents: Why you Need a Customer Context Layer First](https://yomu.fyi/post/real-time-decisioning-for-ai-agents-why-you-need-a-customer-context-la.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Alex Dean
- Published: Apr 21, 2026

Real-time decisioning for AI agents depends on more than customer records: it requires a customer context layer that captures what people are doing now and connects that behavior to identity. The post positions the data platform as the center of a composable martech architecture, with agents and applications operating on shared data in open formats such as Apache Iceberg and Delta Lake. Snowplow’s described approach places structured event collection, schema validation, enrichment, and identity resolution before data reaches the platform, while operating in the customer’s cloud environment. It distinguishes historical profiles from granular behavioral event streams and argues that agent interaction outcomes should return to the foundation as first-class events. The resulting four-stage loop—collect, resolve and enrich, serve, and learn—supports simultaneous real-time and historical context, with decision quality dependent on source data coherence.


### [Concurrency Control in DBMS: How Locking, MVCC and Optimistic Strategies Keep Data Consistent](https://yomu.fyi/post/concurrency-control-in-dbms-how-locking-mvcc-and-optimistic-strategies.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 21, 2026

The guide explains how database management systems coordinate simultaneous transactions to preserve consistency and serializability, preventing dirty reads, lost updates, non-repeatable reads and phantom reads. It compares lock-based control, including shared and exclusive locks and two-phase locking, with MVCC, which gives transactions consistent snapshots and lets readers and writers proceed without blocking. The discussion covers pessimistic and optimistic strategies, isolation levels, deadlocks, retries, lock-granularity trade-offs and MVCC’s storage and garbage-collection costs, including PostgreSQL’s VACUUM. For lakehouse workloads, Delta Lake uses a read-validate-commit process with optimistic concurrency, snapshot isolation and conflict detection for concurrent writes. The source presents Databricks Lakebase as a managed implementation offering optimistic concurrency control, snapshot isolation for reads and write-serializable isolation for writes without lock or deadlock management.


[Newer posts](https://yomu.fyi/topic/delta-lake.md)