Loading…
Delta Lake
28 posts about Delta Lake. Every summary links to the original.
How Unity Catalog managed tables bring interoperability, performance, and unified governance to the Lakehouse
Unity Catalog external access to Unity Catalog managed Delta tables is now in Public Preview, allowing external engines to create, read, and write while governance remains centralized. The announcement addresses the previous multi-engine trade-off: external tables enabled access but lacked managed-table performance optimizations and governance guarantees. Catalog commits coordinate writes through Unity Catalog, making it the source of truth for table state and enabling safe external writes, multi-statement transactions, and auditing of external operations. Predictive Optimization cleans storage, collects query statistics, and selects Liquid clustering columns as query patterns change; the post says these capabilities can deliver up to 50% storage cost savings and 20x faster queries. Support includes Spark, Flink, Starburst, DuckDB, and StreamNative, with open APIs and Delta Kernel extending integrations across Databricks UC and UC OSS.
Alex Jiang, Tathagata “TD” DasNavigating a Synapse Migration to Databricks
The post presents migration from Azure Synapse to Databricks as a way to reduce platform complexity while supporting data engineering, analytics, machine learning, real-time pipelines, and AI applications. It distinguishes migration paths for Dedicated SQL Pools, serverless SQL, Spark Pools, orchestration, governance, and downstream BI connectivity, noting that accumulated T-SQL logic and dependencies require the most redesign, testing, and validation. The proposed programme uses discovery and assessment with Lakebridge Profiler and Analyzer, then selects an architecture, sequences workloads, pilots a real use case, and scales migration in waves. It recommends automating code conversion, reconciling row counts, aggregations, hashes, and tolerances, avoiding direct Synapse distribution-to-Delta partition mappings, and planning compaction, enablement, and rollback. Reported examples include Casey’s halving operational data delivery times from eight hours to four and Italgas cutting workload costs by 73%.
Olga Romanova, Johannes OehlerHow to Evaluate an Enterprise Analytics Platform
Enterprise analytics platform evaluations often overemphasize dashboard interfaces, although the larger decision concerns whether analytics, AI and agents share data, semantics and governance. The post distinguishes point solutions from a unified platform and proposes seven evaluation criteria: workload fit, architecture and openness, governance and compliance, performance and scalability, adoption and usability, AI and ML readiness, and total cost of ownership. It recommends mapping current and three-year workloads, testing production-scale data with realistic concurrency, measuring p95 latency, and examining governance, usability, contracts and operational complexity in a proof of concept. Lakehouse architecture, open formats such as Delta Lake and Apache Iceberg, and shared controls are presented as ways to reduce context gaps; Databricks is offered as a practical example using Unity Catalog, Genie and Agent Bricks. The conclusion favors a weighted, three-year assessment over a feature comparison.
Databricks StaffHow Imperial College London is accelerating dementia research with a modern data platform
Imperial College London’s Care Research and Technology Centre (CR&T) needed to scale Minder, a platform that combines in-home sensors, sleep monitors, and electronic health records to support dementia care and research. Growing data volumes caused competing workloads, tightly coupled storage and compute, and limited access for researchers and clinicians. The re-architected platform validates IoT data through Kubernetes, stores it in Delta Lake on Azure Data Lake Storage using bronze, silver, and anonymized gold layers, and adds Databricks analytics with Unity Catalog governance while retaining FHIR-based EHR interoperability and Kubeflow for model deployment. Research-to-production workflows, reusable pipelines, dashboards, and timestamped data improve collaboration, accessibility, and reproducibility. During migration, uptime remained 100%; new sources integrated in as little as one month, model development fell to about one month, and insights reached 581 people living with dementia in five months.
Kacey HertanData Lake vs. Cloud Data Warehouse: A Practical Guide for Data Scientists
The guide contrasts data lakes and cloud data warehouses for storing and querying data at scale. Data lakes retain raw structured, semi-structured, and unstructured data in low-cost object storage with schema-on-read, while warehouses enforce schema-on-write for structured analytical workloads. Lakes fit petabyte-scale machine learning, data science, and undefined future use cases; warehouses fit fast, concurrent SQL for dashboards, reporting, and operational analytics. It describes Bronze, Silver, and Gold zones, with Parquet and ORC supporting columnar scans and open-format portability. For teams combining ML and BI, lakehouses use Delta Lake, Apache Iceberg, or Apache Hudi to add ACID transactions, schema enforcement, and quality monitoring to lake storage without duplication; catalogs, staged checks, and access controls help prevent data swamps.
Databricks StaffIntroducing OpenSharing: the Next Evolution of Delta Sharing for the Agentic Era
OpenSharing is presented as the next evolution of Delta Sharing, extending an open zero-copy data-sharing protocol from tables and files to models, agents, semantic context, unstructured data, and reusable AI logic. The protocol is now an independent open-source project hosted by the Linux Foundation, while Databricks OpenSharing adds Unity Catalog governance and audit logging, Marketplace discoverability, and enterprise features. Genie Agent Sharing supports governed AI experiences across organizational boundaries, with controls for proprietary instructions, data access, daily prompt quotas, and row exports. SecureConnect removes per-recipient firewall changes through a Databricks-managed proxy, while Global Distribution uses local replicas to reduce egress fees and latency. The launch also supports Apache Iceberg REST Catalog API, external catalogs, and on-premises storage partners; providers define shares in Unity Catalog, recipients query live data through existing tools, and governance enforces access controls.
Huey Han, Harish Gaur, Akram Chetibi, Mengxi ChenWhat’s new with Unity Catalog at Data + AI Summit 2026
At Data + AI Summit 2026, Unity Catalog announcements position the catalog as a runtime governance layer for enterprise data and AI, organized around control, context, and choice. Control additions include Unity AI Gateway for governing models, agents, MCP services, skills, and tools; contextual service policies can allow, deny, or require approval for runtime actions, while budgets, hard caps, tracing, and guardrails address spend, investigation, and safety. Context additions include Glossary and Domains for business meaning and scoped asset organization, plus Metrics that standardize KPIs for SQL, BI tools, APIs, and agents; Genie Ontology is described as a continuously learned enterprise context layer. Choice additions span cross-cloud and cross-region addressability, managed disaster recovery, Delta and Iceberg interoperability, multimodal and geospatial types, and open sharing of data, AI assets, and applications across organizations.
The Unity Catalog Product and Engineering TeamAccelerate search queries with full-text search indexes on Databricks
Databricks introduces full-text search indexes in Beta on Databricks Runtime 18.2 to accelerate substring and keyword queries on large open-format tables without changing their layouts. The indexes tokenize text columns into a compact lookup structure mapping tokens to matching rows; at query time, the engine uses it to identify candidate files and skip most of the table. They are maintained asynchronously, require no query hints, preserve complete results when stale by scanning indexed and non-indexed data as needed, and support Unity Catalog managed Delta and Iceberg tables on serverless and classic compute. A Trust and Safety team reported a substring search running more than 100x faster on a petabyte-scale table, while Liquid clustering remains complementary because it optimizes column-value filters rather than text within fields.
Yu Xu, Yingyi Bu, Ivan VezilićWhat is an open lakehouse? Open data standards, explained.
The piece defines an open lakehouse as a lakehouse whose storage, table format, processing engine, catalog, and ML and AI tooling use open standards and remain interchangeable. It contrasts this architecture with warehouses, lakes, and proprietary lakehouses, emphasizing low-cost object storage, ACID transactions, governance, schema guarantees, and the ability to change engines without rewriting data. Its reference stack combines open table formats such as Delta Lake and Apache Iceberg with Apache Parquet, Apache Spark, Unity Catalog, and MLflow, while allowing engines including DuckDB, Trino, and PyIceberg to work on the same data. The article also distinguishes open standards from open-source code, explains that a table format is only one layer of the stack, and states that the components can be self-hosted or consumed through a managed service.
Lisa CaoIngesting the Milky Way: Petabyte-Scale with Zerobus Ingest
The post benchmarks Zerobus Ingest, a managed, serverless, push-based service that writes producer data directly to Delta tables governed by Unity Catalog, against a petabyte-scale telemetry workload. Using NASA’s NEOWISE dataset and Locust, the test modeled fan-in from 2,048 concurrent streams, using Protocol Buffer 2 data over approximately 24 hours. The design replaces static partition-based ordering with stream-connection ordering, allowing heuristic routing across pods, dynamic partitioning, and autoscaling while existing streams drain. It also uses zeroparser, a zero-copy protobuf decoder whose design relies on Rust’s lifetime system and supports dynamic descriptors at about 1 GB/s per CPU core. The test sustained 12 GB/s to one table, ingested 1.04 trillion rows, and reached 1 petabyte within 24 hours; Zerobus Ingest is generally available, with additional APIs on its roadmap.
Aleksandar Tomić, Victoria Bukta, Nikola Obradović, Danilo Najkov, Branko Grbić, Milos MilovanovicGeospatial Unbounded: Spatial SQL GA with AI/BI Maps, Delta Sharing, and Iceberg v3
Databricks announces Spatial SQL as Generally Available, positioning its platform for geospatial analysis without separate spatial databases, warehouses, and mapping tools. It supports native GEOMETRY columns in Delta or Iceberg, more than 90 OGC-compliant ST_* functions, spatial joins, and boolean set operations. AI/BI dashboards can render Geometry and Geography columns as maps, while Genie can generate spatial queries and dashboards and respect Unity Catalog row filters. Geo columns are supported by Delta Sharing, and Databricks can read and write managed Iceberg tables or read externally written Iceberg tables with geospatial types in Iceberg v3. Benchmarks show eight of twelve SpatialBench queries improved since Public Preview, with gains from 20% to 15X, while areal boolean operations are twice as fast on average versus prior versions.
Kent MartenAnnouncing the winners of the 2026 Databricks Customer Awards
The 2026 Databricks Customer Awards recognize organizations and leaders using the Databricks Data + AI Platform across eight categories and four regions. The announcement names winners including Applied Materials, Virgin Atlantic, Fonterra Co-operative Group, Telefónica | Vivo, Virtue Foundation, Octopus Energy, Axpo, Atlassian, Wassym Bensaid at Rivian and Volkswagen Group Technologies, and Kenan Colson at Lippert. Examples include Applied Materials’ move from a Hadoop-based data lake to a governed lakehouse, with 1,500-plus analysts, more than 100 production machine-learning models and faster pipeline development, while Fonterra centralized data to improve supply-chain and compliance work. Rivian unified vehicle, factory and enterprise data, consolidated legacy systems into Delta and Unity Catalog, and designed for 500–600 petabytes; Lippert deployed AI tools across customer care, finance and HR. The announcement presents data and AI adoption as a cross-functional operating model.
Sara SteffenTransforming solar and wind maintenance reports with Genie and AI agents
Plenitude and Databricks built an agent-based system that turns solar and wind plant maintenance PDFs into structured data for cross-plant analysis. Event-driven ingestion uses Databricks Jobs and the ai_parse_document AI Function to extract text, tables, figures, and metadata, then stores page- and object-level JSON records in Delta Lake with coordinates, version history, and links to source reports. A Genie space uses Unity Catalog metadata, knowledge-store instructions, and SQL generation to answer natural-language questions, produce visualizations, and export results, while Agent Bricks can orchestrate multi-step workflows and downstream actions. The design also applies automatic liquid clustering to dynamic queries and row-level security to restrict results by country. The resulting data layer supports historical trends, plant comparisons, recurring-fault analysis, and a foundation for predictive maintenance, although the source frames predictive use as a future improvement.
Maria VallarelliBeyond parsing X12: Closing the gap for revenue cycle workflows in healthcare
Healthcare X12 pipelines can normalize remittance and enrollment data without giving medical billers a place to act on denials, leaving short-pays in SQL queries and spreadsheets while filing deadlines approach. Genpact and Databricks address that workflow gap with a two-layer operational workbench inside Databricks: production extensions to the open-source x12-edi-parser and a secure web application over Databricks SQL. The engine adds contextual 834 enrollment tracking and decoded 835 CAS adjustments, while the interface provides claim and service-line drill-downs, denial queues, dashboards, quality gates, and appeal or correction drafting. Live Unity Catalog gold-view queries avoid ETL shadow copies and synchronized caches; PHI remains inside the secure perimeter, with row-level security and audit logging for reveals and workflow changes. The proposed next step is Claude-based appeal drafting through Databricks Foundation Model APIs, and a two-week configuration-based trial uses an organization's schema and twenty existing denials.
Aaron Zavora, Neel ShapurPractical Data Lakehouse Examples and Use Cases
The article presents practical data lakehouse patterns for streaming analytics, IoT pipelines, machine learning workflows, and enterprise reporting, addressing the gap between theoretical definitions and deployable examples. It explains how a lakehouse combines low-cost, schema-flexible object storage with schema enforcement, ACID transactions, data versioning, lineage tracking, and query performance, allowing SQL and ML workloads to use shared open-format tables. Examples include second-level fraud detection, medallion-based historical analytics, predictive maintenance from sensor data, and governed customer 360 profiles. The implementation guidance covers raw storage and partitioning, centralized catalogs, decoupled compute, role-based access, time travel, migration coexistence, SLAs, observability, and lifecycle policies. Together, these patterns are presented as a unified architecture that reduces duplication and data movement while supporting governed analytics at scale.
Databricks StaffDebunking 8 data layout myths: why Liquid Clustering outperforms partitioning
Partitioning has long been the standard physical layout for Hadoop- and Hive-era processing, but the post argues that changing Lakehouse workloads make fixed choices brittle. Liquid Clustering treats keys as guidance for file organization, allowing layouts to evolve, supporting low- and high-cardinality filters, maintaining file sizes, and enabling row-level concurrency. Using transaction-log metadata and per-column statistics, the post says modern Delta and Iceberg engines prune files rather than directories, and reports 35% lower clustering time, 22% faster queries, roughly 90% faster metadata-only deletes, and 23-minute planning for a 10-PB table. Conversion examples include Bolt’s 138% higher write throughput and up to 63% faster reads, while a 1.1-PB workload saw 5.9x faster queries, 86% fewer bytes read, and a 27% smaller table after clustering by date, hour, source, and id.
Jeffrey Gong, Yu Xu, Rahul MahadevYou’ve built the media products, now make them personalized
Media companies may have launched streaming services, digital editions, and mobile apps, yet still face a digital product intelligence gap: product teams must answer behavioral data questions quickly enough to personalize experiences and improve engagement. Databricks Genie gives nontechnical product leaders a conversational interface that translates natural-language questions into SQL queries, visualizations, and actionable insights over governed enterprise data, without requiring code or analyst handoffs. The agent queries governed Delta Lake tables fed by streaming clickstreams, views, and session signals, combining event-level behavior, A/B tests, audience segments, and cross-platform data in conversational answers. The source says internal benchmark accuracy improved from 32% to over 90% through multi-LLM orchestration, specialized knowledge search, and parallel reasoning, and presents Genie as reliable enough for production personalization decisions.
Elena TesserPipelineIQ: Forward‑Looking Sales Intelligence That Drives Action
PipelineIQ addresses the administrative drain and unreliable revenue predictability caused by incomplete, inconsistent, and backward-looking CRM data in B2B sales. Rather than build another forecasting system, it applies prescriptive analytics to identify forward signals and produce immediate actions for reps and managers. Built on Databricks, it uses Foundation Model APIs, Unity Catalog, Delta Lake, and AI/BI Dashboards; its confidence scorer sends CRM fields to ai_query() with a Gemma 3 12B model, scores eight MEDDPICC dimensions from 0–10, and limits missing fields to scores of 3 or below. Weighted confidence is refreshed daily, with a fail-safe override to Low when a use case has more than three active blockers. Dashboards and Genie queries connect evidence-based risk explanations, remediation steps, and portfolio views to sales execution.
Sam Le Corre, Dael Williamson, Luis HerreraExpanded interoperability with Unity Catalog Open APIs
Unity Catalog’s expanded Open APIs address data silos by letting organizations use multiple compute engines while retaining centralized governance and a single copy of data. In beta, Apache Spark, Apache Flink, and DuckDB can create, read, write, and stream to or from UC managed Delta tables, with catalog commits providing serialized commits, transactional safety, and auditability. Delta Kernel, an open source Java and Rust library, abstracts low-level protocol details, helping connectors integrate external writes with catalog-managed commits while Predictive Optimization continues to run on accessed tables. Credential vending, now GA for tables, issues short-lived, scoped cloud credentials and supports M2M OAuth plus automatic refresh; volume credential vending is in Public Preview for unstructured data. The roadmap includes functionality for fine-grained row- and column-level ABAC on external reads, while external managed-table access remains in beta.
Alex Jiang, Tathagata DasThe Rosetta stone of CPS: Claroty’s AI-powered library
CPS asset identification is difficult because 88% of CPS assets do not transmit an exact product code and 76% use product codes that differ from the vendor's official records. Claroty's AI-Powered CPS Library addresses this entity-resolution problem with a hybrid architecture that combines classic matching, statistical inference, generative AI, NLP, reasoning agents, and human-in-the-loop review to reconcile protocol data, vendor records, manuals, and vulnerability information. Its Databricks-based lakehouse uses Delta Lake, Unity Catalog, Spark pipelines, Model Serving, MLflow evaluations, Lakebase, and Databricks Apps to maintain governed, auditable mappings across a catalog of more than 17 million assets. The post reports 25% improved vulnerability-attribution accuracy and says early tests gave 56% of analyzed devices new or updated recommendations for outdated firmware, while low-confidence mappings remain subject to expert review.
Ben Hazan, Anton Berlinsky, Ohad Avni, Itay Wagner, Guy Zalcman, Dor Bdolach, Ravid Ariely, Gal Sberro
