Loading…
Topic
Delta Lake
35 posts about Delta Lake. Every summary links to the original.
- 35 posts
- 1 company
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
The data lakehouse is a better data warehouse: 2026 benchmarks and proof
Data warehouses have been the standard for structured SQL analytics for two decades but cannot serve modern AI, ML, streaming, and unstructured data workloads. The data lakehouse now matches warehouse SQL performance while serving all these workloads from a single governed foundation. Databricks Lakehouse improved production SQL workload performance by 77% from a 2022 baseline, with BI queries improving 14% over a recent five-month period. Customer migrations show 90% faster queries and 30–40% cost reduction at Lumen Technologies, and 80–90% faster analytics at Trek Bicycle. The lakehouse eliminates redundant data copies, supports open formats like Delta Lake and Iceberg, and extends governance to AI agents through Unity Catalog and Unity Gateway.
Scaling and Operating a Large dbt Project on Databricks: IFCO's Data Team on Performance, Visibility, and Debugging
IFCO’s data platform team, with Databricks Forward Deployed Engineering, optimized a large dbt project on Databricks to handle billions of daily tracking events for reusable packaging assets. The pipeline faces scale, unpredictable late-arriving data, and the need to reconcile history when late events change previously reported asset timelines. The team tuned incremental model settings to match Delta Lake write behaviors, including liquid clustering, deliberate merge vs. delete+insert strategies, and bounding writes to recent ingestion windows. Diagnosing performance from actual query plans rather than compiled SQL revealed that the consolidation model was scanning far more rows than necessary. These changes cut the core semantic layer job’s daily runtime by over 60 percent, retired a costly nightly full refresh, and reduced daily compute cost by 58 percent.
Your data, your storage, your rules: a 2026 guide to storing Unity Catalog managed tables
Unity Catalog managed tables let you control where your data physically resides while keeping it in cloud storage you own, such as S3, ADLS, or GCS, unlike platforms that store tables in provider-controlled storage. You set managed storage locations at the metastore, catalog, or schema level, with the most specific level winning, and can change where new tables land using ALTER CATALOG or ALTER SCHEMA commands. Managed tables use open formats including Iceberg and Delta, and external engines such as Apache Spark, Flink, and Trino can read and write through Iceberg REST Catalog and Unity Catalog open APIs. When converting an external table to managed, Databricks copies the data and transaction log into the managed storage location you have set. This approach enables a single source of truth across analytics and AI workloads while satisfying compliance, cost allocation, and data residency requirements.
RADAR: Catch gray failures with anomaly detection
Gray failures are partial customer-facing problems that can leave health checks green while a subset of users quietly fails. RADAR (Reliability Anomaly Detection, Alerting, and Root-cause analysis) addresses this gap by collecting error counts and distinct affected users, segmented by error code and region, then applying the unsupervised streaming SPOT model. SPOT learns normal behavior from the previous 14 days using a single risk parameter, while downstream stages enrich, filter, deduplicate, and route alerts before providing anomaly details and AI/BI Genie-backed dashboards for investigation. Running RADAR internally reduced incident-discovery time by 95% with over 90% precision and no human needed to spot the pattern. The post also describes a Databricks scaffold that maps metrics, detection, alerting, and root-cause analysis to platform components and can be assembled by an AI agent from a metric, scaffold, and prompt.
Google Cloud Use Cases: How Enterprises Run Data & AI on GCP with Databricks
The guide presents Google Cloud as a unified platform for enterprise data engineering, analytics, machine learning, and generative AI, with managed services intended to reduce infrastructure and integration work. It focuses on practical architectures, including medallion ETL pipelines that store Bronze, Silver, and Gold Delta Lake tables in Google Cloud Storage and expose Gold data to BigQuery through an optimized connector. For streaming workloads, Cloud Pub/Sub supplies ingestion while Apache Spark Structured Streaming processes continuous micro-batches; together with Delta Lake ACID transactions, this supports exactly-once results despite duplicates or out-of-order messages. The text also describes Cloud Run and Cloud Functions for containerized or event-driven workloads, and identifies retrieval-augmented generation applications on Vertex AI as a common starting point for generative AI. It concludes that integrated service combinations, rather than isolated tools, are the central strategic advantage for organizations building evolving data and AI platforms on Google Cloud.
The next era of the open lakehouse: Apache Iceberg™ v3 in Public Preview on Databricks
Databricks has placed support for Apache Iceberg v3 into Public Preview, bringing new table-format capabilities to its open lakehouse. Unity Catalog Managed Iceberg v3 tables support Row Lineage, Deletion Vectors, and the VARIANT type, while interoperability extends these features to foreign Iceberg tables. Row IDs and modification sequence numbers identify changed rows, and lightweight delete files avoid immediate Parquet rewrites, with data manipulation reported as up to 10x faster than copy-on-write. VARIANT stores evolving semi-structured payloads beside relational columns for direct SQL querying, while Unity Catalog provides cross-catalog governance and Delta-Iceberg interoperability without data duplication. Iceberg v3 is available on Databricks Runtime 18.0+ with Unity Catalog enabled; Databricks is also advancing Iceberg v4 metadata proposals.
Southern Company’s SCOUT: Completing the Storm Intelligence Story
Southern Company’s SCOUT is a real-time storm operations application designed to make outage intelligence accessible beyond control-center desks during active events. Built on the Databricks lakehouse, it combines Delta Lake storage, Databricks Lakehouse warehouses, Unity Catalog governance, and collaborative notebooks, querying a dedicated warehouse every minute for current operational data. The mobile-friendly application unifies outage, customer, GIS, weather, terrain, crew, and reliability information, helping teams assess restoration times, assign appropriately equipped crews, coordinate mutual assistance, and communicate with customers. SCOUT is used across Southern Company’s operating businesses and had 1,139 adopted employees, including more than 250 users on a peak storm day; the source also describes exploration of AI crew recommendations and autonomous-drone integration, not a completed deployment.
How Unity Catalog managed tables bring interoperability, performance, and unified governance to the Lakehouse
Unity Catalog external access to Unity Catalog managed Delta tables is now in Public Preview, allowing external engines to create, read, and write while governance remains centralized. The announcement addresses the previous multi-engine trade-off: external tables enabled access but lacked managed-table performance optimizations and governance guarantees. Catalog commits coordinate writes through Unity Catalog, making it the source of truth for table state and enabling safe external writes, multi-statement transactions, and auditing of external operations. Predictive Optimization cleans storage, collects query statistics, and selects Liquid clustering columns as query patterns change; the post says these capabilities can deliver up to 50% storage cost savings and 20x faster queries. Support includes Spark, Flink, Starburst, DuckDB, and StreamNative, with open APIs and Delta Kernel extending integrations across Databricks UC and UC OSS.
Navigating a Synapse Migration to Databricks
The post presents migration from Azure Synapse to Databricks as a way to reduce platform complexity while supporting data engineering, analytics, machine learning, real-time pipelines, and AI applications. It distinguishes migration paths for Dedicated SQL Pools, serverless SQL, Spark Pools, orchestration, governance, and downstream BI connectivity, noting that accumulated T-SQL logic and dependencies require the most redesign, testing, and validation. The proposed programme uses discovery and assessment with Lakebridge Profiler and Analyzer, then selects an architecture, sequences workloads, pilots a real use case, and scales migration in waves. It recommends automating code conversion, reconciling row counts, aggregations, hashes, and tolerances, avoiding direct Synapse distribution-to-Delta partition mappings, and planning compaction, enablement, and rollback. Reported examples include Casey’s halving operational data delivery times from eight hours to four and Italgas cutting workload costs by 73%.
How to Evaluate an Enterprise Analytics Platform
Enterprise analytics platform evaluations often overemphasize dashboard interfaces, although the larger decision concerns whether analytics, AI and agents share data, semantics and governance. The post distinguishes point solutions from a unified platform and proposes seven evaluation criteria: workload fit, architecture and openness, governance and compliance, performance and scalability, adoption and usability, AI and ML readiness, and total cost of ownership. It recommends mapping current and three-year workloads, testing production-scale data with realistic concurrency, measuring p95 latency, and examining governance, usability, contracts and operational complexity in a proof of concept. Lakehouse architecture, open formats such as Delta Lake and Apache Iceberg, and shared controls are presented as ways to reduce context gaps; Databricks is offered as a practical example using Unity Catalog, Genie and Agent Bricks. The conclusion favors a weighted, three-year assessment over a feature comparison.
How Imperial College London is accelerating dementia research with a modern data platform
Imperial College London’s Care Research and Technology Centre (CR&T) needed to scale Minder, a platform that combines in-home sensors, sleep monitors, and electronic health records to support dementia care and research. Growing data volumes caused competing workloads, tightly coupled storage and compute, and limited access for researchers and clinicians. The re-architected platform validates IoT data through Kubernetes, stores it in Delta Lake on Azure Data Lake Storage using bronze, silver, and anonymized gold layers, and adds Databricks analytics with Unity Catalog governance while retaining FHIR-based EHR interoperability and Kubeflow for model deployment. Research-to-production workflows, reusable pipelines, dashboards, and timestamped data improve collaboration, accessibility, and reproducibility. During migration, uptime remained 100%; new sources integrated in as little as one month, model development fell to about one month, and insights reached 581 people living with dementia in five months.
Data Lake vs. Cloud Data Warehouse: A Practical Guide for Data Scientists
The guide contrasts data lakes and cloud data warehouses for storing and querying data at scale. Data lakes retain raw structured, semi-structured, and unstructured data in low-cost object storage with schema-on-read, while warehouses enforce schema-on-write for structured analytical workloads. Lakes fit petabyte-scale machine learning, data science, and undefined future use cases; warehouses fit fast, concurrent SQL for dashboards, reporting, and operational analytics. It describes Bronze, Silver, and Gold zones, with Parquet and ORC supporting columnar scans and open-format portability. For teams combining ML and BI, lakehouses use Delta Lake, Apache Iceberg, or Apache Hudi to add ACID transactions, schema enforcement, and quality monitoring to lake storage without duplication; catalogs, staged checks, and access controls help prevent data swamps.
Introducing OpenSharing: the Next Evolution of Delta Sharing for the Agentic Era
OpenSharing is presented as the next evolution of Delta Sharing, extending an open zero-copy data-sharing protocol from tables and files to models, agents, semantic context, unstructured data, and reusable AI logic. The protocol is now an independent open-source project hosted by the Linux Foundation, while Databricks OpenSharing adds Unity Catalog governance and audit logging, Marketplace discoverability, and enterprise features. Genie Agent Sharing supports governed AI experiences across organizational boundaries, with controls for proprietary instructions, data access, daily prompt quotas, and row exports. SecureConnect removes per-recipient firewall changes through a Databricks-managed proxy, while Global Distribution uses local replicas to reduce egress fees and latency. The launch also supports Apache Iceberg REST Catalog API, external catalogs, and on-premises storage partners; providers define shares in Unity Catalog, recipients query live data through existing tools, and governance enforces access controls.
What’s new with Unity Catalog at Data + AI Summit 2026
At Data + AI Summit 2026, Unity Catalog announcements position the catalog as a runtime governance layer for enterprise data and AI, organized around control, context, and choice. Control additions include Unity AI Gateway for governing models, agents, MCP services, skills, and tools; contextual service policies can allow, deny, or require approval for runtime actions, while budgets, hard caps, tracing, and guardrails address spend, investigation, and safety. Context additions include Glossary and Domains for business meaning and scoped asset organization, plus Metrics that standardize KPIs for SQL, BI tools, APIs, and agents; Genie Ontology is described as a continuously learned enterprise context layer. Choice additions span cross-cloud and cross-region addressability, managed disaster recovery, Delta and Iceberg interoperability, multimodal and geospatial types, and open sharing of data, AI assets, and applications across organizations.
Accelerate search queries with full-text search indexes on Databricks
Databricks introduces full-text search indexes in Beta on Databricks Runtime 18.2 to accelerate substring and keyword queries on large open-format tables without changing their layouts. The indexes tokenize text columns into a compact lookup structure mapping tokens to matching rows; at query time, the engine uses it to identify candidate files and skip most of the table. They are maintained asynchronously, require no query hints, preserve complete results when stale by scanning indexed and non-indexed data as needed, and support Unity Catalog managed Delta and Iceberg tables on serverless and classic compute. A Trust and Safety team reported a substring search running more than 100x faster on a petabyte-scale table, while Liquid clustering remains complementary because it optimizes column-value filters rather than text within fields.
What is an open lakehouse? Open data standards, explained.
The piece defines an open lakehouse as a lakehouse whose storage, table format, processing engine, catalog, and ML and AI tooling use open standards and remain interchangeable. It contrasts this architecture with warehouses, lakes, and proprietary lakehouses, emphasizing low-cost object storage, ACID transactions, governance, schema guarantees, and the ability to change engines without rewriting data. Its reference stack combines open table formats such as Delta Lake and Apache Iceberg with Apache Parquet, Apache Spark, Unity Catalog, and MLflow, while allowing engines including DuckDB, Trino, and PyIceberg to work on the same data. The article also distinguishes open standards from open-source code, explains that a table format is only one layer of the stack, and states that the components can be self-hosted or consumed through a managed service.
Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest
The post benchmarks Zerobus Ingest, a managed, serverless, push-based service that writes producer data directly to Delta tables governed by Unity Catalog, against a petabyte-scale telemetry workload. Using NASA’s NEOWISE dataset and Locust, the test modeled fan-in from 2,048 concurrent streams, using Protocol Buffer 2 data over approximately 24 hours. The design replaces static partition-based ordering with stream-connection ordering, allowing heuristic routing across pods, dynamic partitioning, and autoscaling while existing streams drain. It also uses zeroparser, a zero-copy protobuf decoder whose design relies on Rust’s lifetime system and supports dynamic descriptors at about 1 GB/s per CPU core. The test sustained 12 GB/s to one table, ingested 1.04 trillion rows, and reached 1 petabyte within 24 hours; Zerobus Ingest is generally available, with additional APIs on its roadmap.
Geospatial Unbounded: Spatial SQL GA with AI/BI Maps, Delta Sharing, and Iceberg v3
Databricks announces Spatial SQL as Generally Available, positioning its platform for geospatial analysis without separate spatial databases, warehouses, and mapping tools. It supports native GEOMETRY columns in Delta or Iceberg, more than 90 OGC-compliant ST_* functions, spatial joins, and boolean set operations. AI/BI dashboards can render Geometry and Geography columns as maps, while Genie can generate spatial queries and dashboards and respect Unity Catalog row filters. Geo columns are supported by Delta Sharing, and Databricks can read and write managed Iceberg tables or read externally written Iceberg tables with geospatial types in Iceberg v3. Benchmarks show eight of twelve SpatialBench queries improved since Public Preview, with gains from 20% to 15X, while areal boolean operations are twice as fast on average versus prior versions.
Announcing the winners of the 2026 Databricks Customer Awards
The 2026 Databricks Customer Awards recognize organizations and leaders using the Databricks Data + AI Platform across eight categories and four regions. The announcement names winners including Applied Materials, Virgin Atlantic, Fonterra Co-operative Group, Telefónica | Vivo, Virtue Foundation, Octopus Energy, Axpo, Atlassian, Wassym Bensaid at Rivian and Volkswagen Group Technologies, and Kenan Colson at Lippert. Examples include Applied Materials’ move from a Hadoop-based data lake to a governed lakehouse, with 1,500-plus analysts, more than 100 production machine-learning models and faster pipeline development, while Fonterra centralized data to improve supply-chain and compliance work. Rivian unified vehicle, factory and enterprise data, consolidated legacy systems into Delta and Unity Catalog, and designed for 500–600 petabytes; Lippert deployed AI tools across customer care, finance and HR. The announcement presents data and AI adoption as a cross-functional operating model.
Transforming solar and wind maintenance reports with Genie and AI agents
Plenitude and Databricks built an agent-based system that turns solar and wind plant maintenance PDFs into structured data for cross-plant analysis. Event-driven ingestion uses Databricks Jobs and the ai_parse_document AI Function to extract text, tables, figures, and metadata, then stores page- and object-level JSON records in Delta Lake with coordinates, version history, and links to source reports. A Genie space uses Unity Catalog metadata, knowledge-store instructions, and SQL generation to answer natural-language questions, produce visualizations, and export results, while Agent Bricks can orchestrate multi-step workflows and downstream actions. The design also applies automatic liquid clustering to dynamic queries and row-level security to restrict results by country. The resulting data layer supports historical trends, plant comparisons, recurring-fault analysis, and a foundation for predictive maintenance, although the source frames predictive use as a future improvement.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.