Loading…
Lakehouse
46 posts about Lakehouse. Every summary links to the original.
Debunking 8 data layout myths: why Liquid Clustering outperforms partitioning
Partitioning has long been the standard physical layout for Hadoop- and Hive-era processing, but the post argues that changing Lakehouse workloads make fixed choices brittle. Liquid Clustering treats keys as guidance for file organization, allowing layouts to evolve, supporting low- and high-cardinality filters, maintaining file sizes, and enabling row-level concurrency. Using transaction-log metadata and per-column statistics, the post says modern Delta and Iceberg engines prune files rather than directories, and reports 35% lower clustering time, 22% faster queries, roughly 90% faster metadata-only deletes, and 23-minute planning for a 10-PB table. Conversion examples include Bolt’s 138% higher write throughput and up to 63% faster reads, while a 1.1-PB workload saw 5.9x faster queries, 86% fewer bytes read, and a 27% smaller table after clustering by date, hour, source, and id.
Jeffrey Gong, Yu Xu, Rahul MahadevWinning under CMS TEAM: Building the learning health system to realize success in VBC today and tomorrow
CMS’s Transforming Episode Accountability Model (TEAM), beginning January 1, 2026, requires selected organizations to manage total cost and quality across five surgical episode categories and a 30-day post-discharge window. The post argues that retrospective dashboards and delayed claims data cannot support intervention or prepare hospitals for downside risk, especially amid complex integration across EHR, claims, post-acute, social-determinants, and provider data. It proposes a modern, cloud-native lakehouse foundation with unified data, operationalized AI/ML, embedded clinical workflows, and scalable architecture. Recommended capabilities include risk stratification, next-best-action decision support, post-acute optimization, provider benchmarking, and predictive performance alerts. In a composite 500-bed medical-center example, the approach is associated with 15% lower SNF costs, 8% better risk-capture completeness, and 12% fewer readmissions.
Kayla Grieme, Benjamin Goldsteen, Shankara EttigiFrom "What Happened?" to "What Will Happen?"
Databricks Genie makes descriptive analytics accessible in natural language, but predictive questions still require specialized data science workflows and carefully prepared datasets. This post presents a multi-agent supervisor deployed as a Databricks App, combining Genie, TabPFN, and Agent Bricks to turn business questions into predictions. The orchestrator asks Genie to use governed Lakehouse data, schemas, relationships, and semantics to generate labeled training data through SQL, then sends it to TabPFN, which predicts in a single forward pass without feature preprocessing, model selection, or hyperparameter tuning. The resulting conversational experience supports descriptive and predictive analytics with Unity Catalog lineage and access control, while an MLflow GenAI evaluation harness monitors reliability and regressions. Its central limitation is that predictions depend on Genie producing a meaningful dataset with a clear label, so missing signals, joins, outcomes, or agent omissions can make results unreliable.
Ryuta Yoshimatsu, Javier Poveda Panter, Dominik Safaric, Philipp Singer, Diana Kriuchkova, Sauraj Gambhir, Dael Williamson, Bryan SmithThe question your commercial data should already be able to answer
In ATTR-CM, a progressive and often fatal form of heart failure, 70 to 80 percent of patients do not know they have the disease, creating a need for better commercial prioritization than static call lists provide. Databricks and Veeva position embedded Genie agents and AI/BI dashboards in Veeva Vault CRM as a way to connect commercial data bidirectionally and answer role-specific questions inside existing workflows. Sales reps can see geographic HCP views with suspected patient counts, formulary scores, office availability, NRx-weighted priorities, and talking points, then rebuild a day after a cancellation. MSLs can generate cited pre-call briefs from approved sources such as Veeva Link, PubMed, ClinicalTrials.gov, and ASNC guidelines. Territory managers receive personalized views of call patterns, unworked signals, and dormant HCPs, while Unity Catalog provides shared access, lineage, and compliance governance.
Christina Busmalis, Tony FariasThe Convergence of Open Table Formats and Open Catalogs: Catalog Commits is Generally Available
Catalog Commits is generally available for Unity Catalog managed Delta tables, aligning Delta with Iceberg’s catalog-oriented model and making the catalog responsible for table discovery, access, and latest table state. The change addresses three coordination problems: metadata “split brain” when engines write directly to storage, fragmented multi-engine governance, and the historical inability to coordinate atomic writes across multiple tables. With Catalog Commits enabled, Unity Catalog brokers table access through standardized APIs, keeping catalog and table state synchronized and enabling consistent authorization, holistic auditability, automated optimizations, and multi-statement, multi-table ACID transactions on Databricks. The release supports Databricks products and engines including Delta Spark, Delta Flink, Starburst Trino, DuckDB, and StreamNative, while Delta Kernel provides a shared path for connector support.
Benjamin Mathew, Michelle Leon, Lukas Rupprecht, Ryan JohnsonAddressing HR's widening capacity gap with AI
HR teams face a widening capacity gap as strategic expectations, complex employee issues, workforce volatility, skills shortages, and demands for personalized support collide with largely unchanged headcount and tools. The article presents AI transformation as an incremental journey: first establish a secure Employee 360 from structured and unstructured enterprise data, then build reusable workforce insights, augment workflows with human oversight, and progress toward broader transformation. It emphasizes data governance, including access controls, auditing, quality, standardization, and reliable interpretations, while noting that trust has limited AI’s business impact so far. MathCo and Databricks support this roadmap through NucliOS, whose Data Studio, AI Studio, and Decision Studio environments connect governed data, explainable models, feedback loops, and decision applications; Databricks supplies the lakehouse foundation, lineage, quality checks, and privacy-compliant access.
Paurul Yadav, Soumya Dash, Bryan Smith10 trillion samples a day: Scaling beyond traditional monitoring infra at Databricks
Databricks’ monitoring infrastructure now tracks 5 billion active timeseries in real time and ingests more than 10 trillion samples daily, exposing scalability, reliability, cost, and operability limits in its older stack. The replacement, Pantheon, is a fork of CNCF Thanos deployed across more than 160 instances in about 70 regions and three cloud providers; tiered storage, differentiated memory retention, isolated replicated Receive groups, multitenancy, and a custom control plane support automated scaling and recovery. Pantheon’s largest instance holds about 300 million in-memory timeseries and handles nearly 1,000 PromQL queries per second, while migration reduced annual cloud costs by millions and monitoring downtime by roughly five times. For high-cardinality troubleshooting, Hydra preserves raw metrics in Delta tables, exposes them through Grafana and SQL, and unifies metric semantics across aggregated and raw paths, with freshness improvements planned.
David Yuan, Yi Jin, Karan Bavishi, HC Zhu, Joey BeydaDriving Budapest forward: How BKK uses Databricks to transform city mobility
BKK, Budapest’s unified transport authority, manages public transit, shared mobility, infrastructure, and traffic systems for a city of 1.7 million people. Its legacy on-premises warehouse and fragmented Excel- and PowerBI-based reporting struggled with growing, varied data, limiting efficient access and analysis. BKK phased a migration to Azure Databricks, starting with mobility datasets and carefully modeling and validating data from vehicle GPS, passenger sensors, boarding schedules, and other systems. The Databricks Lakehouse lets analysts process large geospatial datasets in shared SQL, Python, or R notebooks, supporting minute-level shared-mobility tracking, route-performance analysis, airport-bus forecasting, and dynamic scheduling. These capabilities provide faster insights and more responsive decisions, while governed access and cost visibility support broader collaboration and BKK’s vision for a digital twin of Budapest’s mobility system.
Max von Münster, Estilla Híves, Istvan Viz, Engin ErzenginDatabase modeling: a practical guide to techniques and best practices
Database modeling defines the structure, relationships and constraints that keep systems coherent across transactional, analytical and flexible workloads. The guide presents a three-phase design process: conceptual modeling captures entities and relationships, logical modeling adds attributes, keys, cardinality and normalization, and physical design translates that model into tables, indexes, constraints, partitioning and other platform-specific choices. It compares relational, document/NoSQL, dimensional, hierarchical and network models, linking each to data shape, workload and consistency requirements. It recommends explicit keys, consistent naming, appropriate normalization, query validation and version-controlled DDL, while warning against premature physical design, missing foreign keys, unclear NULL handling and poorly balanced schemas. It concludes that Databricks Lakebase combines a transactional database engine with the Databricks Lakehouse Architecture so one model can support operational and analytical workloads.
Databricks StaffTop Data Warehouse Tools For Modern Data Analytics
Data warehouse tools are presented as a strategic choice for analytics and ML teams facing fragmented estates of warehouses, lakes, and standalone ML systems. The guide proposes evaluating platforms across query performance, scalability, data integration, BI connectivity, total cost of ownership, and governance and security, with attention to MPP, columnar storage, and decoupled compute and storage. It contrasts traditional warehouses, data lakes, and lakehouses: warehouses favor structured SQL analytics, lakes offer native-format flexibility but weaker quality and performance guarantees, and lakehouses combine these capabilities using open formats such as Delta Lake and Apache Iceberg. The stated conclusion is that a lakehouse can provide a single governed foundation for SQL, BI, streaming, ML, and AI workloads, while teams should select tools according to workload, scale, budget, and future AI needs.
Databricks StaffAlert fatigue is a business risk
Enterprise security operations centers may receive tens of thousands of alerts daily, making prioritization necessary and leaving lower-priority signals uninvestigated. Alert fatigue is presented as a data architecture problem: fragmented endpoint, network, identity, and cloud telemetry, combined with proprietary SIEM collection-and-discard practices, limits correlation and overwhelms analysts. Lakewatch proposes an open lakehouse foundation that unifies security, IT, and business telemetry, applies automated OCSF normalization, and uses Agent Bricks for data wrangling and alert triage. Databricks Genie is positioned as a natural-language AI security agent whose autonomous agents can hunt, summarize, and neutralize threats, while Unity Catalog logs queries and actions for audit and forensic purposes. Lakewatch is currently available in Private Preview.
Taylor KainCompanies winning with AI built the data layer first
Trinity Industries’ experience is presented as a case for treating the data layer, rather than models or dashboards, as the foundation of enterprise AI. The railcar manufacturer migrated 95% of its enterprise data to a single Databricks lakehouse, adopted Medallion architecture, moved transformations upstream, and retired legacy dashboards containing nearly 600 measures. That consolidation supports streaming ETA predictions, procurement agents, and Genie conversational analytics: the ETA model is reported as 50% more accurate than industry ETAs, agents helped increase on-time material delivery by 15%, and Genie handles more than 1,000 questions monthly. The migration took close to a year, followed by six to eight months of additional work, but the account argues that trusted, standardized data enables experimentation, automation, and broader employee access to analysis.
Aly McGueRethinking SQL ETL for modern data platforms
SQL ETL is often distributed across a warehouse, transformation framework, orchestrator, monitoring, lineage, and data-quality systems, creating operational fragmentation as data teams scale. The post proposes a unified platform model that combines execution, orchestration, observability, governance, and optimization while supporting analytics engineers, warehouse engineers, and analysts through dbt, scripts and stored procedures, Materialized Views, declarative pipelines, and no-code tools. Databricks implements this model with shared execution, governance, and observability, using serverless infrastructure and AI-driven optimization alongside open table formats and ANSI SQL. The stated benefits include reduced operational overhead, support for batch and real-time workloads, and incremental modernization rather than disruptive rewrites. Reported examples include HP’s 32% cloud savings and 36% lower combined job runtime after moving to serverless compute, and Adobe reducing query execution from eight minutes to three seconds with Materialized Views.
Matt Jones, Shanelle RomanSecurely send first-party conversion signals with Snapchat Conversions API on Databricks Marketplace
Snapchat Conversions API is now available as a Databricks Marketplace listing that lets teams activate first-party conversion signals directly from governed Lakehouse data. The included pre-built notebook reads gold-layer tables, batches web, app, and offline events, and supports deduplication with Snap Pixel, allowing server-side signals without double-counting. Because it runs in the Databricks environment, Unity Catalog preserves governance, lineage, security, and access controls while teams control what data is shared and when. The post says fuller server-side coverage can raise Event Match Quality, while citing that advertisers who improved their EQS from “Poor” to “Good” saw 26% higher ROAS and 49% lower CPI. It also describes fresher custom audiences and Lookalikes plus more complete deduplicated measurement.
Nikhil Gaekwad, Jen McNamee, Katy YuanPowering KPMG UK Audit's AI future with Databricks
KPMG UK is evolving its Audit data platform with Databricks to support AI-enabled analytics while preserving governance, rigour, and professional standards. The program converges structured data, advanced analytics, and AI on a unified cloud-native Lakehouse, with Databricks SQL replacing core SQL Server workloads and Delta underpinning the platform. Lakebridge assessed migration complexity, while Databricks-hosted large language models including Claude Sonnet and Genie Code helped convert T-SQL, refactor stored procedures, modularize queries, and suggest Delta-based optimizations under engineer review. This reduced refactoring time by around 60% and enabled modernization of more than 400 scripts and stored procedures in roughly three months. Databricks SQL Serverless provides elastic compute for spiky workloads, while Genie offers traceable, version-controlled SQL and Delta Sharing supports governed data exchange.
Mark Wallington, Greta NasaiOperational databases: How they work and when to use them
Operational databases, or online transaction processing (OLTP) databases, support day-to-day operations by processing real-time CRUD transactions with low latency, high concurrency, and ACID guarantees. The guide contrasts them with data warehouses and OLAP systems, which favor historical analysis, complex aggregations, and read-heavy workloads, while describing ETL, CDC, and operational data stores in the OLTP-to-OLAP pipeline. It argues that traditional OLTP systems can struggle with real-time analytics and AI because of siloed data, structured-data constraints, rigid schemas, scaling limits, and missing governance features. Databricks Lakebase is presented as a hybrid solution using separate storage and compute, serverless Postgres, elastic scaling, branching, shared lakehouse storage, and open formats. The proposed workflow connects existing OLTP systems through CDC or streaming into Delta Lake for fresh SQL, BI, ML, and AI access.
Databricks StaffBeyond the spreadsheet: How Databricks is delivering the modern CFO in financial services
Financial-services CFOs are being asked to act as Strategists and Catalysts, but fragmented legacy systems, T+1 batch processing, opaque reporting logic, and mismatched business semantics keep them focused on stewardship and retrospective operations. The proposed answer is Databricks as a unified, governed platform combining real-time streaming, centralized lineage, data and AI, with Unity Catalog, Lakeflow, Genie, and Agent Bricks addressing trust, latency, access, and model reproducibility. Unity Catalog can connect semantic definitions and end-to-end lineage from transactions through regulatory reports and models, while Lakeflow supports continuous ledger and liquidity processing. The post cites a global bank reducing liquidity-reporting processing from 10 hours to 8 minutes and Nationwide Insurance reporting a 5-point combined-ratio improvement and 3-point expense-ratio improvement. It presents a Lakehouse-based CFO stack as a shift from reporting historical results toward real-time capital management and previews AI-driven deposit and PPNR modeling.
Jennifer Miller, Marcela Granados, Andrea DeSosa, Alex Oberlander, Kim Hatton, Pavithra Rao, Naeem Rehman, Pravin Varma, Olga Deriy, Prasanna SelvarajIoT in Manufacturing: Strategy, Components, Use Cases, and Challenges
IoT in manufacturing is presented as an operational architecture for collecting machine data, improving production and supply-chain visibility, and preventing equipment failures under strict latency, reliability, and safety requirements. The guide covers device selection, sensor placement, tiered sampling, edge-to-cloud pipelines, platform criteria, security, compliance, workforce training, and a phased deployment roadmap. Its technical split sends high-frequency vibration and acoustic signals to edge gateways for local processing while cloud systems support stateful analytics, predictive maintenance scoring, and model training; protocols such as MQTT, OPC-UA, AMQP, and Modbus are evaluation criteria. Reported examples include multi-factory OEE monitoring across more than 200 production lines, reducing reporting lag from 24 hours to under five minutes, digital twins for maintenance simulation, and IoT logistics routing. The action plan recommends starting with predictive maintenance and OEE monitoring on one line before scaling.
Databricks StaffMultimodal Data Integration: Production Architectures for Healthcare AI
Production healthcare AI often depends on combining genomics, imaging, clinical notes, and wearables, but separate modality stacks can leave initiatives stalled before deployment. A lakehouse pattern addresses this by landing each modality in governed Delta tables under Unity Catalog controls while supporting cross-modal features, lineage, time travel, and reproducible training sets. Genomics processing can use Glow on Spark, imaging embeddings can feed governed tables and vector search, notes can become temporally aware entities, and wearables can flow through Lakeflow Spark Declarative Pipelines for streaming feature windows. The post compares early, intermediate, late, and attention-based fusion, matching them to dimensionality, temporal, and availability constraints, with late fusion presented as a practical baseline when modalities are missing. It argues that modality masking, sparse attention, and transfer learning are recommended responses because incomplete data is the default and connects the architecture to governed tumor board review, faster cohort assembly, and more clinical workflows.
Maks KhomutskyiBuilding an Enterprise Data Management Strategy
Enterprise data management (EDM) is presented as an organizational capability for governing, integrating, securing, and operationalizing data across its lifecycle as volumes and regulatory scrutiny increase. The strategy addresses silos, inconsistent schemas, undocumented transformations, compliance exposure, and the difficulty of supporting analytics and AI across fragmented systems. It combines governance councils and ownership, automated profiling and quality thresholds, classification with RBAC and encryption, reusable ETL/ELT pipelines, lineage tracking, and architecture choices such as lakehouses and medallion patterns. Unity Catalog is described as a unified governance layer for reads and writes across open formats including Delta Lake and Apache Iceberg, while federation can provide access to external catalogs without copying data. Execution begins with cross-functional sponsorship, a high-impact pilot, measurable KPIs, and quarterly reviews; the source frames EDM as continuous rather than one-time work.
Databricks Staff
