Loading…
Topic
Lakehouse
65 posts about Lakehouse. Every summary links to the original.
- 65 posts
- 3 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Transactions are Generally Available, advancing warehousing on the Lakehouse
Databricks made multi-statement transactions generally available on Delta Lake and Apache Iceberg tables, providing atomic guarantees for complex, multi-step workflows directly on the lakehouse. Previously, organizations had to rely on separate warehouse systems for these transactional guarantees, adding expensive data movement, duplicate copies, fragmented governance, and custom consistency workarounds. Row-level concurrency enables concurrent transactions to write to the same table as long as they do not update the same row, delivering significantly stronger performance than traditional cloud and legacy warehouses. The transactions are composable with stored procedures, temporary tables, and streaming tables, and they respect Unity Catalog attribute-based access controls including row-level security and column masking on both reads and writes. Fivetran used transactions to remove consistency-repair code for its slowly changing dimension updates using SCD Type 2, and Accolade processes roughly 1,700 transactions daily for claims and liabilities on the governed lakehouse.
Geospatial queries are up to 38X faster with Lakehouse//RT
Databricks announced that Spatial SQL and spatial joins now run on Lakehouse//RT, delivering up to 38X faster performance under high concurrency compared to traditional cloud data warehouses. The platform supports GEOMETRY and GEOGRAPHY data types, over 100 spatial functions, and high-performance spatial joins, all within the open lakehouse using Apache Spark and Parquet files. Organizations in insurance, agriculture, energy, telecom, retail, and logistics can consolidate geospatial workloads onto a single governed platform, eliminating separate stacks like PostGIS. Benchmarks using SpatialBench show Lakehouse//RT is 3X faster on average and up to 20X faster for single-query concurrency, with no code changes required. The service is available in Beta, allowing users to run existing spatial queries directly on Delta and Iceberg tables without copying data.
Reducing the noise: How Atlassian built an OCSF security lakehouse on Databricks
Atlassian faced a security data pipeline ingesting 44 billion events daily from 20 years of growth and acquisitions, each with proprietary logging formats. To unify this fragmented telemetry, the team standardized on the Open Cybersecurity Schema Framework (OCSF) and built a security lakehouse on Databricks. They embedded the entire OCSF schema into Go structs (TypeGo), guaranteeing compile-time correctness and reducing AI hallucinations during automated transformation. Table optimization with partitioning and Z-ORDER cut query times by 95%, and Graviton clusters with Photon reduced cost and runtime by two-thirds. The result was a self-service platform where SOC analysts can onboard new data sources in days, and detection teams moved from one-to-one alerts to signal-based clustering using standardized OCSF values.
Databricks is a Visionary the Gartner® Magic Quadrant™ for Analytics and Business Intelligence Platforms
Databricks debuted as a Visionary in the 2026 Gartner Magic Quadrant for Analytics and Business Intelligence Platforms, achieving the highest debut performance in the report’s 20+ year history. The company also received one of the highest Completeness of Vision scores among all vendors evaluated. Databricks attributes this recognition to its ability to deliver on three major trends reshaping businesses: data and AI platform convergence, agentic analytics, and semantics at the core. The Databricks lakehouse integrates BI directly, eliminating data silos and the need for extracts, so agents can reason over complete, governed context. With Genie and AI/BI, the platform automates complex workflows and provides a unified semantic layer through Unity Catalog Business Semantics and Genie Ontology.
The lakehouse is a better data warehouse: 2026 benchmarks and proof
The data warehouse served structured SQL workloads for two decades, but the lakehouse now supports BI, data engineering, ML, streaming, and AI agents from a single governed platform. Databricks reports a 77% improvement in its production SQL workload mix since 2022, with customer migrations showing 90% faster queries (Lumen) and 80-90% faster analytics (Trek Bicycle). Lakehouse//RT delivers sub-second analytical queries at high concurrency directly on governed lakehouse tables, with Databricks reporting sub-100-ms latency at 12,000 queries per second, collapsing the traditional architecture of separate serving databases. The lakehouse eliminates data copying between systems, handles unstructured data, and runs on open formats like Delta Lake and Apache Iceberg, governed by Unity Catalog. Customer cost reductions range from 25% to 75%, with AXA Japan reducing warehouse costs by 60% and ETL costs by 70%, driven by eliminating redundant systems and data copies.
Unlocking Data Portability: Preventing Catalog Lock-in with REGISTER and UNREGISTER APIs
The post explains how the REGISTER and UNREGISTER APIs in the Apache Iceberg REST catalog specification enable table portability between catalogs without copying or moving data files. REGISTER attaches an existing table to a new catalog by providing the metadata location, but using it alone creates a split-brain scenario where two catalogs both think they own the table, leading to inconsistent writes and data loss. The UNREGISTER endpoint, contributed to the specification, cleanly removes the table from the old catalog and returns the latest metadata pointer needed by the next catalog. The handoff process combines unregister, receiving the location, and register to ensure exactly one managing catalog at all times. Unity Catalog supports these endpoints in private preview, allowing organizations to move tables between catalogs as their architecture evolves.
Connecting customer context to measurable ROI with agentic marketing
Customer journeys have become non-linear while identity signals decay, and marketing value is shifting from reach frequency to a prediction economy that rewards anticipating needs. AI agents can recommend or take the next-best action for each customer within human-defined guardrails, with each action measured incrementally against what would have happened without it. Identity resolution must be continuous, connecting known and pseudonymous signals across channels, to provide reliable context for agent decisions. The context required includes customer behavior, business objectives, decision history, and control guardrails, with a single transaction carrying up to 50 metadata parameters. A practical starting point is focusing on one meaningful decision, such as reactivating lapsed customers, and expanding as trust and results build.
IP Functions are Generally Available, bringing high-performance network analytics to the Lakehouse
The announcement introduces the general availability of IP Functions, a feature that brings high-performance network analytics to the Lakehouse by enabling SQL-based processing of IP addresses. The post explains that IP data is warehousing data, as firewalls, load balancers, VPNs, CDN edges, DNS resolvers, Kubernetes clusters, and application servers emit streams of records keyed on IP addresses, generating tens of billions of events daily for large enterprises. These streams support valuable analytics such as threat detection, fraud investigation, and network governance. The post notes that network analytics used to be painful because IP addresses were deceptively hard to handle in SQL, with no regex support. The feature aims to simplify this by providing native IP functions for the Lakehouse.
How Databricks Helps Baseball Teams Gain an Edge with Data & AI
Baseball teams can use Databricks to turn Statcast and team-derived pitch-level data into game-day plans for hitters, bullpen changes, pinch-hit choices, advance scouting, and roster decisions. Genie, on Unity Catalog, queries governed Delta tables and reusable Unity Catalog SQL functions for tendencies by count, handedness, pitch type, location, and base-runner state. A Multi-Agent Supervisor built with Agent Bricks and deployed on Model Serving simulates matchup pockets, compares reliever arsenals with hitter profiles, and produces explanations that staff distill into printed decision cards. For advance work, AI Search indexes pitcher and hitter embeddings to find similar arms, while Lakebase Postgres keeps reports, tags, and decisions close to the source data for shared operational memory.
The next era of the open lakehouse: Apache Iceberg™ v3 in Public Preview on Databricks
Databricks has placed support for Apache Iceberg v3 into Public Preview, bringing new table-format capabilities to its open lakehouse. Unity Catalog Managed Iceberg v3 tables support Row Lineage, Deletion Vectors, and the VARIANT type, while interoperability extends these features to foreign Iceberg tables. Row IDs and modification sequence numbers identify changed rows, and lightweight delete files avoid immediate Parquet rewrites, with data manipulation reported as up to 10x faster than copy-on-write. VARIANT stores evolving semi-structured payloads beside relational columns for direct SQL querying, while Unity Catalog provides cross-catalog governance and Delta-Iceberg interoperability without data duplication. Iceberg v3 is available on Databricks Runtime 18.0+ with Unity Catalog enabled; Databricks is also advancing Iceberg v4 metadata proposals.
Business Analytics Tools: A Complete Guide for Data
Business analytics tools have evolved beyond static reporting as organizations seek explanations, forecasts, and immediate actions. This guide categorizes visualization and dashboard platforms, self-service analytics, statistical tools, spreadsheets, and SQL-based query tools, then examines how AI adds natural-language querying, anomaly detection, summaries, and forecasting. It argues that inconsistent, duplicated, or differently governed data can undermine even sophisticated dashboards. Lakehouse architecture combines data-lake scalability with warehouse governance, performance, and SQL compatibility, enabling tools such as Tableau, Power BI, and Looker to use broader, fresher datasets alongside analytics and AI workloads. Reported examples include 94% faster BI queries at Anker Innovations, 98% faster query performance at AnyClip, and consolidation across 120+ global analysts at JLL.
Databricks Announces Lakewatch: New, Agentic SIEM
Databricks announces Lakewatch, an agentic SIEM intended to help organizations defend against increasingly sophisticated, AI-driven attackers. Launching in Private Preview, it unifies security, IT, and business data in a governed lakehouse built on Unity Catalog, while supporting open OCSF formats and storage in customer-controlled cloud object stores. Lakeflow Connect ingests and normalizes major security sources, and Detection-as-Code lets teams define YAML rules with SQL queries or Python notebooks, backtest them against historical data, and deploy through CI/CD. Genie Code and Genie Spaces provide AI-assisted ingestion, detection authoring, rule refinement, investigation, and natural-language threat hunting, while custom models can support anomaly detection, behavioral analytics, and entity risk scoring. The announcement also introduces an Open Security Lakehouse Ecosystem and acquisitions of Antimatter and SiftD.ai; Lakewatch is available in Private Preview, with Adobe and Dropbox among the named customers.
How FSIs eliminate silos between clients, operations, and finance
Financial institutions often split portfolio, operations, actuarial, and finance work across separate systems, even when teams depend on the same portfolios, positions, liabilities, and transactions. Databricks presents a unified, governed platform combining Databricks SQL, Unity Catalog Metric Views, Lakebase, Genie, Databricks Apps, and Lakeflow Designer to connect analysis with operational action. A worked workflow follows an actuary's duration analysis, a portfolio manager's scenario modeling and mandate changes, operations' IBOR/ABOR reconciliation, and finance's approval of adjustments and ledger entries. Governance features including row-level security, column masking, RBAC, ABAC, and shared business semantics keep access and terminology consistent, while Lakebase records corrections in auditable, timestamped tables. The result described is a closed-loop process in which strategy, risk, and accounting use the same governed data rather than reconciling separate systems.
Powering clinical research innovation: How TriNetX uses Databricks to accelerate drug development
TriNetX operates a federated real-world health data network spanning more than 230 healthcare organizations and needed infrastructure to make growing volumes of RWD easier to use for clinical research. Databricks now provides the centralized lakehouse architecture, consolidating electronic health record data while supporting custom datasets, consulting, ML models, proprietary algorithms, APIs and self-service access. TriNetX is extending this foundation with a beta Query Assistant for natural-language analysis and a Support Assistant prototype built with Databricks’ Agent Bricks, requiring no programming expertise for the former. In 2025, pharmaceutical clients reduced protocol amendments by up to 50%, while an AI-enhanced site-identification effort achieved a 63% site acceptance rate and nine-day average response time. Other models suggest enrollment conversion could rise from 33% to 85%, with expanded APIs and Genie exploration planned for 2026.
How energy teams turn theft detection into governed action with Genie and AI business processes
Energy theft detection often stops at an ML risk score, leaving revenue-protection teams to manually interpret cases, prepare investigations, dispatch crews, recover losses, and report results. The described Databricks workflow turns that score into a governed operational loop: Lakeflow prepares features, models run through Model Serving and Unity Gateway, and a Databricks App uses Lakebase to store live case state. AI-generated summaries explain why accounts were flagged, while dispatch-ready reports provide next steps, evidence checklists, and safety notes. Genie One answers leadership questions using metric definitions in Unity Catalog, and an Agent Bricks Multi-Agent Supervisor can assemble a traceable month-end board pack. Unity Catalog lineage, PII labels, access controls, model guardrails, and Unity Gateway routing support regulated use under frameworks including Ofgem’s and ELEXON’s requirements, while the same pattern is presented as applicable to other ML-driven interventions.
How Databricks’ marketers use data 3x more with Genie, an AI analytics assistant
Databricks marketers struggled to get trusted answers from fragmented marketing data quickly, prompting the creation of Marge, a marketing implementation of Genie Agents. The team first unified campaign, sales, web, event and advertising information in a governed Marketing Lakehouse, then grounded the assistant in business definitions, verified logic and example queries. Marge translates natural-language questions into analytical queries, uses Unity Catalog for lineage and role-based access, and can ask clarifying questions or perform multi-step analysis. The reported results include threefold greater data use in decisions, more than 85% adoption, 50% quarter-over-quarter growth, over 5,000 answered questions and a 25% reduction in flagged incorrect responses. Databricks attributes the rollout to continuous feedback, benchmark evaluation, focused use cases and embedding Genie in existing workflows rather than treating conversational analytics as a one-time deployment.
The 40-year-old database rule agents just broke: How LTAP unifies OLTP and OLAP workloads
LTAP (Lake Transactional/Analytical Processing) addresses the long-standing separation between operational row-oriented databases and analytical column-oriented systems, which traditionally required pipelines and copied data. It uses Lakebase’s stateless, ephemeral compute decoupled from durable lake storage to expose live operational data for analytical queries without moving it or loading the transaction-serving system. The storage layer preserves Postgres data types and physical representation while writing data into Parquet, with a hot row-format tier for operational access and a cooler columnar tier for analytical reads; Apache Spark and SQL can read that representation directly. By separating serverless operational and analytical compute, LTAP allows each workload to use specialized engines and independent capacity, while a unified catalog applies governance across both. The stated motivation is agentic workloads such as fraud detection, where stale copies miss fast events and uncontrolled analytical queries can degrade transactions.
Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption
Grab is transitioning its petabyte-scale data lake from Hive Parquet tables managed through the Hive Metastore to a table-centric architecture centered on Apache Iceberg, while retaining storage-compute separation on S3. The move addresses catalog listing latency, severe small-file fragmentation, manual partition registration, absent native ACID operations, and catalog state becoming stale after direct S3 changes. Rather than converting the entire lake at once, Grab prioritizes high-value tables and uses features such as Z-ordering; representative workloads cut one query from 70 seconds to 6, reduced daily S3 API costs by up to 95%, and halved cluster resource usage. Its open-sourced UnifiedSparkCatalog detects table formats through metastore metadata or path inference and routes operations to the appropriate Spark catalog without format-specific prefixes. Deployment also required resolving Hive lock contention, timestamp incompatibilities, and storage-tier cost spikes.
Data Warehouse Modernization: Roadmap, Architecture, and Services
Data warehouse modernization addresses legacy systems that cannot scale efficiently with growing data volumes, real-time analytics, machine learning, and self-service access. The proposed roadmap spans two to four years for large estates, moving from assessment and architecture design through high-impact workload migration, governance embedding, and optimization rather than relying on a risky big-bang cutover. Its target architecture favors a lakehouse or enhanced cloud data warehouse, with open formats such as Apache Iceberg or Delta Lake, separate compute and storage, and Bronze, Silver, and Gold layers supporting incremental ELT and lineage. The source says modernization can reduce infrastructure maintenance costs by 30–50%, compress query latency from hours to seconds, and halve redundant ETL pipelines, while also improving governance for sensitive data and enabling BI, machine learning, and generative AI workloads on a shared foundation.
Introducing the Agentic CDP: A New Species of CDP for a New Era of Agents
Traditional customer data platforms (CDPs) were built for human-managed, batch-based campaigns, but the post argues that agentic buying requires millisecond speed, hyper-personalization, and richer context. It contrasts the familiar Golden Record with Golden Context, which combines customer data with current business goals and the history and outcomes of prior decisions. The proposed Agentic CDP uses “Infinity Campaigns,” always-on engagement loops that use LLMs and agents to adapt messaging, timing, and channels for individuals. It is also embedded in the data foundation, bringing customer, business, and decision context together under existing governance, and is designed for agents and humans from the outset. Databricks presents CustomerLake as an implementation of these principles for its platform.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.