Loading…
Talk to all your data, wherever it lives
John Spencer
- Source
- Databricks
- Published
- Added to Yomu
Summary
Lakehouse Federation addresses the challenge of reasoning across enterprise data spread among AWS Glue, Snowflake, Oracle, BigQuery, Postgres, and legacy formats without first migrating it. By connecting external sources in place and syncing their metadata into Unity Catalog, Databricks applies shared permissions, lineage, and access controls while preserving source data. Federated comments and descriptions give Genie schema context, while Unity Catalog Semantics lets teams define governed metrics such as ROI once for consistent use across Genie, dashboards, and notebooks. The example connects an AWS Glue marketing database, carries its metadata, defines an ROI metric view, and asks Genie which campaigns led ROI last quarter. The post reports an immediate, accurate answer from live Glue data, and describes planned richer semantics, broader federation, and possible performance gains from managed tables.
Context
Enterprise data may be distributed across multiple platforms or stored in legacy proprietary formats, making migration to one system time-consuming. Business users nevertheless want natural-language answers that span those sources, with consistent governance and reliable business definitions.
Approach / What changed
Create a Lakehouse Federation connection to an external source, sync its tables and metadata into Unity Catalog without copying data, define reusable metrics with Unity Catalog Semantics, and query the federated data through Genie. The example uses an AWS Glue database containing marketing campaign data.
Takeaways
- Lakehouse Federation syncs external tables into Unity Catalog in place, avoiding data copies, migration work, and disruption to the source system.
- Source comments and descriptions can provide Genie with schema context; support is available for Glue and BigQuery, with expanded support for several other systems in preview.
- Unity Catalog metric views can define an ROI measure once on federated tables so Genie, AI/BI dashboards, and notebooks use the same business calculation.