Loading…
Apache Iceberg
13 posts about Apache Iceberg. Every summary links to the original.
Introducing OpenSharing: the Next Evolution of Delta Sharing for the Agentic Era
OpenSharing is presented as the next evolution of Delta Sharing, extending an open zero-copy data-sharing protocol from tables and files to models, agents, semantic context, unstructured data, and reusable AI logic. The protocol is now an independent open-source project hosted by the Linux Foundation, while Databricks OpenSharing adds Unity Catalog governance and audit logging, Marketplace discoverability, and enterprise features. Genie Agent Sharing supports governed AI experiences across organizational boundaries, with controls for proprietary instructions, data access, daily prompt quotas, and row exports. SecureConnect removes per-recipient firewall changes through a Databricks-managed proxy, while Global Distribution uses local replicas to reduce egress fees and latency. The launch also supports Apache Iceberg REST Catalog API, external catalogs, and on-premises storage partners; providers define shares in Unity Catalog, recipients query live data through existing tools, and governance enforces access controls.
Huey Han, Harish Gaur, Akram Chetibi, Mengxi ChenWhat’s new with Unity Catalog at Data + AI Summit 2026
At Data + AI Summit 2026, Unity Catalog announcements position the catalog as a runtime governance layer for enterprise data and AI, organized around control, context, and choice. Control additions include Unity AI Gateway for governing models, agents, MCP services, skills, and tools; contextual service policies can allow, deny, or require approval for runtime actions, while budgets, hard caps, tracing, and guardrails address spend, investigation, and safety. Context additions include Glossary and Domains for business meaning and scoped asset organization, plus Metrics that standardize KPIs for SQL, BI tools, APIs, and agents; Genie Ontology is described as a continuously learned enterprise context layer. Choice additions span cross-cloud and cross-region addressability, managed disaster recovery, Delta and Iceberg interoperability, multimodal and geospatial types, and open sharing of data, AI assets, and applications across organizations.
The Unity Catalog Product and Engineering TeamAccelerate search queries with full-text search indexes on Databricks
Databricks introduces full-text search indexes in Beta on Databricks Runtime 18.2 to accelerate substring and keyword queries on large open-format tables without changing their layouts. The indexes tokenize text columns into a compact lookup structure mapping tokens to matching rows; at query time, the engine uses it to identify candidate files and skip most of the table. They are maintained asynchronously, require no query hints, preserve complete results when stale by scanning indexed and non-indexed data as needed, and support Unity Catalog managed Delta and Iceberg tables on serverless and classic compute. A Trust and Safety team reported a substring search running more than 100x faster on a petabyte-scale table, while Liquid clustering remains complementary because it optimizes column-value filters rather than text within fields.
Yu Xu, Yingyi Bu, Ivan VezilićWhat is an open lakehouse? Open data standards, explained.
The piece defines an open lakehouse as a lakehouse whose storage, table format, processing engine, catalog, and ML and AI tooling use open standards and remain interchangeable. It contrasts this architecture with warehouses, lakes, and proprietary lakehouses, emphasizing low-cost object storage, ACID transactions, governance, schema guarantees, and the ability to change engines without rewriting data. Its reference stack combines open table formats such as Delta Lake and Apache Iceberg with Apache Parquet, Apache Spark, Unity Catalog, and MLflow, while allowing engines including DuckDB, Trino, and PyIceberg to work on the same data. The article also distinguishes open standards from open-source code, explains that a table format is only one layer of the stack, and states that the components can be self-hosted or consumed through a managed service.
Lisa CaoAnnouncing New OpenSharing and Marketplace capabilities for the AI era
Databricks announces new OpenSharing and Marketplace capabilities aimed at sharing data, AI assets, and partner applications without moving data. OpenSharing, a Linux Foundation project and evolution of Delta Sharing, adds vendor-neutral sharing for Agent Skills, AI models, unstructured data, Iceberg clients, Lakebase tables and change data feed, plus governed multi-cloud connectivity and agentic sharing through Genie Agent Sharing. The release also introduces identity resolution in Databricks Clean Rooms, where partners such as LiveRamp and Acxiom can work against protected first-party data without seeing raw customer records. Third-party apps are now available through Databricks Marketplace, allowing customers to deploy them in their workspaces without separate infrastructure, while providers gain distribution. Marketplace Commit Drawdown lets customers use existing Databricks universal commits to acquire Marketplace data and AI solutions.
Akram Chetibi, Harish Gaur, Huey Han, Tia Chang, Mengxi Chen, Lin Zhou, DJ SharkeyGeospatial Unbounded: Spatial SQL GA with AI/BI Maps, Delta Sharing, and Iceberg v3
Databricks announces Spatial SQL as Generally Available, positioning its platform for geospatial analysis without separate spatial databases, warehouses, and mapping tools. It supports native GEOMETRY columns in Delta or Iceberg, more than 90 OGC-compliant ST_* functions, spatial joins, and boolean set operations. AI/BI dashboards can render Geometry and Geography columns as maps, while Genie can generate spatial queries and dashboards and respect Unity Catalog row filters. Geo columns are supported by Delta Sharing, and Databricks can read and write managed Iceberg tables or read externally written Iceberg tables with geospatial types in Iceberg v3. Benchmarks show eight of twelve SpatialBench queries improved since Public Preview, with gains from 20% to 15X, while areal boolean operations are twice as fast on average versus prior versions.
Kent MartenAzure Databricks at Data + AI Summit 2026 featuring Industry Leaders and Partners
Data + AI Summit 2026 brings Databricks and Microsoft leaders, partners, and customers together June 15–18, with in-person and virtual programming focused on Azure Databricks. The collaboration presents Azure Databricks as a first-party Microsoft offering for unifying data, analytics, and AI on a secure, scalable foundation, with sessions covering ecosystem integration, federated analytics, governance, modernization, and AI applications. One technical example introduces zero-copy federation between Azure Data Manager for Energy and Databricks compute, preserving ADME as the source of truth while avoiding large-scale data copies. Another shows Unity Catalog External Locations extending governed access to Microsoft OneLake without ETL pipelines, while customer sessions describe Apache Iceberg and Apache Spark integration, fragmented data consolidation, and production-grade finance workflows using Azure Document Intelligence.
Kiriana StukasPractical Data Lakehouse Examples and Use Cases
The article presents practical data lakehouse patterns for streaming analytics, IoT pipelines, machine learning workflows, and enterprise reporting, addressing the gap between theoretical definitions and deployable examples. It explains how a lakehouse combines low-cost, schema-flexible object storage with schema enforcement, ACID transactions, data versioning, lineage tracking, and query performance, allowing SQL and ML workloads to use shared open-format tables. Examples include second-level fraud detection, medallion-based historical analytics, predictive maintenance from sensor data, and governed customer 360 profiles. The implementation guidance covers raw storage and partitioning, centralized catalogs, decoupled compute, role-based access, time travel, migration coexistence, SLAs, observability, and lifecycle policies. Together, these patterns are presented as a unified architecture that reduces duplication and data movement while supporting governed analytics at scale.
Databricks StaffIntroducing Cross-Engine ABAC
Cross-engine ABAC is announced in Beta, extending Unity Catalog's fine-grained governance to external engines through Iceberg REST Catalog APIs. It supports tag-based row filters and column masks, including conditional logic and SQL UDFs, while allowing policies to be defined once and enforced across engines. For an external query, the engine sends a scan request, Unity Catalog evaluates entitlements and applicable policies, and returns a filtered scan plan before the engine processes authorized files. Enforcement remains at the catalog layer, so engines need not implement governance logic and can use the open scan APIs. Apache Spark is supported today through Iceberg-Spark and Delta-Spark connectors, with Starburst and DuckDB integrations coming soon; the Beta also points toward Apache Iceberg label exchange for future governance metadata sharing.
Alex Jiang, Alex Reid, Michelle LeonAdvancing Apache Iceberg on Databricks: Iceberg v3 GA, Open Sharing, and Unified Governance
Databricks announces a broad set of Apache Iceberg capabilities in Unity Catalog, spanning General Availability, previews, and beta releases. Managed Iceberg is GA, supporting table creation, reads, writes, optimization, governance, and sharing, while Iceberg v3 adds deletion vectors, row tracking, and VARIANT across managed, foreign, and UniForm-enabled tables. Unity Catalog also federates external catalogs, vends credentials, shares live data with Iceberg REST-compatible clients through Delta Sharing, and applies attribute-based access control during server-side scan planning for supported external engines. These capabilities are presented as a unified approach to open APIs, cross-engine governance, zero-copy sharing, and production performance without copying data. The post also outlines Iceberg v4 and a proposal for Delta 5.0 to adopt an adaptive metadata tree structure.
Jason Reid, Ryan Blue, Daniel Weeks, Michelle LeonThe Convergence of Open Table Formats and Open Catalogs: Catalog Commits is Generally Available
Catalog Commits is generally available for Unity Catalog managed Delta tables, aligning Delta with Iceberg’s catalog-oriented model and making the catalog responsible for table discovery, access, and latest table state. The change addresses three coordination problems: metadata “split brain” when engines write directly to storage, fragmented multi-engine governance, and the historical inability to coordinate atomic writes across multiple tables. With Catalog Commits enabled, Unity Catalog brokers table access through standardized APIs, keeping catalog and table state synchronized and enabling consistent authorization, holistic auditability, automated optimizations, and multi-statement, multi-table ACID transactions on Databricks. The release supports Databricks products and engines including Delta Spark, Delta Flink, Starburst Trino, DuckDB, and StreamNative, while Delta Kernel provides a shared path for connector support.
Benjamin Mathew, Michelle Leon, Lukas Rupprecht, Ryan JohnsonInteroperability between Unity Catalog and Google BigQuery via catalog federation
Databricks and Google Cloud announce bidirectional catalog federation that lets customers access the same data from Unity Catalog or BigQuery without duplication. The interoperability uses Apache Iceberg REST catalog APIs, enabling both platforms to read and write the same tables through a common open interface. Google Cloud's catalog federation is in preview for reading Unity Catalog-managed tables from BigQuery, while Google Cloud's Lakehouse federation is in private preview for Databricks users to govern and read foreign Iceberg tables. Unity Catalog's policies, fine-grained access controls, and lineage tracking apply to federated data, and users can query it with Genie using natural language. The announcement positions open standards as a path to interoperability across data platforms and previews future catalog metadata exchange through Iceberg REST catalog APIs.
John Spencer, Jason ReidReal-Time Decisioning for AI Agents: Why you Need a Customer Context Layer First
Real-time decisioning for AI agents depends on more than customer records: it requires a customer context layer that captures what people are doing now and connects that behavior to identity. The post positions the data platform as the center of a composable martech architecture, with agents and applications operating on shared data in open formats such as Apache Iceberg and Delta Lake. Snowplow’s described approach places structured event collection, schema validation, enrichment, and identity resolution before data reaches the platform, while operating in the customer’s cloud environment. It distinguishes historical profiles from granular behavioral event streams and argues that agent interaction outcomes should return to the foundation as first-class events. The resulting four-stage loop—collect, resolve and enrich, serve, and learn—supports simultaneous real-time and historical context, with decision quality dependent on source data coherence.
Alex Dean
