BigQuery in the Right Lane: Five Checkpoints for AI Data Pipelines
A five-checkpoint workflow for keeping BigQuery in the right lane: land and shape data once, tune partitions and history windows, and federate instead of copying.
Loading…
Topic
Apache Iceberg
23 posts about Apache Iceberg. Every summary links to the original.
- 23 posts
- 4 companies
- 2 explainers
See the idea
Apache Iceberg explainers
Apache Iceberg vs. Hive: The Table Format Trade-off
Apache Iceberg replaces Hive's partition-based metadata with a versioned catalog layer, trading operational simplicity for ACID guarantees, schema evolution, and cross-engine compatibility.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
The data lakehouse is a better data warehouse: 2026 benchmarks and proof
Data warehouses have been the standard for structured SQL analytics for two decades but cannot serve modern AI, ML, streaming, and unstructured data workloads. The data lakehouse now matches warehouse SQL performance while serving all these workloads from a single governed foundation. Databricks Lakehouse improved production SQL workload performance by 77% from a 2022 baseline, with BI queries improving 14% over a recent five-month period. Customer migrations show 90% faster queries and 30–40% cost reduction at Lumen Technologies, and 80–90% faster analytics at Trek Bicycle. The lakehouse eliminates redundant data copies, supports open formats like Delta Lake and Iceberg, and extends governance to AI agents through Unity Catalog and Unity Gateway.
Unlocking Data Portability: Preventing Catalog Lock-in with REGISTER and UNREGISTER APIs
The post explains how the REGISTER and UNREGISTER APIs in the Apache Iceberg REST catalog specification enable table portability between catalogs without copying or moving data files. REGISTER attaches an existing table to a new catalog by providing the metadata location, but using it alone creates a split-brain scenario where two catalogs both think they own the table, leading to inconsistent writes and data loss. The UNREGISTER endpoint, contributed to the specification, cleanly removes the table from the old catalog and returns the latest metadata pointer needed by the next catalog. The handoff process combines unregister, receiving the location, and register to ensure exactly one managing catalog at all times. Unity Catalog supports these endpoints in private preview, allowing organizations to move tables between catalogs as their architecture evolves.
Introducing Cloudflare Basin: an open, serverless data platform, now generally available
Cloudflare announced the general availability of Basin, a serverless data analytics platform built on Apache Iceberg and R2 Object Storage. Basin consists of three integrated products: Pipelines for ingesting and transforming events via SQL, Catalog for managing Iceberg metadata and automatic table maintenance, and SQL for distributed querying. The platform emphasizes speed with instant resource creation and query execution, scalability through metadata compaction and statistics-driven query planning, and openness by supporting any Iceberg-compatible engine like DuckDB and Snowflake. Basin offers cost efficiency with zero egress fees and usage-based pricing, charging only when data is ingested, processed, or queried. Early adopters have used Basin for use cases including real-time e-commerce optimization, billing metrics storage, and infrastructure telemetry analysis.
Read Restrictions and Catalog Labels: Unifying governance across engines and catalogs
The Apache Iceberg community has adopted two additions to the Iceberg REST Catalog — read restrictions and catalog labels — each addressing distinct governance challenges. Read restrictions standardize delegated enforcement: a catalog evaluates policy for a principal and returns column-projection actions and row-filter expressions from a bounded vocabulary for a trusted engine to enforce. Catalog labels enable portable governance by allowing catalogs to exchange opaque key-value metadata at table and column level, so a consuming catalog applies native policies without per-user calls to the source. Read restrictions suit engine-to-catalog access where the source catalog trusts the engine and policies fit the standard vocabulary, whereas catalog labels fit federation between heterogeneous catalogs needing scalable governance and independent audit trails. These building blocks, along with centralized enforcement via cross-engine ABAC, advance the vision of unified governance across the open lakehouse.
ClickHouse expands collaboration with Microsoft, bringing Fabric integration, deeper OneLake interoperability, and enterprise deployment flexibility
ClickHouse announced a significant expansion of its strategic collaboration with Microsoft at The European Microsoft Fabric + SQL Community Conference, including the public preview of a native ClickHouse workload for Microsoft Fabric that brings sub-second analytical query latency to OneLake. The company also made generally available the ability to read Apache Iceberg tables from OneLake using OneLake Table APIs, and introduced a public preview of writing Iceberg tables back to OneLake. These capabilities eliminate data duplication and ingestion steps, allowing teams to keep OneLake as the governed single source of truth while leveraging ClickHouse’s speed. ClickHouse Bring Your Own Cloud (BYOC) is now available in the Microsoft Marketplace, enabling customers to run ClickHouse Cloud inside their own Azure tenant with full control over IAM, networking, and encryption. The integrations aim to serve real-time agentic workloads, interactive dashboards, and observability without separate cluster provisioning or ETL pipelines.
ClickHouse now writes Apache Iceberg tables to Microsoft OneLake
ClickHouse previously wrote Iceberg tables to object storage without catalog registration, limiting their discoverability. Now it writes Iceberg tables directly to Microsoft OneLake, registering in the catalog via Iceberg REST Table APIs and securing with OneLake Security. Use cases include AI agents monitoring order data and publishing real-time anomaly detection results to OneLake, and applications writing directly via ClickHouse without a separate ingestion layer. The connection uses the DataLakeCatalog engine with catalog_type=‘onelake’, warehouse parameter combining workspace and lakehouse IDs, and Microsoft Entra ID service principal authentication; writing uses standard INSERT INTO syntax with backtick-quoted namespace.table names. OneLake automatically syncs Iceberg and Delta Lake formats, provides governance with lineage tracking and data protection, and its security APIs are opening to third-party engines; this is the first catalog to support writes from ClickHouse, with more catalogs planned.
The next era of the open lakehouse: Apache Iceberg™ v3 in Public Preview on Databricks
Databricks has placed support for Apache Iceberg v3 into Public Preview, bringing new table-format capabilities to its open lakehouse. Unity Catalog Managed Iceberg v3 tables support Row Lineage, Deletion Vectors, and the VARIANT type, while interoperability extends these features to foreign Iceberg tables. Row IDs and modification sequence numbers identify changed rows, and lightweight delete files avoid immediate Parquet rewrites, with data manipulation reported as up to 10x faster than copy-on-write. VARIANT stores evolving semi-structured payloads beside relational columns for direct SQL querying, while Unity Catalog provides cross-catalog governance and Delta-Iceberg interoperability without data duplication. Iceberg v3 is available on Databricks Runtime 18.0+ with Unity Catalog enabled; Databricks is also advancing Iceberg v4 metadata proposals.
Unifying governance across engines and catalogs in the Open Lakehouse
Apache Iceberg’s REST Catalog additions—read restrictions and catalog labels—address different cross-platform governance problems: delegating enforcement to trusted engines and carrying governance context across federated catalogs. Read restrictions let a catalog evaluate policy for a principal and return bounded column-projection actions and row-filter expressions that a trusted engine must enforce, while leaving trust establishment outside the protocol. Catalog labels exchange opaque key-value metadata at table and column level, allowing a consuming catalog to map classifications locally, preserve native policy expressiveness, and maintain its own audit trail. Together with centralized enforcement through Cross-Engine ABAC, they provide distinct options for direct access and catalog federation, although read restrictions cannot express policies outside their standardized vocabulary.
One Write Path, Multiple Entries: How Rippling uses Unity Catalog-managed Apache Iceberg™
Rippling describes an architecture in which Databricks writes some tables while Trino, Snowflake, and other lakehouse engines read them, making catalog ownership a design decision rather than a metadata detail. Its rule is that the catalog follows the writer: AWS-native workloads use Glue, Databricks workloads use Unity Catalog, and Snowflake-produced shared data uses Horizon Catalog with Snowflake-managed Iceberg tables. For selected Databricks outputs, Unity Catalog-managed Apache Iceberg stores table data and metadata in Rippling-owned S3, exposes access through the Iceberg REST Catalog API or federation, and uses credential vending for scoped storage access. This lets downstream engines read Parquet directly without duplicate materialization or producer-side query execution. Databricks also provides Predictive Optimization for maintenance, while the trade-off is continued Unity Catalog dependence for metadata, permissions, lineage, and lifecycle.
Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption
Grab is transitioning its petabyte-scale data lake from Hive Parquet tables managed through the Hive Metastore to a table-centric architecture centered on Apache Iceberg, while retaining storage-compute separation on S3. The move addresses catalog listing latency, severe small-file fragmentation, manual partition registration, absent native ACID operations, and catalog state becoming stale after direct S3 changes. Rather than converting the entire lake at once, Grab prioritizes high-value tables and uses features such as Z-ordering; representative workloads cut one query from 70 seconds to 6, reduced daily S3 API costs by up to 95%, and halved cluster resource usage. Its open-sourced UnifiedSparkCatalog detects table formats through metastore metadata or path inference and routes operations to the appropriate Spark catalog without format-specific prefixes. Deployment also required resolving Hive lock contention, timestamp incompatibilities, and storage-tier cost spikes.
Introducing OpenSharing: the Next Evolution of Delta Sharing for the Agentic Era
OpenSharing is presented as the next evolution of Delta Sharing, extending an open zero-copy data-sharing protocol from tables and files to models, agents, semantic context, unstructured data, and reusable AI logic. The protocol is now an independent open-source project hosted by the Linux Foundation, while Databricks OpenSharing adds Unity Catalog governance and audit logging, Marketplace discoverability, and enterprise features. Genie Agent Sharing supports governed AI experiences across organizational boundaries, with controls for proprietary instructions, data access, daily prompt quotas, and row exports. SecureConnect removes per-recipient firewall changes through a Databricks-managed proxy, while Global Distribution uses local replicas to reduce egress fees and latency. The launch also supports Apache Iceberg REST Catalog API, external catalogs, and on-premises storage partners; providers define shares in Unity Catalog, recipients query live data through existing tools, and governance enforces access controls.
What’s new with Unity Catalog at Data + AI Summit 2026
At Data + AI Summit 2026, Unity Catalog announcements position the catalog as a runtime governance layer for enterprise data and AI, organized around control, context, and choice. Control additions include Unity AI Gateway for governing models, agents, MCP services, skills, and tools; contextual service policies can allow, deny, or require approval for runtime actions, while budgets, hard caps, tracing, and guardrails address spend, investigation, and safety. Context additions include Glossary and Domains for business meaning and scoped asset organization, plus Metrics that standardize KPIs for SQL, BI tools, APIs, and agents; Genie Ontology is described as a continuously learned enterprise context layer. Choice additions span cross-cloud and cross-region addressability, managed disaster recovery, Delta and Iceberg interoperability, multimodal and geospatial types, and open sharing of data, AI assets, and applications across organizations.
Accelerate search queries with full-text search indexes on Databricks
Databricks introduces full-text search indexes in Beta on Databricks Runtime 18.2 to accelerate substring and keyword queries on large open-format tables without changing their layouts. The indexes tokenize text columns into a compact lookup structure mapping tokens to matching rows; at query time, the engine uses it to identify candidate files and skip most of the table. They are maintained asynchronously, require no query hints, preserve complete results when stale by scanning indexed and non-indexed data as needed, and support Unity Catalog managed Delta and Iceberg tables on serverless and classic compute. A Trust and Safety team reported a substring search running more than 100x faster on a petabyte-scale table, while Liquid clustering remains complementary because it optimizes column-value filters rather than text within fields.
What is an open lakehouse? Open data standards, explained.
The piece defines an open lakehouse as a lakehouse whose storage, table format, processing engine, catalog, and ML and AI tooling use open standards and remain interchangeable. It contrasts this architecture with warehouses, lakes, and proprietary lakehouses, emphasizing low-cost object storage, ACID transactions, governance, schema guarantees, and the ability to change engines without rewriting data. Its reference stack combines open table formats such as Delta Lake and Apache Iceberg with Apache Parquet, Apache Spark, Unity Catalog, and MLflow, while allowing engines including DuckDB, Trino, and PyIceberg to work on the same data. The article also distinguishes open standards from open-source code, explains that a table format is only one layer of the stack, and states that the components can be self-hosted or consumed through a managed service.
Announcing New OpenSharing and Marketplace capabilities for the AI era
Databricks announces new OpenSharing and Marketplace capabilities aimed at sharing data, AI assets, and partner applications without moving data. OpenSharing, a Linux Foundation project and evolution of Delta Sharing, adds vendor-neutral sharing for Agent Skills, AI models, unstructured data, Iceberg clients, Lakebase tables and change data feed, plus governed multi-cloud connectivity and agentic sharing through Genie Agent Sharing. The release also introduces identity resolution in Databricks Clean Rooms, where partners such as LiveRamp and Acxiom can work against protected first-party data without seeing raw customer records. Third-party apps are now available through Databricks Marketplace, allowing customers to deploy them in their workspaces without separate infrastructure, while providers gain distribution. Marketplace Commit Drawdown lets customers use existing Databricks universal commits to acquire Marketplace data and AI solutions.
Geospatial Unbounded: Spatial SQL GA with AI/BI Maps, Delta Sharing, and Iceberg v3
Databricks announces Spatial SQL as Generally Available, positioning its platform for geospatial analysis without separate spatial databases, warehouses, and mapping tools. It supports native GEOMETRY columns in Delta or Iceberg, more than 90 OGC-compliant ST_* functions, spatial joins, and boolean set operations. AI/BI dashboards can render Geometry and Geography columns as maps, while Genie can generate spatial queries and dashboards and respect Unity Catalog row filters. Geo columns are supported by Delta Sharing, and Databricks can read and write managed Iceberg tables or read externally written Iceberg tables with geospatial types in Iceberg v3. Benchmarks show eight of twelve SpatialBench queries improved since Public Preview, with gains from 20% to 15X, while areal boolean operations are twice as fast on average versus prior versions.
Azure Databricks at Data + AI Summit 2026 featuring Industry Leaders and Partners
Data + AI Summit 2026 brings Databricks and Microsoft leaders, partners, and customers together June 15–18, with in-person and virtual programming focused on Azure Databricks. The collaboration presents Azure Databricks as a first-party Microsoft offering for unifying data, analytics, and AI on a secure, scalable foundation, with sessions covering ecosystem integration, federated analytics, governance, modernization, and AI applications. One technical example introduces zero-copy federation between Azure Data Manager for Energy and Databricks compute, preserving ADME as the source of truth while avoiding large-scale data copies. Another shows Unity Catalog External Locations extending governed access to Microsoft OneLake without ETL pipelines, while customer sessions describe Apache Iceberg and Apache Spark integration, fragmented data consolidation, and production-grade finance workflows using Azure Document Intelligence.
Practical Data Lakehouse Examples and Use Cases
The article presents practical data lakehouse patterns for streaming analytics, IoT pipelines, machine learning workflows, and enterprise reporting, addressing the gap between theoretical definitions and deployable examples. It explains how a lakehouse combines low-cost, schema-flexible object storage with schema enforcement, ACID transactions, data versioning, lineage tracking, and query performance, allowing SQL and ML workloads to use shared open-format tables. Examples include second-level fraud detection, medallion-based historical analytics, predictive maintenance from sensor data, and governed customer 360 profiles. The implementation guidance covers raw storage and partitioning, centralized catalogs, decoupled compute, role-based access, time travel, migration coexistence, SLAs, observability, and lifecycle policies. Together, these patterns are presented as a unified architecture that reduces duplication and data movement while supporting governed analytics at scale.
Introducing Cross-Engine ABAC
Cross-engine ABAC is announced in Beta, extending Unity Catalog's fine-grained governance to external engines through Iceberg REST Catalog APIs. It supports tag-based row filters and column masks, including conditional logic and SQL UDFs, while allowing policies to be defined once and enforced across engines. For an external query, the engine sends a scan request, Unity Catalog evaluates entitlements and applicable policies, and returns a filtered scan plan before the engine processes authorized files. Enforcement remains at the catalog layer, so engines need not implement governance logic and can use the open scan APIs. Apache Spark is supported today through Iceberg-Spark and Delta-Spark connectors, with Starburst and DuckDB integrations coming soon; the Beta also points toward Apache Iceberg label exchange for future governance metadata sharing.
Advancing Apache Iceberg on Databricks: Iceberg v3 GA, Open Sharing, and Unified Governance
Databricks announces a broad set of Apache Iceberg capabilities in Unity Catalog, spanning General Availability, previews, and beta releases. Managed Iceberg is GA, supporting table creation, reads, writes, optimization, governance, and sharing, while Iceberg v3 adds deletion vectors, row tracking, and VARIANT across managed, foreign, and UniForm-enabled tables. Unity Catalog also federates external catalogs, vends credentials, shares live data with Iceberg REST-compatible clients through Delta Sharing, and applies attribute-based access control during server-side scan planning for supported external engines. These capabilities are presented as a unified approach to open APIs, cross-engine governance, zero-copy sharing, and production performance without copying data. The post also outlines Iceberg v4 and a proposal for Delta 5.0 to adopt an adaptive metadata tree structure.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.