Automatic Upgrades: best practice features for your lakehouse tables

Summary

Automatic Upgrades gives Unity Catalog managed tables a hands-off path to adopting best-practice table features without manually auditing thousands of tables or running ALTER TABLE repeatedly. It observes table access over a 100-day window, verifies that every accessing Databricks client supports the feature, skips inactive or externally accessed tables, and applies eligible changes through a lightweight background job. New tables inherit a feature when it becomes a schema default, while explicit table properties take precedence. Supported capabilities include Row Tracking, Automatic Liquid Clustering, Deletion Vectors, Column Mapping, Parquet V2, Catalog Commits, and Checkpoint V2, with benefits spanning query speed, storage and write costs, interoperability, and reliability. Changes are visible in DESCRIBE HISTORY and Catalog Explorer, reversible per table, and not re-enabled after a user disables one.

Context

Adopting new open table format features has required teams to identify eligible tables, verify client compatibility, and run ALTER TABLE across potentially thousands of tables. The manual effort has prevented many teams from receiving improvements in performance, reliability, interoperability, and cost.

Approach / What changed

Automatic Upgrades observes access to Unity Catalog managed tables, verifies client compatibility over a 100-day window, skips tables it cannot fully verify, and applies eligible features through lightweight background ALTER TABLE jobs. It can also make verified features schema defaults for newly created tables, while preserving explicit table properties and allowing per-table reversals.

Takeaways

• A single accessing Databricks client that lacks support for a feature prevents Auto Upgrades from enabling it; tables accessed by external clients are currently out of scope.
• Automatic Upgrades can enable Row Tracking, Deletion Vectors, Column Mapping, Parquet V2, Catalog Commits, Checkpoint V2, and other listed capabilities as tables become eligible.
• Every automatic change is recorded distinctly in DESCRIBE HISTORY and Catalog Explorer; the system.storage.table_auto_upgrade_operations_history system table provides account-wide event visibility.

Related reading