---
title: "Accelerate GPU training on Unity Catalog tables with UCDataset"
description: "Databricks released UCDataset in Beta on AI Runtime, a native PyTorch data loader that streams Unity Catalog tables directly to every GPU in a training job without exports or custom loader code. The bottleneck in GPU training is often data delivery, not the model, and common workarounds like exporting to Parquet or using toPandas() are slow or do not scale. UCDataset reads a table’s files directly from cloud storage, splits matching rows evenly across all GPUs and workers, and applies filters and deletion vectors as data is read. On a simulated production workload with a 300 GB feature table across 8 NVIDIA H100 GPUs, UCDataset cut training time by 10x compared to reading an exported Parquet copy. The package is available on PyPI and reads Delta tables registered in Unity Catalog, including tables with deletion vectors."
---

# Accelerate GPU training on Unity Catalog tables with UCDataset

[Databricks](https://yomu.fyi/company/databricks) · John Spencer, Milan Stefanovic, Bruce Fontaine, Tejas Sundaresan · Nov 9, 2026

**Type:** Announcement

## Summary

Databricks released UCDataset in Beta on AI Runtime, a native PyTorch data loader that streams Unity Catalog tables directly to every GPU in a training job without exports or custom loader code. The bottleneck in GPU training is often data delivery, not the model, and common workarounds like exporting to Parquet or using toPandas() are slow or do not scale. UCDataset reads a table’s files directly from cloud storage, splits matching rows evenly across all GPUs and workers, and applies filters and deletion vectors as data is read. On a simulated production workload with a 300 GB feature table across 8 NVIDIA H100 GPUs, UCDataset cut training time by 10x compared to reading an exported Parquet copy. The package is available on PyPI and reads Delta tables registered in Unity Catalog, including tables with deletion vectors.

## Context

GPUs often sit idle because getting data from storage to the GPU is slow. Common workarounds—exporting to Parquet, pulling the whole table into memory with toPandas(), or streaming through a warehouse—each introduce delays, extra storage costs, or scaling limits. UCDataset aims to eliminate these bottlenecks by making Unity Catalog tables a first-class data source for distributed PyTorch training.

## Approach / What changed

UCDataset is a native PyTorch data loader that streams Unity Catalog tables directly to GPUs by reading the table’s files from cloud storage, splitting matching rows evenly across every GPU and data loader worker, and applying filters and deletion vectors as data is read.

## Takeaways

- UCDataset streams Unity Catalog tables directly to GPUs by reading the table’s files from cloud storage, eliminating the need for an export step or a second copy of the data.
- On a simulated production workload with a 300 GB feature table across 8 NVIDIA H100 GPUs, UCDataset reduced training time by 10x compared to loading data from an exported Parquet copy.
- UCDataset enforces Unity Catalog permissions on AI Runtime, but in Beta it cannot read tables that have row filters or column masks.

## FAQ

### What is UCDataset?

UCDataset is a native PyTorch data loader that streams Unity Catalog tables directly to the GPUs in a training job. It is available in Beta on Databricks AI Runtime.

### Do I need to export Unity Catalog tables to Parquet to train on them?

No. UCDataset reads the live table in Unity Catalog, so there is no export step and no second copy to store or keep in sync. It applies your filter and the table's deletion vectors as it reads.

### Does UCDataset enforce Unity Catalog permissions?

Yes. On AI Runtime, training jobs read data as the identity that runs them, and Unity Catalog permissions on the table apply. In the Beta, UCDataset cannot read tables that have row filters or column masks.

**Tags:** [AI Runtime](https://yomu.fyi/topic/ai-runtime), [Databricks](https://yomu.fyi/topic/databricks), [GPU](https://yomu.fyi/topic/gpu), [PyTorch](https://yomu.fyi/topic/pytorch), [Unity Catalog](https://yomu.fyi/topic/unity-catalog)

- Source: [Databricks](https://www.databricks.com/blog/accelerate-gpu-training-unity-catalog-tables-ucdataset)
- Source URL: https://www.databricks.com/blog/accelerate-gpu-training-unity-catalog-tables-ucdataset
- Ingested by Yomu: 2026-11-09T18:00:12.545Z

[Read original post](https://www.databricks.com/blog/accelerate-gpu-training-unity-catalog-tables-ucdataset)
