---
title: "What Is an AI Runtime? Four Jobs Between Your Model and the Metal"
description: "An AI runtime is the execution layer between model code and hardware: it feeds data to accelerators, places and reshards tensors, schedules work, and holds latency steady. Here are its four core jobs."
---

# What Is an AI Runtime? Four Jobs Between Your Model and the Metal

**Type:** [What is it?](https://yomu.fyi/visual-explainers/category/what-is-it.md)

![A left-to-right spine runs from model code through a central AI Runtime panel to accelerators, with four numbered panels below the runtime for data delivery, placement and reshard, scheduling and latency, and failure and serving path.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/598403e4-6384-41df-9755-c0b808093487/image-0698283b-e645-43c7-8579-27b413c2f01d.webp)

> An AI runtime is the execution layer between model code and hardware: it feeds data to accelerators, places and reshards tensors, schedules work, and holds latency steady. Here are its four core jobs.

## Explanation

An AI runtime is the layer that sits between your model code and the accelerators it runs on. It decides how training data reaches each GPU, where each tensor block lives and how it is reshaped when it moves, when work runs and on which cores, and what happens when a host fails or inference traffic arrives. These decisions are invisible in the model definition but dominate throughput, cost, and tail latency. Reading the runtime as four separate jobs makes it easier to see which one is actually your bottleneck.

## FAQ

### Why can data delivery matter more than model speed in an AI runtime?

In the UCDataset example, streaming Unity Catalog table files straight to every GPU cut training time 10x versus reading an exported Parquet copy across 8 H100s. Exporting or calling toPandas() is slow and does not scale. The runtime layer, not the model, was the bottleneck.

### How does a runtime refit weights without moving a full model copy?

Gather-and-broadcast refit rebuilds the whole tensor as an intermediate, so a 2.4 trillion parameter model moves roughly 9.6 TB per refit. A reshard collective treats refit as a shard-to-shard transfer and sends only the regions each destination GPU needs, reporting 2.7x to 5.7x speedups.

### What should the runtime do when a host fails or tail latency spikes?

Pathways remaps blocks around a failed host instead of restarting the whole job. On the latency side, Holoscan groups operators into one pinned partition and isolates cores, cutting p99.99 latency about 5x in C++ and 9.6x in Python. Both are runtime decisions rather than model changes.

**Tags:** [AI Runtime](https://yomu.fyi/topic/ai-runtime.md), [GPU](https://yomu.fyi/topic/gpu.md), [Latency](https://yomu.fyi/topic/latency.md), [Model Serving](https://yomu.fyi/topic/model-serving.md), [Multi-GPU](https://yomu.fyi/topic/multi-gpu.md), [Orchestration](https://yomu.fyi/topic/orchestration.md)

## Related reading

### [Make a Robot Arm Feel Weightless with NVIDIA Holoscan 5](https://yomu.fyi/post/make-a-robot-arm-feel-weightless-with-nvidia-holoscan-5.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Nov 9, 2026
- Type: Tutorial

Gravity compensation for a robot arm requires continuous inverse dynamics calculation based on precise models of link dimensions, masses, and moments of inertia, and must run with low jitter to avoid instability. NVIDIA Holoscan 5 introduces a robotic control framework that uses partitions to group chains of operators on a single thread, avoiding cross-thread handoffs and unnecessary syscalls to reduce latency and jitter. This tutorial builds three custom Holoscan operators wired into a graph and tested on Jetson Orin Nano, Jetson Thor, and x86 Ubuntu 24.04 systems. The operators use Pinocchio’s Recursive Newton-Euler Algorithm solver and the motorbridge library to read joint state and send torque commands, combining gravity feedforward with light stiffness and damping. Latency measurements show that pinning the control partition to dedicated CPU cores and isolating them with cgroups reduces p99.99 latency by roughly 5x for C++ and 9.6x for Python.


### [Accelerate GPU training on Unity Catalog tables with UCDataset](https://yomu.fyi/post/accelerate-gpu-training-on-unity-catalog-tables-with-ucdataset.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: John Spencer, Milan Stefanovic, Bruce Fontaine, Tejas Sundaresan
- Published: Nov 9, 2026
- Type: Announcement

Databricks released UCDataset in Beta on AI Runtime, a native PyTorch data loader that streams Unity Catalog tables directly to every GPU in a training job without exports or custom loader code. The bottleneck in GPU training is often data delivery, not the model, and common workarounds like exporting to Parquet or using toPandas() are slow or do not scale. UCDataset reads a table’s files directly from cloud storage, splits matching rows evenly across all GPUs and workers, and applies filters and deletion vectors as data is read. On a simulated production workload with a 300 GB feature table across 8 NVIDIA H100 GPUs, UCDataset cut training time by 10x compared to reading an exported Parquet copy. The package is available on PyPI and reads Delta tables registered in Unity Catalog, including tables with deletion vectors.


### [Langfuse launches AI Gateway](https://yomu.fyi/post/langfuse-launches-ai-gateway.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Nov 9, 2026
- Type: Announcement

Langfuse, the open-source AI observability platform acquired by ClickHouse in January 2026, launches the AI Gateway in public beta. The gateway serves as a central entry point to govern all AI traffic, logging every model call natively in Langfuse for cost tracking and analysis. It is built in Rust, adds under 3ms median latency with tracing on, and supports 1,000+ concurrent streams on a single 2 vCPU node. The gateway is MIT-licensed, deploys as a single 22MB Docker image, and requires only changing the base URL and using a gateway key. OpenAI and Anthropic are supported in beta, with more providers planned.


### [What is Pathways? Single-controller training for large models](https://yomu.fyi/post/what-is-pathways-single-controller-training-for-large-models.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Sujeeth Jinesh, Shaurya Gupta, Luke Baumann, Monica Song, Ivan Nardini, Akanksha Gupta, Anthony Owusu, Jing Yang, Sadi Kneipp, Yu Li
- Published: Nov 9, 2026
- Type: Explainer

Large-model training with SPMD struggles when jobs become non-uniform, such as pipeline parallelism or RL training loops, and cannot survive hardware failures without restarting. Pathways addresses this by introducing a single Python client that describes the entire computation as a graph of blocks and tensor edges. A distributed runtime places blocks on virtual devices, moves and reshards tensors, orders operations across the cluster, and remaps around a failed host. This separates model building from execution, allowing reasoning about the whole cluster instead of one slice. Pathways runs JAX programs on TPUs and is available on Google Cloud.


### [Accelerating Reinforcement Learning Weight Refit with the Reshard Collective](https://yomu.fyi/post/accelerating-reinforcement-learning-weight-refit-with-the-reshard-coll.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Nov 6, 2026
- Type: Problem & solution

Reinforcement learning post-training uses a disaggregated setup where inference and training run on separate GPU resources, requiring periodic weight transfers called refit. Training and inference instances employ different parallelism configurations, causing weights to be sharded across GPU groups differently. Conventional gather-and-broadcast refit recnstructs the full weight tensor as an intermediate representation, resulting in a logical payload of roughly twice the model’s weight size—for a 2.4 trillion parameter model, about 9.6 TB per refit. The Reshard collective reformulates refit as a DTensor-to-DTensor reshard operation, transferring only the exact shard regions each destiation GPU needs, while NCCL M2N Reshard groups replicas by NVLink domain to maximize bandwith. A NeMo RL implementation achieves 2.7x to 5.7x speedups over gather-and-broadcast refit and is open-sourced in the NVIDIA/nccl-extensions reposity.


### [Real-time fraud detection with Feature Views](https://yomu.fyi/post/real-time-fraud-detection-with-feature-views.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Shyam Sankararaman, Hedvig Vahlberg
- Published: Nov 6, 2026
- Type: Tutorial

Real-time fraud detection needs three feature types: recent activity (e.g., five-minute transaction count), spending baselines (30-day average and standard deviation), and current payment details. Databricks Feature Views address this by defining each feature once for both training and serving, producing point-in-time correct training data and online inference pipelines. The walkthrough follows a $900 payment from a Kafka stream through RollingWindow and SawtoothWindow aggregations to a fraud score from Model Serving. Feature Views prevent training-serving skew and data leakage; on-demand features compare the current amount against stored baselines at request time. Unity Catalog governs definitions, permissions, and lineage, while MLflow records model dependencies and Model Serving retrieves features from the Online Feature Store.
