What Is an AI Factory? Four Stages From Data to Served Answer
An AI factory turns raw data and GPU-hours into served model output. This diagram breaks it into four stages — feed, shape, serve, restore — plus the control plane that owns them.
An AI runtime is the execution layer between model code and hardware: it feeds data to accelerators, places and reshards tensors, schedules work, and holds latency steady. Here are its four core jobs.
An AI runtime is the layer that sits between your model code and the accelerators it runs on. It decides how training data reaches each GPU, where each tensor block lives and how it is reshaped when it moves, when work runs and on which cores, and what happens when a host fails or inference traffic arrives. These decisions are invisible in the model definition but dominate throughput, cost, and tail latency. Reading the runtime as four separate jobs makes it easier to see which one is actually your bottleneck.
In the UCDataset example, streaming Unity Catalog table files straight to every GPU cut training time 10x versus reading an exported Parquet copy across 8 H100s. Exporting or calling toPandas() is slow and does not scale. The runtime layer, not the model, was the bottleneck.
Gather-and-broadcast refit rebuilds the whole tensor as an intermediate, so a 2.4 trillion parameter model moves roughly 9.6 TB per refit. A reshard collective treats refit as a shard-to-shard transfer and sends only the regions each destination GPU needs, reporting 2.7x to 5.7x speedups.
Pathways remaps blocks around a failed host instead of restarting the whole job. On the latency side, Holoscan groups operators into one pinned partition and isolates cores, cutting p99.99 latency about 5x in C++ and 9.6x in Python. Both are runtime decisions rather than model changes.
The posts behind it
Gravity compensation for a robot arm requires continuous inverse dynamics calculation based on precise models of link dimensions, masses, and moments of inertia, and must run with low jitter to avoid instability. NVIDIA Holoscan 5 introduces a robotic control framework that uses partitions to group chains of operators on a single thread, avoiding cross-thread handoffs and unnecessary syscalls to reduce latency and jitter. This tutorial builds three custom Holoscan operators wired into a graph and tested on Jetson Orin Nano, Jetson Thor, and x86 Ubuntu 24.04 systems. The operators use Pinocchio’s Recursive Newton-Euler Algorithm solver and the motorbridge library to read joint state and send torque commands, combining gravity feedforward with light stiffness and damping. Latency measurements show that pinning the control partition to dedicated CPU cores and isolating them with cgroups reduces p99.99 latency by roughly 5x for C++ and 9.6x for Python.
Databricks released UCDataset in Beta on AI Runtime, a native PyTorch data loader that streams Unity Catalog tables directly to every GPU in a training job without exports or custom loader code. The bottleneck in GPU training is often data delivery, not the model, and common workarounds like exporting to Parquet or using toPandas() are slow or do not scale. UCDataset reads a table’s files directly from cloud storage, splits matching rows evenly across all GPUs and workers, and applies filters and deletion vectors as data is read. On a simulated production workload with a 300 GB feature table across 8 NVIDIA H100 GPUs, UCDataset cut training time by 10x compared to reading an exported Parquet copy. The package is available on PyPI and reads Delta tables registered in Unity Catalog, including tables with deletion vectors.
Langfuse, the open-source AI observability platform acquired by ClickHouse in January 2026, launches the AI Gateway in public beta. The gateway serves as a central entry point to govern all AI traffic, logging every model call natively in Langfuse for cost tracking and analysis. It is built in Rust, adds under 3ms median latency with tracing on, and supports 1,000+ concurrent streams on a single 2 vCPU node. The gateway is MIT-licensed, deploys as a single 22MB Docker image, and requires only changing the base URL and using a gateway key. OpenAI and Anthropic are supported in beta, with more providers planned.
Large-model training with SPMD struggles when jobs become non-uniform, such as pipeline parallelism or RL training loops, and cannot survive hardware failures without restarting. Pathways addresses this by introducing a single Python client that describes the entire computation as a graph of blocks and tensor edges. A distributed runtime places blocks on virtual devices, moves and reshards tensors, orders operations across the cluster, and remaps around a failed host. This separates model building from execution, allowing reasoning about the whole cluster instead of one slice. Pathways runs JAX programs on TPUs and is available on Google Cloud.
Reinforcement learning post-training uses a disaggregated setup where inference and training run on separate GPU resources, requiring periodic weight transfers called refit. Training and inference instances employ different parallelism configurations, causing weights to be sharded across GPU groups differently. Conventional gather-and-broadcast refit recnstructs the full weight tensor as an intermediate representation, resulting in a logical payload of roughly twice the model’s weight size—for a 2.4 trillion parameter model, about 9.6 TB per refit. The Reshard collective reformulates refit as a DTensor-to-DTensor reshard operation, transferring only the exact shard regions each destiation GPU needs, while NCCL M2N Reshard groups replicas by NVLink domain to maximize bandwith. A NeMo RL implementation achieves 2.7x to 5.7x speedups over gather-and-broadcast refit and is open-sourced in the NVIDIA/nccl-extensions reposity.
Real-time fraud detection needs three feature types: recent activity (e.g., five-minute transaction count), spending baselines (30-day average and standard deviation), and current payment details. Databricks Feature Views address this by defining each feature once for both training and serving, producing point-in-time correct training data and online inference pipelines. The walkthrough follows a $900 payment from a Kafka stream through RollingWindow and SawtoothWindow aggregations to a fraud score from Model Serving. Feature Views prevent training-serving skew and data leakage; on-demand features compare the current amount against stored baselines at request time. Unity Catalog governs definitions, permissions, and lineage, while MLflow records model dependencies and Model Serving retrieves features from the Online Feature Store.
More like this
An AI factory turns raw data and GPU-hours into served model output. This diagram breaks it into four stages — feed, shape, serve, restore — plus the control plane that owns them.
An audit is a provenance chain: what was stored, where each fact came from, and what was used for each decision. This explainer maps the three links that keep agent behavior traceable and fixable.
Chrome DevTools Connect gives coding agents a live channel into the browser's DevTools so they can run browser tests and drive fix cycles instead of reasoning from source alone.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.