---
title: "What Is an AI Factory? Power In, Tokens Out"
description: "An AI factory is a fixed power and cooling budget converted into model output. This explainer maps its four parts and the throughput-per-watt trade-offs that connect them."
---

# What Is an AI Factory? Power In, Tokens Out

**Type:** [What is it?](https://yomu.fyi/visual-explainers/category/what-is-it.md)

![A central AI factory panel labeled power budget to tokens, surrounded by four bordered parts: capacity envelope, compute fleet, fabric and scheduling, and serving and routing, with arrows into the center and a bottom strip of three short trade-off chips.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/c4c30ba6-5e75-4ee3-92fd-6f0f8f3ee379/image-645b1639-0b98-4a4f-93e1-2c17c240a0ec.webp)

> An AI factory is a fixed power and cooling budget converted into model output. This explainer maps its four parts and the throughput-per-watt trade-offs that connect them.

## Explanation

An AI factory is not simply a room of GPUs; it is a bounded power and cooling envelope that has to be converted into useful model output. Its essential parts are the capacity envelope, the accelerator fleet, the fabric and scheduling layer that keeps that fleet fed, and the serving layer that turns compute into tokens. Because the envelope is fixed, most gains come from reallocating it: dynamic power policies can fit more accelerators into the same kilowatt budget, but tail latency can rise. Engineers should read an AI factory as a throughput-per-watt system with verification and governance attached, not as a hardware shopping list.

## FAQ

### Does higher throughput always mean a better serving setup?

No, it is workload-specific. Quail's query-planner-plus-inference-engine design reached over a billion tokens per minute per H100 on a multi-join AI-SQL query, more than 10x faster than vLLM, but fell behind vLLM on agent trace processing. The lesson is to match the serving layer to the request shape rather than optimizing a single headline number.

**Tags:** [AI Factories](https://yomu.fyi/topic/ai-factories.md), [Data Centers](https://yomu.fyi/topic/data-centers.md), [GPU](https://yomu.fyi/topic/gpu.md), [Inference](https://yomu.fyi/topic/inference.md), [Model Serving](https://yomu.fyi/topic/model-serving.md), [Performance](https://yomu.fyi/topic/performance.md)

## Related reading

### [How NVIDIA DSX MaxLPS Maximizes AI Factory Throughput and Efficiency](https://yomu.fyi/post/how-nvidia-dsx-maxlps-maximizes-ai-factory-throughput-and-efficiency.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Sarah McKenney
- Published: Sep 28, 2026
- Type: Benchmark

AI factories often underutilize power capacity due to static provisioning for peak GPU draw. NVIDIA DSX MaxLPS uses policy-governed dynamic power allocation to reclaim stranded capacity. In a joint evaluation with Nscale using Kimi K2.5 workloads on GB300 NVL72 systems, DSX MaxLPS enabled 37% more GPUs (192 vs 140) within the same 264.4 kW budget. Aggregate throughput increased 49.2% to 1.62 million tokens/s, while per-instance throughput remained unchanged and median latency stayed within 5% of baseline. However, P99 time to first token increased 17%, highlighting the need to evaluate tail latency alongside capacity gains.


### [Deploy and manage coding agents at scale with the Unity Gateway CLI](https://yomu.fyi/post/deploy-and-manage-coding-agents-at-scale-with-the-unity-gateway-cli.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: The Databricks AI Product and Engineering Team
- Published: Sep 24, 2026
- Type: Announcement

Organizations face a challenge as new frontier models appear roughly every five days, making it difficult to balance flexibility with governance. Databricks introduces the Unity Gateway CLI, which lets admins centrally configure approved coding agents, models, MCP servers, skills, and spending policies. Developers launch any approved agent with a single ug command, and the CLI handles authentication and applies the published settings. Features include Smart Routing that selects cost-appropriate models per task, budget-aware defaults that switch to lower-cost options as spending thresholds are reached, and tracing to capture tool calls and skill invocations for waste analysis. In internal use, the system processed over 61 billion input tokens across roughly 360,000 requests, and tracing helped identify bugs that saved an estimated 1.2 million per year.


### [Efficient MoE Training for Biological Foundation Models](https://yomu.fyi/post/efficient-moe-training-for-biological-foundation-models.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Sep 24, 2026
- Type: Tutorial

Mixture-of-experts (MoE) architectures scale model capacity more efficiently than dense transformers by activating only a subset of experts per token, but naive implementations suffer from fragmented expert computation, routing overhead, and memory pressure. NVIDIA Transformer Engine (TE) addresses these bottlenecks with optimized primitives including GroupedLinear for batched expert GEMMs, MXFP8 low-precision training, and a fused GroupedMLP kernel that combines quantization, SwiGLU activation, and routing-weight scaling. The BioNeMo recipe implements these techniques for biological foundation models, using TE’s autocast API for MXFP8 precision and the Sequential API to fuse the gate-up, ScaledSwiGLU, and down projections into a single operation. On eight NVIDIA B200 GPUs, the recipe achieved up to 2.21x the training throughput of the Hugging Face baseline for a Mixtral-8x7B model. The tutorial provides prerequisites, code examples, and configuration commands to reproduce the optimizations.


### [Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine](https://yomu.fyi/post/hitting-a-billion-tokens-per-minute-on-one-gpu-by-combining-a-query-pl.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Sep 24, 2026
- Type: Problem & solution

Modal and CMU built Quail, a query-aware inference layer for AI-SQL that combines a query planner with an inference engine to process structured LLM transformations of database entries. On a multi-join query, Quail achieves over a billion tokens per minute per H100 GPU, over 10x faster than vLLM, costing under 6 cents per billion tokens on Modal. The gain comes from using the query plan to order requests for better KV cache management, including a revised Hydragen-style cascade attention and batching small requests to reduce host overhead. On a new AI-SQL benchmark, Quail runs 1.84x faster than vLLM, but falls behind on agent trace processing. The authors see this as part of a trend toward custom inference engines for specific workloads, with a reusable core like databases post-DataFusion, and note that much work was done by prompting coding agents.


### [Reproducing OLMo 3 7B Pre-training in MaxText: case study of large scale training on TPUs](https://yomu.fyi/post/reproducing-olmo-3-7b-pre-training-in-maxtext-case-study-of-large-scal.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Gagik Amirkhanyan, Ran Ran, Aireen Mei, Matt Davidow, TPU Inference Software Engineering Team, Google Cloud, AI2 Team
- Published: Sep 24, 2026
- Type: Problem & solution

The post describes the reproduction of AI2’s OLMo 3 7B pre-training using MaxText on Google Cloud TPUs, covering both stages, matching the reference loss curve and held-out metrics. PyTorch weights were converted to JAX with logit parity, KL divergence 1.5e-3 and 98.75% top-1 agreement, and held-out evaluations caught a data loader bug that made incorrect gains appear. The run demonstrated reliability via exact checkpoint-resume with zero loss delta, resilience to a 75% capacity loss via resize with 1% throughput preservation, and seamless TPU generation change. Performance optimizations achieved 44.5% MFU on Ironwood, and head reshaping from 32 heads by 128 to 16 by 256 yielded 12.4% faster throughput at identical parameters and FLOPs. The post emphasizes that faithful reproduction requires verification on generalization metrics and provides instructions for readers to reproduce the run with open sourced code.


### [Validate GPU Cluster Readiness Before AI Workloads Land](https://yomu.fyi/post/validate-gpu-cluster-readiness-before-ai-workloads-land.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Sep 23, 2026
- Type: Explainer

GPU clusters can pass standard health checks yet fail under real AI training workloads due to a single degraded GPU, network link, or misrouted traffic. NVIDIA Cluster Readiness Engine (NVCRE) is an open source Kubernetes controller that runs real distributed workloads across topology-aware node groups before production jobs land, measuring results and reporting which nodes failed each test. Its layered API uses Custom Resource Definitions for Certification, Workflow, and Job resources, attributing each failure to a specific node and category such as NCCL bandwidth or training goodput. The WorkloadRun API handles platform detection, NCCL settings, shared-memory volumes, and gang scheduling, so operators do not need to write those manifests by hand. NVCRE integrates with NVSentinel for passive telemetry and is available under Apache 2.0, with a roadmap for inference and automated lifecycle validation.