What Is an AI Factory? Power In, Tokens Out
An AI factory is a fixed power and cooling budget converted into model output. This explainer maps its four parts and the throughput-per-watt trade-offs that connect them.
Loading…
Topic
Model Serving
40 posts about Model Serving. Every summary links to the original.
- 40 posts
- 15 companies
- 1 explainer
See the idea
Model Serving explainers
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Endpoint Candidates: evaluate and promote inference endpoints
Modal introduced Endpoint Candidates to stage alternative inference deployments for Dedicated Endpoints, allowing evaluation of quality, latency, throughput, and cost without affecting production traffic. Users create a Candidate by selecting a model from Modal’s catalog or importing custom weights, then set a percentage of production traffic to mirror. The Pingora-based proxy forks each request stream into separate bounded buffers per Candidate and shares body chunks across mirrors with a per-process memory budget; if a buffer fills or the budget is exceeded, the proxy cancels that mirrored request while production continues. Each Candidate receives its own URL for running existing evaluations on output quality and performance benchmarks under load, and all mirrored compute uses standard Modal compute pricing. When ready, users promote from the dashboard to roll out to the production endpoint, leaving other Candidates available for continued independent testing and enabling Auto Endpoints to validate automated inference optimizations.
Canary rollouts: upgrade models in production without downtime
Deploying a new model checkpoint in production typically requires a tradeoff: a hard swap exposes all users to regressions at once, while a DIY staged cutover relies on manual traffic shifting. Together AI’s rollout system automates this by moving traffic between source and target deployments through configurable canary steps, such as 5%, 25%, 50%, and 100%. Each step runs health checks before shifting traffic, enforces a propagation wait for routing caches, and drains the source only after traffic has moved. Optional metric gates, like p95 latency, run after each step; if a gate trips, the rollout pauses automatically and waits for human action. In a real canary from Qwen2.5-7B to Qwen3.5-9B, a 137% p95 regression was caught at 10% traffic, allowing reversal with no failed requests.
How a global fintech scaled coding agent traffic with Dedicated Model Inference
A global fintech runs its coding assistant on GLM 5.2 through Together’s Dedicated Model Inference (DMI), but spiky engineering-hours traffic outpaced static capacity planning. DMI gives the customer’s teams direct control over endpoint provisioning, scaling, configuration, model changes, and performance data through APIs, UI, and CLI. The workload is peak-load and relatively low-TPS, with ISL p50/p90/p95 of roughly 81K/163K/178K tokens and RPS p50/p90/p95 of 3/6/7, so concurrency headroom mattered more than raw throughput; the deployment used dozens of B200s, multiple replicas, and 256K context. When a reshaped endpoint later reached near-100% prefill capacity, requests queued for one to three minutes and decode throughput fell to roughly 5 tokens per second, the team restored cache-session-aware routing and widened max-inflight-per-worker, pushing the change same-day with zero downtime.
Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each
Dense models activate every parameter for every token, while Mixture-of-Experts (MoE) models route each token through a selected subset of feed-forward experts. This changes the relationship between capacity, memory, and compute: total parameters determine the memory required to load a model, whereas active parameters largely determine per-token work. At equal total size, MoE models often provide higher throughput, especially at batch size one, but routing, simultaneous expert residency, and reduced KV-cache room add serving complexity; latency advantages narrow at high concurrency. The comparison uses Nemotron 3.5 Lightning, a roughly 30B-parameter Mamba-2 plus MoE plus Attention hybrid with 3B active parameters, alongside dense and MoE examples, and concludes that dense models suit simpler serving and fine-tuning while MoE models suit throughput-oriented deployments with manageable memory and operational complexity.
From zero-shot forecast to purchase order with Amazon Bedrock AgentCore
This post presents an inventory automation architecture that moves from zero-shot demand forecasts to validated purchase orders while applying safety stock, minimum order quantities, budgets, and promotional adjustments. It combines Amazon Chronos2, hosted on SageMaker Serverless Inference, with four Claude-powered agents built using the Strands Agents SDK and deployed on Amazon Bedrock AgentCore; deterministic Python tools perform computation and agents handle judgment. Product data and future covariates arrive through CSV files in S3, while business rules remain in JSON, enabling what-if forecasts and onboarding without per-SKU training. In internal testing across 50 SKUs over a four-week horizon, the system reached 12.3% median WAPE, reduced onboarding from 2–3 weeks to under five minutes, and lowered estimated monthly inference cost from about $1,091 to $15. A conditional retry loop surfaces budget violations for user decisions instead of silently truncating orders, with traces and evaluator scores supporting auditability.
How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra
Production serving for agentic AI must maximize concurrent users on fixed GPU infrastructure while preserving latency, especially with long prompts, reused context, and streamed responses. The article presents NVIDIA NIM as a deployable microservice that packages validated model- and GPU-aware choices, then measures Nemotron 3 Ultra NIM 2.0.12 against an open-source baseline on four B200 GPUs using a 64K/400 workload, 76% KV reuse, and 50 TPS per user (20 ms ITL). The optimized stack combines autotuned MoE and Mamba kernels, tensor and expert-aware parallelism, prefix and Mamba state-cache reuse, scheduling, batching, memory tuning, and MTP speculative decoding. At the stated target, throughput rises from 718 to 1,997 output tokens per second—more than 2.5x—while the article recommends replaying representative traffic with NVIDIA AIPerf to select a Pareto point meeting the application’s latency constraint.
From token consumer to token provider: Building your org's AI API
Enterprise AI usage can become costly and fragmented when teams rely on separate provider accounts, personal keys, and independently operated inference stacks, leaving spending, workloads, and governance difficult to see. The post presents Model-as-a-Service (MaaS) in Red Hat OpenShift AI 3.4 as a shared service in which a platform team deploys approved models and exposes them through a standard, OpenAI-compatible API. Administrators can use the model catalog, distributed inference through llm-d on NVIDIA L40s, scoped API keys, rate limits, and dashboards for token use, latency, errors, GPU utilization, and queue length. Developers obtain self-service endpoints and can connect tools such as Claude Code, OpenCode, or Continue with existing clients. The architecture combines OpenShift, vLLM, an AI gateway, Prometheus, Grafana, and Jaeger to centralize inference, governance, and cost attribution across local and external model providers.
When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving
Encode-prefill-decode (EPD) disaggregation separates a multimodal model’s vision encoder from LLM prefill and decode, addressing contention when image or video processing delays language-model work. Using NVIDIA Dynamo, the post compares aggregated serving with colocated encoder workers and a heterogeneous, disaggregated encoder tier, with vision embeddings transferred to PD workers through NIXL over UCX RC/TCP Ethernet. EPD is most useful for media-heavy requests, short-to-medium outputs, smaller or quantized models, and mixed text/multimodal traffic; long decode sequences or light media processing reduce its value. In the reported image-heavy test, colocated and heterogeneous EPD reduced TTFT by 58% and 50%, while the heterogeneous tier served 70% more traffic at the same under-100-ms ITL SLO. Mixed-traffic measurements also reduced mean TTFT for text requests from 92.3 to 53.3 ms.
Real-Time Intelligence with IBM Time Series Models on Confluent
IBM and Confluent are making IBM Granite Time Series foundation models available in Early Access on Confluent Cloud, bringing forecasting and other time-series capabilities into streaming workflows. The announcement addresses the cost and delay of bespoke models by offering models that generalize across previously unseen series and can be used for forecasting, anomaly detection, similarity search, classification, gap-filling and optimization. Native inference runs directly in Apache Flink through Confluent’s AI_FORECAST and AI_DETECT_ANOMALIES functions, with Flink maintaining keyed, fault-tolerant state for each series and results written to Kafka topics. Confluent manages serving, scaling and runtime operations while retaining schemas, lineage, access controls and replayable data within the platform. Early Access starts on Confluent Cloud on AWS at no charge, with Confluent Platform planned next for on-premises and hybrid environments.
How to Size GPUs for AI Inference and TCO Without Overspending
GPU sizing for AI inference depends on use case, token patterns, latency targets, concurrency, cache hit rate, model choice and deployment strategy rather than raw hardware specifications alone. The post proposes a core-and-flex capacity model, combining baseline on-prem or reserved cloud capacity with elastic spot or on-demand GPUs for surges, launches and experiments. It also recommends matching GPU memory to prompt lengths and concurrency, while using quantization, pruning and distillation to reduce footprint and operating cost. A NeMo example prunes Qwen3-8B from 36 to 24 layers or narrows its dimensions toward a roughly 6B student before teacher-student distillation. In the illustrative run, width pruning reached lower final validation loss than depth pruning, 3.21 versus 3.60, while depth pruning converged faster.
Orchestrate production RAG with OpenShift AI
This guide shows how to move a streaming retrieval-augmented generation (RAG) workflow from a monolithic Ray Data script to modular AI pipelines on OpenShift AI. It addresses failure recovery and run traceability by separating parsing and chunking, ingestion, model download, and deployment into five reusable components. Parsed and chunked JSONL is stored in S3-compatible storage so KFP can rerun ingestion without repeating document processing, and each run records parameters, timestamps, logs, and status. The data chain uses Docling’s HybridChunker and Ray map_batches with ActorPoolStrategy, while the model chain downloads a Hugging Face model and deploys it through vLLM and KServe in parallel. For embedding ingestion, the workflow supports local CPU processing with granite-embedding-125m-english or a dedicated GPU vLLM service for larger collections and offers notebooks for deployment and query validation.
Run NVIDIA BioNeMo NIM Microservices for Protein Structure Prediction in Claude Science
Claude Science is used with the NVIDIA BioNeMo Agent Toolkit and NVIDIA NIM microservices to investigate how the predicted Seh1 structure changes alone versus with proposed partner C1HCX1. The workflow retrieves UniProt sequences, generates unpaired and species-paired multiple-sequence alignments with MSA Search, and sends those inputs to OpenFold3 and Boltz-2 for independent monomer and heteromer predictions. Using MSA input, heteromer iPTM reached 0.85 for OpenFold3 and 0.82 for Boltz-2, versus 0.14 and 0.19 without it; five-sample runs had standard deviations no greater than 0.006. Both models placed partner β-strands at Seh1’s WD40 propeller closure, while core Cα-RMSD remained 0.68 Å and 0.65 Å, respectively, but the predicted interaction remains experimentally unverified.
A/B test models in production
A/B testing model changes in production by sending fixed shares of live endpoint traffic to one control and up to 20 variants, allowing teams to compare user-facing outcomes rather than relying only on shadow-traffic checks. The endpoint router keeps variants out of the base traffic split, resamples requests that reach the control, and treats experiment percentages as absolute shares independent of replica counts or autoscaling. A 95/5 experiment can be updated to 80/20 and then 50/50 by resending the complete member list; an etag rejects stale concurrent updates. Measurement joins per-deployment platform metrics with product signals using the deployment ID in response metadata. To finish, a winning variant is promoted through a blue-green rollout before deleting the experiment, while deletion alone returns traffic to the control and can unwind a losing test.
Autoscaling endpoints for LLM inference
Dedicated LLM inference deployments face a trade-off between paying for idle GPUs and suffering sharp p95 latency when replicas hit batching limits. The autoscaling loop converts observed load into desired replicas with ceil(N × observed/target), then applies scale-up and scale-down windows, replica bounds, and GPU placement. It presents inflight_requests as a leading, engine-agnostic default, while TTFT and end-to-end latency target responsiveness but trail demand, and GPU or token utilization prioritize efficiency with less latency headroom. Cold starts can take several minutes because replicas require GPU placement, weight downloads, VRAM loading, and warm-up, so policies should scale early; streaming is required for TTFT, decoding speed, and throughput-per-replica metrics, whereas end-to-end latency supports both streaming and non-streaming workloads.
Configuring Dedicated Model Inference
Dedicated Model Inference is organized around three parts: an endpoint, deployments that run a model revision with replicas, and immutable configs that define how it runs. A deployment binds one model revision to one config and an autoscaling policy, while the endpoint provides a stable API-facing name and traffic split. Routing uses each deployment’s weight multiplied by its ready-replica count, so equal weights produce a 25%/75% split when deployments have one and three ready replicas; unavailable replicas contribute no capacity. The configuration workflow uses certified model-and-config profiles, selectable by accelerator, GPU count, optimization target, and topology, while new config revisions preserve tested rollback targets. This capacity-aware design makes scaling, rollouts, A/B tests, shadow traffic, and zero-downtime changes possible, but raw API or SDK deployments must be added to the traffic split before they receive requests.
ZeroEntropy is joining Notion
Notion announces it has acquired ZeroEntropy, whose team will join Notion. In less than two years, ZeroEntropy built high-efficiency training and inference for task-specific models using custom kernels, an inference stack, and model-specialization recipes. As a Notion customer, its reranker made unified search up to 30% faster, cut reranking latency by 85%, maintained constant answer quality, and lowered inference cost. ZeroEntropy has also trained zembed-1 and zerank-2 and published the zELO method; founder and CEO Ghita Houir Alami will lead Notion’s newly formed Model Research team. The acquisition supports Notion’s goal of delivering AI that is fast, affordable, and tailored to specific work for businesses of every size.
The production platform for open-weight AI inference
The update presents a production inference platform for open-weight, licensed closed-weight, and fine-tuned models, aimed at giving teams control over performance, cost, quality, and functionality without building an inference stack from scratch. It supports uploading weights or adapters from Hugging Face, S3, or local machines, uses tested deployment profiles for hardware, quantization, tensor parallelism, speculative decoding, and serving engine choices, and lets operators tune placement and autoscaling signals. Endpoints can keep a stable interface while multiple deployments receive traffic through canary, blue-green, rolling, A/B, or shadow testing, with threshold-based automatic rollback. The rebuilt caching and distribution layer proactively warms shared weights; internal testing reports roughly 4× faster warm starts across several frontier models, while a closed beta adds full-weight and LoRA reinforcement learning and supervised fine-tuning with deployable checkpoints.
Introducing Claude Haiku 4.5
Claude Haiku 4.5, the latest small model, is available to all users, offering a new option for near-frontier performance with lower cost and greater speed. Compared with Claude Sonnet 4, it delivers similar coding performance at one-third the cost and more than twice the speed, while surpassing Sonnet 4 on some computer-use tasks. Priced at $1/$5 per million input and output tokens, the model targets real-time assistants, customer service agents, pair programming, Claude Code workflows, and sub-agent orchestration, where Sonnet 4.5 can plan and delegate parallel subtasks to multiple Haiku 4.5 instances. Safety evaluations found low rates of concerning behavior and greater alignment than Haiku 3.5; the model is released under ASL-2 and is available through Claude apps and Code, the API, Amazon Bedrock, and Google Cloud’s Vertex AI.
What happens in the milliseconds after you tap pay
At checkout, fraud scoring must combine transaction inference with customer-specific rules while keeping latency low enough for an interactive payment. The retail-app sample pairs a FastAPI backend and React frontend in a Databricks App with Model Serving route optimization and Lakebase Postgres, where the model retrieves historical features and the backend reads profile controls. A transaction is scored first, then checked against daily spending, international-transaction, and country rules; pooled OAuth-authenticated connections and token-rotation handling avoid repeated handshakes and stale credentials. In the supplied benchmark, route-optimized calls reached 27.2 ms p50 and 37.3 ms p95 end-to-end, with feature lookup at 8.9 ms p50, CatBoost inference at 0.4 ms, and network overhead at 17.4 ms p50; actual performance varies by model.
Systems Engineering Playbook: Optimizing Qwen 3.5-397B MoE on Ironwood (TPU7x)
Serving Qwen3.5-397B, a 397-billion-parameter sparse Mixture-of-Experts model, on Ironwood (TPU v7x) requires fitting roughly 400 GB of weights into HBM while sustaining utilization across contrasting workloads. The report applies a modular, model-agnostic playbook built from reusable JAX/Pallas kernels and hardware-aware cost models, focusing new work on Gated DeltaNet linear attention and Attention Data Parallelism while integrating changes into vLLM and SGLang. The architecture activates exactly 17 billion parameters per token and combines 15 repeating blocks of three GDN layers and one GQA layer. Benchmarks use a single host with four Ironwood chips, eight logical devices, 8K/1K prefill-heavy and 1K/8K decode-heavy workloads, and concurrency tiers from 64 to 512. Between April and June 2026, stated optimizations improved inference performance by approximately 4.7x for 512-concurrency prefill-heavy workloads and 3.1x for decode-heavy workloads.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.