---
title: "GPU"
description: "107 posts about GPU, summarised, each linking to the original."
---

# GPU
> 107 posts about GPU, summarised, each linking to the original.

## Articles

### [Building Pinterest’s VLM Serving Stack on NVIDIA Dynamo](https://yomu.fyi/post/building-pinterest-s-vlm-serving-stack-on-nvidia-dynamo.md)
- Company: [Pinterest](https://yomu.fyi/company/pinterest.md)
- Author: Pinterest Engineering
- Published: Sep 10, 2026
- Type: Explainer

Pinterest describes building a VLM serving stack on NVIDIA Blackwell B200 GPUs and NVIDIA Dynamo for multimodal products including Pinterest Assistant, hybrid search, multimodal reranking, content understanding, signal generation, and content safety. Unlike text-only workloads, requests may contain thousands of images, costly and variable prefill, long multi-turn histories, and large KV caches, making preprocessing, routing, memory, and latency central serving concerns. The implementation combines Dynamo's distributed orchestration, multimodal frontend and vLLM support, KV-aware routing, disaggregated encoder/prefill/decode serving, and LMCache-based KV offloading with vLLM, an OpenAI Chat Completions interface, model-aware routing, EKS, and custom deployment components. Production-oriented AIPerf DAG benchmarks model QPS patterns, multi-turn conversations, many-image requests, and agentic DAGs; the team concludes that payload design, cache-aware routing, offload tiers, and a shared platform are essential to meeting product targets and accelerating new launches.


### [How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra](https://yomu.fyi/post/how-full-stack-nim-optimizations-deliver-2-5x-more-users-on-nemotron-3.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 10, 2026
- Type: Benchmark

Production serving for agentic AI must maximize concurrent users on fixed GPU infrastructure while preserving latency, especially with long prompts, reused context, and streamed responses. The article presents NVIDIA NIM as a deployable microservice that packages validated model- and GPU-aware choices, then measures Nemotron 3 Ultra NIM 2.0.12 against an open-source baseline on four B200 GPUs using a 64K/400 workload, 76% KV reuse, and 50 TPS per user (20 ms ITL). The optimized stack combines autotuned MoE and Mamba kernels, tensor and expert-aware parallelism, prefix and Mamba state-cache reuse, scheduling, batching, memory tuning, and MTP speculative decoding. At the stated target, throughput rises from 718 to 1,997 output tokens per second—more than 2.5x—while the article recommends replaying representative traffic with NVIDIA AIPerf to select a Pareto point meeting the application’s latency constraint.


### [Introducing preemptible compute: the same compute, half the price](https://yomu.fyi/post/introducing-preemptible-compute-the-same-compute-half-the-price.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Sep 10, 2026
- Type: Announcement

Together GPU Clusters is introducing preemptible compute in public preview for Kubernetes clusters in all regions, offering interruptible GPU capacity at a flat 50% of the on-demand rate. Preemptible nodes use the same NVIDIA accelerated compute, can be reclaimed when unused capacity is needed, and receive a maximum five-minute drain window for checkpointing and graceful exit. Clusters retain the requested preemptible target and automatically refill as capacity returns, while usage is metered every one to two minutes and billed sub-hourly. The feature suits resumable, retryable, or requeueable work such as experiments, batch inference, and temporary bursts, but critical services, uncheckpointed multi-day runs, and strict-SLO serving should remain on standard nodes; users must schedule eligible pods with the together.ai/compute-class=preemptible label.


### [To Infinity and Beyond: ThunderKittens Now on NVIDIA Vera Rubin NVL72!](https://yomu.fyi/post/to-infinity-and-beyond-thunderkittens-now-on-nvidia-vera-rubin-nvl72.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Sep 10, 2026
- Type: Explainer

Together’s kernels team ported ThunderKittens to NVIDIA Vera Rubin NVL72 and investigated why existing Blackwell GEMMs underuse the platform. The work targets NVFP4 and FP8 GEMMs: although Rubin preserves Blackwell’s programming model, the original kernels reach only about 42.1% and 44.4% of roofline because tensor cores consume operands twice as fast as the old pipeline supplies them. The team integrates Rubin’s 64-byte K step, exclusive tensor memory extending to 576 columns, oversized shared memory up to 328 KiB, B-side collector buffers, and earlier A-release signaling into its NVFP4 kernel. The resulting kernel exceeds 22 PFLOPS and is described as competitive with cuBLAS and CuTe DSL, while the post also compares Rubin’s specifications with NVIDIA HGX B200.


### [Anthropic expands Google and Broadcom compute deal](https://yomu.fyi/post/anthropic-expands-google-and-broadcom-compute-deal.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: Sep 9, 2026
- Type: Announcement

Anthropic announced a new agreement with Google and Broadcom for multiple gigawatts of next-generation TPU capacity expected to come online starting in 2027. The company says the expansion will support its frontier Claude models and meet customer demand, with most of the new compute sited in the United States as an expansion of its November 2025 commitment to invest $50 billion in American computing infrastructure. Anthropic reports that run-rate revenue surpassed $30 billion, up from approximately $9 billion at the end of 2025, while the number of business customers spending more than $1 million annually rose from over 500 to more than 1,000 in less than two months. It will continue using AWS Trainium, Google TPUs, and NVIDIA GPUs, and says Claude remains available across Amazon Web Services, Google Cloud, and Microsoft Azure.


### [When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving](https://yomu.fyi/post/when-to-use-encode-prefill-decode-disaggregation-to-accelerate-multimo.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Sep 9, 2026
- Type: Explainer

Encode-prefill-decode (EPD) disaggregation separates a multimodal model’s vision encoder from LLM prefill and decode, addressing contention when image or video processing delays language-model work. Using NVIDIA Dynamo, the post compares aggregated serving with colocated encoder workers and a heterogeneous, disaggregated encoder tier, with vision embeddings transferred to PD workers through NIXL over UCX RC/TCP Ethernet. EPD is most useful for media-heavy requests, short-to-medium outputs, smaller or quantized models, and mixed text/multimodal traffic; long decode sequences or light media processing reduce its value. In the reported image-heavy test, colocated and heterogeneous EPD reduced TTFT by 58% and 50%, while the heterogeneous tier served 70% more traffic at the same under-100-ms ITL SLO. Mixed-traffic measurements also reduced mean TTFT for text requests from 92.3 to 53.3 ms.


### [Higher usage limits and a SpaceX compute deal](https://yomu.fyi/post/higher-usage-limits-and-a-spacex-compute-deal.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: Sep 9, 2026
- Type: Announcement

An agreement with SpaceX will give the company access to all compute capacity at the Colossus 1 data center, adding more than 300 megawatts and over 220,000 NVIDIA GPUs within the month. Combined with other compute partnerships, the expansion supports higher usage limits: Claude Code five-hour limits double for Pro, Max, Team, and seat-based Enterprise plans, peak-hour reductions end for Pro and Max, and Claude Opus API limits rise considerably. The company says the new capacity will directly improve service for Claude Pro and Claude Max subscribers while it trains and runs Claude on AWS Trainium, Google TPUs, and NVIDIA GPUs. It is also expanding inference capacity in Asia and Europe and exploring orbital AI compute with SpaceX.


### [Personalize your product's text-to-speech voice for any language: Fine-tuning with Kubeflow Trainer on Red Hat OpenShift AI](https://yomu.fyi/post/personalize-your-product-s-text-to-speech-voice-for-any-language-fine.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Dmytro Hryshchenko, Abhijeet Dhumal
- Published: Sep 9, 2026
- Type: Tutorial

Training an English-centered text-to-speech model for Turkish can produce fluent but hard-to-understand pronunciation for native speakers. This tutorial fine-tunes unsloth/orpheus-3b-0.1-pretrained with about 81,000 Turkish text-audio pairs on Red Hat OpenShift AI using a Kubeflow Trainer TrainJob distributed across two nodes with two A100 GPUs each. LoRA freezes the 3-billion-parameter base and trains 97.3M adapter parameters, while preprocessing converts text to Llama-3 tokenizer tokens and audio to three-layer SNAC codes, masking text labels during scoring. The merged model is evaluated against the base on ten Turkish sentences using Whisper transcription and WER/CER metrics. Mean WER falls from 2.44 to 0.35 and CER from 1.59 to 0.09, though the authors describe the small test as directional.


### [Introducing CUDA Rust: Two Tracks for Writing GPU Kernels](https://yomu.fyi/post/introducing-cuda-rust-two-tracks-for-writing-gpu-kernels.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 8, 2026
- Type: Announcement

NVIDIA’s CUDA Rust effort targets the GPU-kernel gap in Rust-based systems, where host code can be Rust but kernels often require another language. It presents two tracks: cuda-oxide compiles #\[kernel\] functions through Rust MIR, Pliron, and LLVM IR to PTX for SIMT programming, while cutile-rs expresses computations as tiles and JIT-compiles embedded ASTs through CUDA Tile IR. The SIMT example uses DisjointSlice, typed thread indices, and launch contracts to constrain per-thread output access and validate launch configuration; Tile leaves thread mapping and shared-memory management to its compiler. The post recommends starting with Tile because the compiler decides tile mapping per architecture, and using SIMT when explicit thread or memory control is needed. Both projects are early-stage, with cuda-oxide in early alpha and cutile-rs further along but still incomplete and subject to API changes.


### [Understanding W8A8 INT8 LLM quantization: Half the size, better performance, same accuracy](https://yomu.fyi/post/understanding-w8a8-int8-llm-quantization-half-the-size-better-performa.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Sana Fayyaz
- Published: Sep 7, 2026
- Type: Tutorial

Serving Llama 3.1 8B in BF16 requires roughly 14.9 GB for weights, leaving little VRAM on a 16 GB GPU for KV cache and activations, which limits practical deployment. The workflow benchmarks the base model, applies INT8 W8A8 quantization with llm-compressor, then repeats accuracy and performance tests under identical hardware and load conditions. SmoothQuant modifies weights to reduce activation outliers, while GPTQ uses calibration data and error compensation before rounding weights to INT8; the experiment used 512 WikiText-2 samples and targeted deployment with vLLM. On the Llama 3.1 8B Instruct model, compression reduced storage from 14.9 GB to 8.0 GB, a 46% reduction, and accuracy was preserved across MMLU, ARC, HellaSwag, and IFeval; generation-speed results are reserved for part 2.


### [Achieving Extreme Efficiency through Specialized GPU Kernel Generation](https://yomu.fyi/post/achieving-extreme-efficiency-through-specialized-gpu-kernel-generation.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Daya Khudia, Leo Li, Lesheng Jin
- Published: Sep 4, 2026
- Type: Explainer

Production inference systems often use generic GPU kernels even though operation shapes combine static model parameters with request-time dimensions such as token counts. Proteus addresses this mismatch by having agents propose specialized kernels, then applying controlled-reference validation, static checks, isolated repeated timing, hidden tests, and consistency checks before retaining measured winners. Its knowledge layer uses actionable, scoped lessons retrieved through hierarchical tags and hybrid keyword-semantic search, while background jobs handle deeper distillation instead of burdening each attempt. On NVIDIA B200 GPUs with a Triton backend, a Gated DeltaNet packed-decode case for Qwen 3.5 122B produced shape-specific wins, including 1.6× for a Batch=4, Key=128, Value=128 path and 0.018 ms latency for Candidate 030. Across the reported kernels, Proteus achieved 1.8–5.2× speedups over the best available in vLLM, with validation and context management identified as the main difficulties.


### [Speeding up LLM inference with P-EAGLE in vLLM Speculators](https://yomu.fyi/post/speeding-up-llm-inference-with-p-eagle-in-vllm-speculators.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Helen Zhao, Megan Flynn, Dipika Sikka
- Published: Sep 3, 2026
- Type: Explainer

P-EAGLE (Parallel EAGLE), implemented in Speculators v0.6.0, extends EAGLE-3 with parallel drafting so a draft model predicts multiple future tokens in one forward pass rather than sequentially. Speculative decoding pairs this small draft model with a verifier model, which checks the proposed sequence in parallel and accepts only its correct prefix while preserving the verifier’s output distribution. The post adds Conditional Drop-token (COD) sampling, learnable mask embeddings and hidden states, and a custom flex-attention mask to support multi-depth prediction within the EAGLE-3 framework. The post presents an end-to-end workflow covering data preparation, vLLM hidden-state extraction, training and deployment, including a Qwen3-8B example that takes about 50 minutes on four H100s. A published Qwen3-a8B draft model reports an average length of 3.044 across the listed datasets.


### [The Modern CUDA Toolbox in Practice: A Step-by-Step Optimization Walkthrough](https://yomu.fyi/post/the-modern-cuda-toolbox-in-practice-a-step-by-step-optimization-walkth.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 2, 2026
- Type: Tutorial

An image-processing pipeline serves as the test case for a modern CUDA toolbox, starting with RGB-to-grayscale conversion, 32-by-32 tile median computation, and host/device transfers. It advances through six incremental changes: adopting CCCL and Compute Sanitizer for indexing bugs, adding NVTX for Nsight Systems benchmarks, replacing hand-rolled work with CUB algorithms, using pooled device containers and pinned host containers, and assigning each image its own stream with asynchronous copies. The implementation uses cuda::stream, cuda::copy\_bytes, cuda::device\_buffer, and CUB calls on per-thread streams, allowing kernels and transfers to overlap instead of executing sequentially on the default stream. The reported total duration falls from 6.8 seconds to about 23 milliseconds for three images, a roughly 300x improvement, while the post characterizes the code as safer and easier to maintain without low-level optimizations.


### [Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference](https://yomu.fyi/post/co-designing-ai-models-using-speculative-decoding-for-faster-llm-infer.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Published: Sep 2, 2026
- Type: Explainer

Speculative decoding accelerates large language model inference by having a smaller draft model propose multiple tokens and a target model verify them in parallel, preserving the standard output sequence under its acceptance criteria. It reduces decoding iterations and increases target-model arithmetic intensity, but speedup depends on acceptance length, verification time, draft latency, batch size, and hardware behavior. For attention-heavy workloads, the recommended starting point is D = 128/G - 1, where G is query heads per KV head; larger D values should align G × (1 + D) with 128-token kernel tiles. The remaining guidance covers compute-bound GEMMs, low-latency draft-cost tradeoffs, and choosing among external models, EAGLE-3, MTP, DFlash, DSpark, and suffix or n-gram methods; SPEED-Bench, TensorRT LLM, and NVIDIA/Model-Optimizer support measurement, training, and quantization workflows.


### [LLM quantization guide: How to do it, and how it helps](https://yomu.fyi/post/llm-quantization-guide-how-to-do-it-and-how-it-helps.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Cedric Clyburn
- Published: Sep 2, 2026
- Type: Tutorial

LLM quantization addresses the gap between rapidly growing model sizes and GPU memory by storing weights and activations at lower precision than BF16. It explains how quantization targets linear layers in transformer blocks, distinguishes weight quantization for memory-bound decoding from activation quantization for compute-bound prefill, and maps formats such as GGUF, FP8, NVFP4, AWQ-INT4, and MLX to their inference engines. Calibrated methods including GPTQ, AWQ, and SmoothQuant protect important weights using representative data, while reported NVFP4 results recover about 99% of BF16 accuracy for 70B–235B models and 95–98% for 7B–14B models. In a Llama 3 70B benchmark on two H100s, FP8 weights-and-activations increased throughput from 158 to 474 input tokens/sec and reduced time to first token under load from over 30,000ms to about 4,800ms.


### [How Botika runs full-stack generative AI on Modal](https://yomu.fyi/post/how-botika-runs-full-stack-generative-ai-on-modal.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Sep 2, 2026
- Type: Problem & solution

Botika builds agentic e-commerce teams that automate visual production for global fashion brands, using proprietary foundation models and a 100-terabyte image dataset. The company runs its entire full-stack AI operation on Modal, with the data pipeline processing 100 terabytes at less than 1% error rate across thousands of containers. Before Modal, the team struggled with Kubernetes fleet management and slow GCP Batch jobs; now a single researcher can launch 50 experiments daily using Claude, and they set up reinforcement learning infrastructure in less than a day. Modal handles autoscaling, cold starts, and multi-node training with RDMA, allowing Botika to double traffic in seconds, and they serve about 15 models on L4s, L40S, A100s, and H100s. The infrastructure is not Modal-specific, enabling the team to run open-source and custom frameworks without modification.


### [How to Size GPUs for AI Inference and TCO Without Overspending](https://yomu.fyi/post/how-to-size-gpus-for-ai-inference-and-tco-without-overspending.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 1, 2026
- Type: Tutorial

GPU sizing for AI inference depends on use case, token patterns, latency targets, concurrency, cache hit rate, model choice and deployment strategy rather than raw hardware specifications alone. The post proposes a core-and-flex capacity model, combining baseline on-prem or reserved cloud capacity with elastic spot or on-demand GPUs for surges, launches and experiments. It also recommends matching GPU memory to prompt lengths and concurrency, while using quantization, pruning and distillation to reduce footprint and operating cost. A NeMo example prunes Qwen3-8B from 36 to 24 layers or narrows its dimensions toward a roughly 6B student before teacher-student distillation. In the illustrative run, width pruning reached lower final validation loss than depth pruning, 3.21 versus 3.60, while depth pruning converged faster.


### [Run NVIDIA BioNeMo NIM Microservices for Protein Structure Prediction in Claude Science](https://yomu.fyi/post/run-nvidia-bionemo-nim-microservices-for-protein-structure-prediction.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Aug 31, 2026
- Type: Tutorial

Claude Science is used with the NVIDIA BioNeMo Agent Toolkit and NVIDIA NIM microservices to investigate how the predicted Seh1 structure changes alone versus with proposed partner C1HCX1. The workflow retrieves UniProt sequences, generates unpaired and species-paired multiple-sequence alignments with MSA Search, and sends those inputs to OpenFold3 and Boltz-2 for independent monomer and heteromer predictions. Using MSA input, heteromer iPTM reached 0.85 for OpenFold3 and 0.82 for Boltz-2, versus 0.14 and 0.19 without it; five-sample runs had standard deviations no greater than 0.006. Both models placed partner β-strands at Seh1’s WD40 propeller closure, while core Cα-RMSD remained 0.68 Å and 0.65 Å, respectively, but the predicted interaction remains experimentally unverified.


### [Fine-Tuning SOTA Object Detection Models on Real-World Datasets](https://yomu.fyi/post/fine-tuning-sota-object-detection-models-on-real-world-datasets.md)
- Company: [Jetbrains](https://yomu.fyi/company/jetbrains.md)
- Author: Arina Belova
- Published: Aug 31, 2026
- Type: Tutorial

This practical comparison examines how YOLO12, YOLO26, and RF-DETR behave when moved from COCO to specialized real-world object-detection datasets, and how fine-tuning changes the outcome. The authors first evaluate six pretrained checkpoints on COCO val2017, then use cable-damage, bone-fracture, and soda-bottles datasets from RF100-VL for zero-shot testing, fine-tuning, and accuracy/latency comparison. Because the model implementations require incompatible dependencies, the workflow uses one PyCharm project with three isolated uv environments and remote GPU execution; the reported latency stays in native frameworks rather than TensorRT. All models perform well on COCO, but direct performance on specialized data falls close to zero, while 10 fine-tuning epochs adapt the detectors well to cable damage and soda bottles. Bone-fracture remains difficult despite fine-tuning, with RF-DETR performing best, indicating that domain shift may require more data, longer training, or domain-specific pretraining.


### [Stop wasting GPU allocation in Kubernetes with GPU-pruner](https://yomu.fyi/post/stop-wasting-gpu-allocation-in-kubernetes-with-gpu-pruner.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Fahim Uddin
- Published: Aug 31, 2026
- Type: Problem & solution

GPU-pruner addresses idle GPU allocation in Kubernetes by identifying workloads whose hardware activity remains below a configured threshold, rather than relying on browser sessions, network traffic, or pod lifetime. It queries Prometheus for NVIDIA Data Center GPU Manager metrics, including engine activity and GPU utilization, and treats peak performance below 1% throughout a default 35-minute observation window as idle. After tracing an idle pod through ownerReferences or KServe labels, it scales the parent resource to zero, preserving workload metadata for later restoration. Slack notifications, grace periods, lookback checks, and dry-run mode provide safeguards, while Prometheus label handling must preserve the workload namespace and pod identity in DCGM telemetry.


[Newer posts](https://yomu.fyi/topic/gpu/page/2.md) · [Older posts](https://yomu.fyi/topic/gpu/page/4.md)