---
title: "Performance"
description: "817 posts about Performance, summarised, each linking to the original."
---

# Performance
> 817 posts about Performance, summarised, each linking to the original.

## Articles

### [Training Design for Text-to-Image Models: Lessons from Ablations](https://yomu.fyi/post/training-design-for-text-to-image-models-lessons-from-ablations.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: David Bertoin, Roman Frigg, Jon Almazán
- Published: Feb 3, 2026
- Type: Benchmark

The second part of a series on training efficient text-to-image models from scratch documents training-design ablations for the PRX model. The authors establish a Flow Matching baseline (PRX-1.2B, 100K steps, 256×256, AdamW, GemmaT5, RoPE, 1M MidJourneyV6 synthetic images) and benchmark interventions against FID, CMMD, DINO-MMD, and throughput across four buckets: Representation Alignment, Training Objectives, Token Routing and Sparsification, and Data. REPA representation alignment boosts early convergence when used as a burn-in, while the Muon optimizer lowered FID from 18.20 to 15.55 and improved CMMD and DINO-MMD. Storing denoiser weights in BF16 instead of using BF16 autocast raised FID to 21.87, and token routing methods such as TREAD/SPRINT were minor at 256² but became major wins at higher resolutions. Long captions proved critical, synthetic versus real data shifted texture versus structure, and upcoming work releases the full training code alongside a 24-hour end-to-end speedrun.


### [Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective](https://yomu.fyi/post/unlocking-agentic-rl-training-for-gpt-oss-a-practical-retrospective.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Jason Zhu, Hejian Sang, Arup De, Rohit Jain, Yanning Chen
- Published: Jan 27, 2026
- Type: Problem & solution

Engineers evaluated the GPT-OSS model for agentic reinforcement learning across multi-step tasks such as ReTool and GSM8K using the verl framework. Initial training runs suffered from exploding gradient norms, increasing KL divergence, and stagnating rewards. Investigation revealed that non-deterministic expert routing in the Mixture of Experts architecture caused log-probability mismatches between forward passes, which falsely triggered Proximal Policy Optimization clipping. The team resolved this by substituting detached current log-probabilities for old log-probabilities during on-policy updates. To address further training instabilities and memory bottlenecks, the implementation integrated an attention-sink-aware backward pass into FlashAttention v3 and applied sequence parallelism using all-to-all communication around attention layers.


### [SkyPilot at Shopify: Multi-cloud GPUs without the pain](https://yomu.fyi/post/skypilot-at-shopify-multi-cloud-gpus-without-the-pain.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 26, 2026
- Type: Explainer

Shopify uses SkyPilot to run machine-learning training workloads across GPU clusters on Nebius and GCP, addressing scarce hardware, fragmented capacity, and provider-specific interfaces. Persistent Kubernetes clusters remain managed by Shopify, while SkyPilot launches jobs and a company plugin routes requests, validates labels, injects configuration, and applies organizational policies. H200 workloads go to Nebius for InfiniBand-equipped distributed training, while L4 and CPU-only jobs go to GCP; labels also connect jobs to cost ownership, Kueue quotas, priority classes, and development-environment rules. The plugin automatically configures InfiniBand access, shared caches, and temporary storage, reducing manual setup and speeding repeated workloads. The resulting YAML-based interface preserves direct resource control while hiding cloud-selection decisions, giving teams common workflows as providers, prices, and GPU availability change.


### [Docker lazy loading at Grab: Accelerating container startup times](https://yomu.fyi/post/docker-lazy-loading-at-grab-accelerating-container-startup-times.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Huong Vuong
- Published: Jan 21, 2026
- Type: Problem & solution

Large Docker images for Airflow and Spark Connect caused minute-scale downloads, delaying cold starts and weakening autoscaling responsiveness. The post describes testing eStargz and Seekable OCI (SOCI) with containerd remote snapshotters, which expose registry-backed layers through a virtual lowerdir and fetch file content on demand. In production, SOCI delivered 30–40% faster startup times for Airflow and Spark Connect, while benchmark results showed SOCI matching standard images’ 5.0-second Airflow application startup; eStargz took 25.0 seconds. Configuration tuning increased download concurrency, unpack concurrency, and chunk size, reducing fresh-node image download time from 60 seconds to 24, a 60% improvement. Grab chose SOCI because Bottlerocket supports it natively, while noting that lazy loading redistributes rather than removes download work and requires workload-specific tuning.


### [Differential Transformer V2](https://yomu.fyi/post/differential-transformer-v2.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Li Dong
- Published: Jan 20, 2026
- Type: Explainer

Differential Transformer V2 introduces an architectural refinement to differential attention that pairs and subtracts query heads within the same grouped-query attention group. The architecture doubles query heads while preserving key-value head counts, which avoids increasing key-value cache demands and removes the need for custom kernels. DIFF V2 replaces the per-head RMSNorm of its predecessor with a per-token, per-head projected lambda passed through a sigmoid function. This formulation expands the context root-mean-square range to between zero and root two, preventing massive gradient magnifications, mitigating attention sinks, and stabilizing large learning rate pretraining. Ablations confirm that sharing keys and values across subtracted heads and constraining lambda scaling are necessary to prevent training instability and elevated loss.


### [Introducing Waypoint-1: Real-time interactive video diffusion from Overworld](https://yomu.fyi/post/introducing-waypoint-1-real-time-interactive-video-diffusion-from-over.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Andrew Lapp, Louis Castricato, Scott Fox, Shahbuland Matiana, David Rossi
- Published: Jan 20, 2026
- Type: Announcement

Overworld introduced Waypoint-1, a real-time interactive video diffusion model controllable through text prompts, mouse movements, and keyboard inputs. The architecture uses a frame-causal rectified flow transformer trained on compressed latent representations across 10,000 hours of video game footage paired with control signals. Pre-training uses diffusion forcing with causal attention, while post-training incorporates self forcing via DMD to mitigate autoregressive rollout error accumulation, enable one-pass classifier-free guidance, and support few-step denoising. To achieve low latency on consumer hardware, Overworld developed WorldEngine, a pure Python streaming inference library. Utilizing AdaLN feature caching, static rolling key-value caching with Flex Attention, fused matrix multiplications, and PyTorch compilation, the 2.3B parameter Waypoint-1-Small model reaches up to 60 frames per second on an RTX 5090 GPU.


### [Lyft’s Feature Store: Architecture, Optimization, and Evolution](https://yomu.fyi/post/lyft-s-feature-store-architecture-optimization-and-evolution.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Rohan Varshney
- Published: Jan 6, 2026
- Type: Explainer

Lyft built a centralized Feature Store to optimize feature management, deployment, and data consistency for machine learning workflows across its rideshare platform. The system ingests batch features defined by Spark SQL and JSON metadata into Airflow DAGs, which simultaneously write data to Hive for model training and to an online serving layer for low-latency inference. Real-time streaming features are processed with Apache Flink and Apache Kafka before being ingested into the serving infrastructure. The online layer, dsfeatures, couples DynamoDB with a ValKey write-through LRU cache and OpenSearch to serve embeddings. Infrastructure optimizations reduced standard P95 read latency by a third while supporting over a trillion additional read and write operations.


### [NVIDIA brings agents to life with DGX Spark and Reachy Mini](https://yomu.fyi/post/nvidia-brings-agents-to-life-with-dgx-spark-and-reachy-mini.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Jeff Boudier, Nader Khalil, Alec Fong
- Published: Jan 5, 2026
- Type: Tutorial

At CES 2026, NVIDIA presents a desk-based personal assistant that combines DGX Spark processing with a Reachy Mini robot, allowing an agent to converse, perceive camera input, speak, and perform actions. The tutorial assembles NVIDIA Nemotron 3 Nano for reasoning, NVIDIA Nemotron Nano 2 VL for vision, ElevenLabs for text-to-speech, and Reachy Mini hardware or simulation through the NeMo Agent Toolkit and Pipecat. Its configuration routes text, visual, and action requests to different models or a ReAct tool-calling agent, while the robot daemon exposes movement and state capabilities. The guide covers local, cloud, and serverless deployment, then starts the services needed for voice and vision interaction; the completed setup greets users and supports example text and camera prompts.


### [Token-count-based Batching: Faster, Cheaper Embedding Inference for Queries](https://yomu.fyi/post/token-count-based-batching-faster-cheaper-embedding-inference-for-quer.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Chengcheng Pei, Yuan Lin
- Published: Dec 18, 2025
- Type: Problem & solution

Serving embedding models for short search queries often suffers from poor GPU efficiency because traffic is spiky and memory-bound. Traditional time-window and request-count batching strategies lead to inconsistent GPU utilization, while tensor padding wastes compute on empty tokens. To resolve these bottlenecks, Voyage AI implemented token-count-based batching paired with padding removal in inference engines like vLLM. The architecture uses Redis with Lua scripts to atomically aggregate pending requests until reaching an optimal hardware saturation token threshold. Across production deployments, this approach achieved a 50% reduction in GPU inference latency with 3X fewer GPUs and improved throughput by up to 8×.


### [From Python3.8 to Python3.10: Our Journey Through a Memory Leak](https://yomu.fyi/post/from-python3-8-to-python3-10-our-journey-through-a-memory-leak.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Jay Patel
- Published: Dec 15, 2025
- Type: Incident / postmortem

During an initiative to upgrade Python services from version 3.8 to 3.10, Lyft engineers encountered severe latency spikes and timeouts in one service within their test environment. Profiling DynamoDB queries revealed that gevent thread joins were taking up to 30 seconds while pod memory consumption climbed steadily. To isolate the issue, engineers used an internal tracemalloc-based profiler triggered via SIGUSR2 signals, temporarily disabling Gunicorn preload to prevent workers from terminating prematurely on signal receipt. Memory traces pointed to botocore and an incompatibility between weakref.finalize and gevent monkey patching in urllib3 version 1.26.16, which prevented connections from returning to the pool. Downgrading urllib3 to version 1.26.15 immediately resolved both the timeouts and the memory leak before a permanent fix arrived in gevent and urllib3 updates.


### [CUGA on Hugging Face: Democratizing Configurable AI Agents](https://yomu.fyi/post/cuga-on-hugging-face-democratizing-configurable-ai-agents.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Jim Laredo, Avi Yaeli, Sami Marreed, Ayhan Sebin, Merve Unuvar
- Published: Dec 15, 2025
- Type: Announcement

Many existing AI agent frameworks suffer from brittleness, tool misuse, and failures when executing complex workflows. To address these limitations, the open-source Configurable Generalist Agent (CUGA) introduces structured orchestration that decomposes user goals into programmatic subtasks tracked by a dynamic task ledger. The framework integrates agentic patterns like planner-executor and code-act, delegating subtasks to specialized agents that generate pseudo-code before running execution in a secure sandbox. Released under the Apache 2.0 license, CUGA integrates with Langflow for low-code visual workflow assembly and supports multi-tool environments through OpenAPI specs, MCP servers, and LangChain. Testing on inference platforms like Groq with open models such as gpt-oss-120b demonstrates rapid response times during multi-step planning and validation.


### [Tangle: An open-source ML experimentation platform built for scale (2025) - Shopify](https://yomu.fyi/post/tangle-an-open-source-ml-experimentation-platform-built-for-scale-2025.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 5, 2025
- Type: Announcement

Shopify introduces Tangle, an open-source, platform-agnostic machine-learning experimentation platform built for Search and Discovery workflows at commerce scale. It targets six stated failure modes, including repeated data preparation, poor reproducibility, slow deployment, unstructured notebooks, manual query tracking, and limited sharing. Users compose visual directed acyclic graph pipelines from reusable YAML components that wrap containerized or inline command-line programs in multiple languages, while file-based execution moves artifacts through storage. Tangle’s content-based caching reuses results when output content hashes remain identical, including artifacts from still-running executions, rather than relying only on lineage changes. Shopify reports a ten-hour pipeline completing in twenty minutes after one component changes, more than a year of compute time savings, and production use across ranking, semantic search, recommendation, and feature engineering workloads.


### [Behind the build: 2025's BFCM globe in a pinball machine 🌎🕹️](https://yomu.fyi/post/behind-the-build-2025-s-bfcm-globe-in-a-pinball-machine.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 3, 2025
- Type: Explainer

The 2025 BFCM live globe was implemented inside a browser-based pinball machine to represent real-time merchant sales and celebrate milestones. Built with three.js and react-three-fiber, the experience runs at 120fps, uses Server-Sent Events for order arcs and live metrics, includes a 128x32 shader-driven dot-matrix display, and supports AR through react-three/xr. For performance, the team simulated pinball physics in 2D with Rapier2D and used raycasts for ramps and elevation, reducing the physics step from 2ms to 0.2ms. The scene also includes interactive mini games, infrastructure statistics, Easter eggs, and milestone-triggered reactions, while Apache FlinkSQL pipelines process event streams behind the commerce visualizations. The globe was also displayed on the Exosphere in Las Vegas for the third consecutive year.


### [Continuous batching from first principles](https://yomu.fyi/post/continuous-batching-from-first-principles.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Rémi Ouazan Reboul, Arthur Zucker, Luc Georges
- Published: Nov 25, 2025
- Type: Explainer

Large language model serving requires running expensive next-token generation across multiple concurrent user requests. Traditional batching approaches introduce severe padding inefficiencies when mixing variable-length prompts and different generation phases, especially under static shape constraints like CUDA graphs. Continuous batching resolves these inefficiencies by combining key-value caching, chunked prefill, ragged batching, and dynamic request scheduling. Ragged batching eliminates the traditional batch axis by concatenating token sequences into a single tensor and using boolean attention masks to isolate independent sequences. By dynamically removing completed prompts and packing decoding tokens alongside chunked prefill tokens up to a hardware memory budget, serving systems maintain high hardware utilization and throughput.


### [Building Deep Research: How we Achieved State of the Art](https://yomu.fyi/post/building-deep-research-how-we-achieved-state-of-the-art.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Michael Griff, Dean Sacoransky, Noah Nefsky
- Published: Nov 24, 2025
- Type: Problem & solution

Building production AI research agents presents challenges around context window pollution, escalating token costs, and architectural brittleness across model updates. To resolve these issues, Tavily rebuilt its deep research system around simplified orchestration, compact tooling, and active context curation. The agent mimics human research workflows by distilling tool outputs into concise reflections for ongoing reasoning, withholding raw retrieved web data until the final deliverable stage. Compared to traditional ReAct propagation architectures where token consumption scales quadratically, this reflection-based approach achieves linear token growth. As a result, the system reduced token consumption by 66% compared to Open Deep Research while reaching state-of-the-art performance on DeepResearch Bench.


### [Unlocking Peak Performance on Qualcomm NPU with LiteRT](https://yomu.fyi/post/unlocking-peak-performance-on-qualcomm-npu-with-litert.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Lu Wang, Weiyi Wang, Andrew Zhang
- Published: Nov 24, 2025
- Type: Announcement

Google introduces the LiteRT Qualcomm AI Engine Direct (QNN) Accelerator, developed with Qualcomm to replace the previous TFLite QNN delegate and simplify NPU deployment across supported Qualcomm SoCs. It integrates vendor compilers and runtimes behind a unified workflow, supports ahead-of-time and on-device compilation, and can fall back to GPU or CPU when needed. Across 72 canonical vision, audio, and NLP models, the accelerator supports 90 LiteRT ops, enables full NPU delegation for 64, and reports up to 100x CPU and 10x GPU speedups. On Snapdragon 8 Elite Gen 5, FastVLM-0.5B uses int8 weights, int16 activations, and specialized transformer kernels to reach 0.12 second time-to-first-token, over 11,000 prefill tokens per second, and over 100 decode tokens per second.


### [20x Faster TRL Fine-tuning with RapidFire AI](https://yomu.fyi/post/20x-faster-trl-fine-tuning-with-rapidfire-ai.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Kamran Bigdely, Arun Kumar, Quentin Gallouédec
- Published: Nov 21, 2025
- Type: Announcement

Hugging Face TRL now officially integrates with RapidFire AI to accelerate LLM fine-tuning and post-training experimentation through concurrent configuration evaluation. Instead of executing training configurations sequentially, the platform shards datasets into discrete chunks and cycles multiple model configurations across GPUs at chunk boundaries. The runtime handles automatic checkpointing via a shared-memory-based adapter and model spilling mechanism to maintain consistent training state. Experimenters can monitor live metrics on an MLflow-based dashboard and use Interactive Control Ops to stop underperforming runs or clone and warm-start promising configurations mid-flight. Benchmark results on NVIDIA A100 GPUs demonstrate experimentation speedups ranging from 15x to 20x over sequential runs when training TinyLlama-1.1B and Llama-3.2-1B models.


### [Open ASR Leaderboard: Trends and Insights with New Multilingual & Long-Form Tracks](https://yomu.fyi/post/open-asr-leaderboard-trends-and-insights-with-new-multilingual-long-fo.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Eric Bezzam, Steven Zheng, Eustache Le Bihan, Vaibhav Srivastav
- Published: Nov 21, 2025
- Type: Benchmark

The Open ASR Leaderboard expanded its evaluation framework by introducing dedicated multilingual and long-form transcription tracks across eleven datasets. Benchmarking more than sixty open- and closed-source models from eighteen organizations revealed clear operational trade-offs between accuracy and inference speed. Systems combining Conformer encoders with large language model decoders achieved the lowest word error rates in short-form English speech recognition. In contrast, connectionist temporal classification and token-and-duration transducer decoders yielded tenfold to hundredfold improvements in throughput with only slight accuracy penalties. While closed-source commercial systems maintained performance leads in long-form audio transcription, specialized single-language models consistently outperformed generalized systems supporting hundreds of languages.


### [Apriel-H1: The Surprising Key to Distilling Efficient Reasoning Models](https://yomu.fyi/post/apriel-h1-the-surprising-key-to-distilling-efficient-reasoning-models.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Torsten Scholak, Oleksiy Ostapenko, Raymond Li, Luke Kumar, Joel Lamy-Poirier
- Published: Nov 19, 2025
- Type: Problem & solution

Retrofitting efficiency into existing large language models often fails when naive distillation disrupts complex multi-step reasoning mechanisms. The Apriel-H1 project addresses this challenge by converting a 15B full-attention reasoning model into a Mamba hybrid using targeted distillation on high-quality SFT reasoning traces rather than diffuse pretraining data. Built on the open-source Fast-LLM framework, the process applies reverse KL divergence alongside a staged layer replacement strategy combining Leave-One-Out analysis and dynamic loss evaluations. Across fifty total blocks, replacing thirty attention layers with Mamba mixers yielded the flagship Apriel-H1-15b-Thinker-SFT checkpoint. This configuration achieved a 2.1x throughput increase with minimal benchmark degradation after 76.8B training tokens.


### [Easily Build and Share ROCm Kernels with Hugging Face](https://yomu.fyi/post/easily-build-and-share-rocm-kernels-with-hugging-face.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Abdennacer Badaoui, Daniel Huang, colorswind, Zesen Liu
- Published: Nov 17, 2025
- Type: Tutorial

Compiling custom deep learning GPU kernels across architectures often leads to complex build flag management, compiler errors, and ABI issues. Hugging Face's kernels library and kernel-builder streamline this workflow by supporting backends such as ROCm, CUDA, Metal, and XPU. Using the RadeonFlow FP8 block-wise GEMM kernel optimized for the AMD Instinct MI300X GPU as an example, developers organize files into specific headers and HIP implementations. The build is configured via a build.toml manifest and flake.nix to ensure reproducible compilation across supported PyTorch and ROCm versions. Artifacts can then be pushed to the Hugging Face Hub, allowing users to load and execute custom operators directly via get\_kernel without traditional installation steps.


[Newer posts](https://yomu.fyi/topic/performance/page/25.md) · [Older posts](https://yomu.fyi/topic/performance/page/27.md)