---
title: "GPU"
description: "107 posts about GPU, summarised, each linking to the original."
---

# GPU
> 107 posts about GPU, summarised, each linking to the original.

## Articles

### [Claude Science, an AI workbench for scientists](https://yomu.fyi/post/claude-science-an-ai-workbench-for-scientists.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: Aug 27, 2026
- Type: Announcement

Claude Science is a beta AI workbench for scientists that brings databases, research tools, computing resources, and domain-specific workflows into one environment. Researchers can analyze literature, run multi-step studies, create figures and manuscripts, and revise them through conversation. Every generated artifact includes its code, environment, plain-language creation description, and message history, while a reviewer agent checks citations, calculations, and consistency between figures and code. A coordinating agent uses more than 60 curated skills and connectors across fields including genomics, single-cell, proteomics, structural biology, and cheminformatics; it can manage jobs on local systems, HPC clusters over SSH, or Modal, scaling from one GPU to hundreds. Beta access is available on macOS and Linux for Pro, Max, Team, and Enterprise users, with reported examples including faster glioma analyses and large literature reviews.


### [Mixture-of-Kittens: our open-source MoE megakernel for NVL72s](https://yomu.fyi/post/mixture-of-kittens-our-open-source-moe-megakernel-for-nvl72s.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Stuart Sul, Nash Brown, Henry Wildermuth, William Lin & Federico Cassano
- Published: Aug 4, 2026
- Type: Problem & solution

Mixture-of-Kittens (MoK) is an open-source production mixture-of-experts (MoE) training megakernel designed for GB300 NVL72 systems, where the MoE layer had become a major training bottleneck and could consume more than half of end-to-end training time. It redesigns the layer around a single fully deterministic kernel that fuses communication and computation, minimizing CPU work and CPU-GPU synchronization while overlapping operations across the 72-GPU NVLink domain. The post describes target-specific scheduling, push-versus-pull communication choices, pipelined expert computation, ring token buffers, megakernel structure, determinism, and MXFP8 support for DeepSeek-V3-style layers. Across four model shapes, MoK is reported as up to 2.37x faster for MXFP8 forward and 1.78x for backward, while production training on 512 GPUs reached 1.41x tokens per second versus a DeepEP-based implementation. MoK is fully open-sourced and intended to remain modifiable for other platforms.


### [GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model](https://yomu.fyi/post/gem-training-how-meta-doubled-the-efficiency-of-its-llm-scale-ads-foun.md)
- Company: [Meta](https://yomu.fyi/company/meta.md)
- Published: Aug 3, 2026
- Type: Explainer

Meta’s Generative Ads Recommendation Model (GEM), a hybrid foundation model for ads recommendations on Instagram and Facebook, trains across several thousand latest-generation GPUs and combines trillions of sparse embedding parameters with billions of dense parameters. The post frames its engineering problem around recommendation-specific jagged and asymmetric sequences, memory-bound operations, numerical sensitivity, communication overhead, activation pressure, and load skew that make conventional LLM training techniques insufficient. Its co-design approach combines custom kernels such as Jagged Flash Attention (JFA), BlockAttention, and Generalized Dot-Product Attention (GDPA), mixed ultra-low-precision training including MXFP8 attention and MLP, topology-aware 5D parallelism, SM-free collectives, and memory and load-balancing techniques. The reported result is a doubling of end-to-end training efficiency to 20–25% Model FLOPs Utilization (MFU) while total training FLOPs increased fourfold over 12 months; Base Batch Shuffling (BBS) alone delivered a reported 4% efficiency gain, with 4% QPS improvement and 4% peak memory reduction.


### [Autoscaling endpoints for LLM inference](https://yomu.fyi/post/autoscaling-endpoints-for-llm-inference.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 31, 2026
- Type: Explainer

Dedicated LLM inference deployments face a trade-off between paying for idle GPUs and suffering sharp p95 latency when replicas hit batching limits. The autoscaling loop converts observed load into desired replicas with ceil(N × observed/target), then applies scale-up and scale-down windows, replica bounds, and GPU placement. It presents inflight\_requests as a leading, engine-agnostic default, while TTFT and end-to-end latency target responsiveness but trail demand, and GPU or token utilization prioritize efficiency with less latency headroom. Cold starts can take several minutes because replicas require GPU placement, weight downloads, VRAM loading, and warm-up, so policies should scale early; streaming is required for TTFT, decoding speed, and throughput-per-replica metrics, whereas end-to-end latency supports both streaming and non-streaming workloads.


### [Together AI and Y Combinator partner to launch the first dedicated GPU cluster for the YC community](https://yomu.fyi/post/together-ai-and-y-combinator-partner-to-launch-the-first-dedicated-gpu.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 20, 2026
- Type: Announcement

Together AI and Y Combinator are partnering to provide YC portfolio companies with a dedicated GPU cluster, addressing compute access as a growing bottleneck for AI-native startups. The service is designed for both inference and training, letting founders reserve, provision, and manage GPUs through Together’s self-service portal rather than making long-term commitments or routing usage through YC. Startups can run short-term sprints at long-term rates, use capacity from single-node workloads to larger deployments, and receive GPUs in minutes, while the cluster is already running at full utilization and companies can plan needs months ahead. Together says its research on attention mechanisms and the Mamba architecture targets faster inference and improved unit economics. The partners plan to expand the cluster and support startups as their requirements grow.


### [What does 99.9% uptime mean for inference?](https://yomu.fyi/post/what-does-99-9-uptime-mean-for-inference.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 16, 2026
- Type: Explainer

The post defines three inference reliability tiers by the failure domains each must survive: 99% requires surviving node-level GPU failures like ECC errors and thermal throttling through automated health checks and fast replica replacement within a single data center. 99.9% requires surviving a full data-center failure by deploying weights across two facilities with enough capacity on each side and routing live traffic to both continuously, not a cold standby. 99.99% requires multi-region deployment with AZ redundancy and reserved failover capacity. The authors argue that vague SLA definitions hide which failure domains are covered, and that infrastructure ownership—whether a provider controls its hardware or rents from a hyperscaler—determines how quickly failures are resolved.


### [New in Together GPU Clusters: Reliability and control for production GPU clusters](https://yomu.fyi/post/new-in-together-gpu-clusters-reliability-and-control-for-production-gp.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 15, 2026
- Type: Announcement

Together GPU Clusters introduces changes for production training and inference workloads, addressing hardware failures, scheduler problems, and the limits of single-admin access. Platform health improvements include continuously running passive health checks, human-approved node repair recommendations, and Slurm-on-Kubernetes 2.0, which adds self-healing daemons, durable job accounting, descendant-process cleanup, and refreshed GPU state after reschedules. Operational controls include a cluster details view with health, utilization, events, configuration, and node actions, plus external OIDC, startup scripts, and an acceptance-test opt-out. Newly provisioned Slurm clusters use the latest stack by default, while existing managed clusters can be migrated during a maintenance window at no cost; the stated result is faster recovery, fewer support tickets, and greater visibility and control for growing teams running large clusters.


### [How to price serverless GPUs](https://yomu.fyi/post/how-to-price-serverless-gpus.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Charles Frye
- Published: Jul 6, 2026
- Type: Explainer

Modal’s engineering blog presents a cost model comparing serverless and reserved GPU pricing for AI workloads, centered on the peak-to-average demand ratio. Serverless GPUs charge only for usage, while reserved GPUs require long-term contracts for fixed capacity, often achieving under 30% utilization. The model shows that when the peak-to-average ratio exceeds the reservation discount rate (typically 2-5x), serverless GPUs become cheaper despite higher base rates, with real workloads like inference and training having ratios of 5-10x, making serverless an obvious cost win. An interactive tool allows teams to estimate costs using preset workloads or custom traffic patterns, defaulting to current B200 GPU rates on Modal and hyperscaler reservations. The post discusses model limitations, including assumptions of perfect prediction and instant allocation, and notes that serverless also reduces engineering costs through faster development and reduced dev/prod skew, while mixed strategies can combine baseline reservations with serverless bursting for high peak-to-average bursts.


### [How we keep GPUs reliable across Databricks AI](https://yomu.fyi/post/how-we-keep-gpus-reliable-across-databricks-ai.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Steven Chen, Feng Wang, Bhavik Soni, Chengguang Yang, Albert Zhong, Naren Loganathan, Harsh Panchal, Jianwei Xie
- Published: Jul 1, 2026
- Type: Explainer

Databricks AI describes GPU reliability as a fleet-scale problem involving crashed jobs, silent slowdowns, and numerical corruption during distributed training. Its approach combines stress testing with demanding reinforcement learning, agentic coding, and document intelligence workloads, plus gpu-monitor, a multi-stage service that checks nodes during bootstrap, continuously while active, and periodically between workloads. The checks cover compute burn-in, peer connectivity, NCCL correctness and bandwidth, RDMA, ECC and HBM memory, PCIe integrity, throttling, thermal conditions, XID errors, and fabric health. A production incident showed why cumulative InfiniBand port downtime matters: a single outage exceeded the lower-level NCCL\_IB\_TIMEOUT, killing a collective before the longer PyTorch watchdog timeout. Nodes that fail checks are quarantined, while inter-node probes sweep message sizes from 8 bytes to 2 GiB and apply latency or BusBW criteria; the system is continuously expanded as new failure modes emerge.


### [Announcing our $800M Series C to accelerate the shift to open-source AI](https://yomu.fyi/post/announcing-our-800m-series-c-to-accelerate-the-shift-to-open-source-ai.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 1, 2026
- Type: Announcement

Together AI announces an $800M Series C to accelerate its open-source AI platform as production demand for inference grows. The company says the round includes $800 million from investors including Aramco Ventures, NVIDIA, Vista Equity, General Catalyst, and others, plus commitments for more than 500 MW of compute capacity to be capitalized independently by new investors. Together argues that closed frontier-model economics become unsustainable as agents and workflows scale, while open-weights models such as DeepSeek, Nemotron, MiniMax, Kimi, and GLM can provide equal or better performance at lower cost. Its full-stack approach spans models, kernels, compilers, inference systems, training infrastructure, hardware utilization, and software, with cited work including FlashAttention-4, Together Megakernel, together.compile, and expanded post-training APIs for tool calling, reasoning, and vision-language models.


### [Anthropic integration with Modal brings scalable compute to Claude Science](https://yomu.fyi/post/anthropic-integration-with-modal-brings-scalable-compute-to-claude-sci.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 30, 2026
- Type: Announcement

Anthropic launched Claude Science, an AI workbench where life sciences researchers run computational workloads from conversation with Claude. The workbench addresses the fragmented toolchain of local machines, HPC clusters, and cloud VMs by centralizing execution in one place. Modal integrates with Claude Science as the compute backend, routing GPU and CPU-heavy workloads to Modal sandboxes automatically and fanning out parallel tasks like virtual screening across hundreds of containers to reduce hours to minutes. Researchers can prompt for tasks such as predicting GFP structure, and Claude writes the folding code, runs it on a Modal GPU, and returns the structure with confidence scores inside the conversation, with Modal handling scheduling, shared storage, and reproducible environments. Modal is committing up to $100,000 in compute for the Claude Science Cohort, with 500– 2,000 allocations per project.


### [Together AI at ICML 2026: frontier research across the full stack](https://yomu.fyi/post/together-ai-at-icml-2026-frontier-research-across-the-full-stack.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jun 30, 2026
- Type: Announcement

Together AI says nine papers from its researchers and peers were accepted to ICML 2026 in Seoul, spanning frontier agents, model shaping, algorithmic and systems optimization, and kernels. The collection is presented as a full-stack view of AI research, with work ranging from DSGym’s unified evaluation and training API for more than 1,000 data-science tasks to ThunderAgent’s program-aware scheduling, TTT-Discover’s test-time reinforcement learning, and RARO’s verifier-free adversarial training. Reported results include up to 3.6× agent throughput, state-of-the-art discoveries from an open 120B model, a 25% expert-response win rate without a verifier, and 5M-token context on one node for Untied Ulysses. Aurora illustrates the research-to-production loop: its adaptive speculative-decoding work ships as the ATLAS speculator on the Together platform.


### [How Databricks is turning video into searchable, actionable intelligence](https://yomu.fyi/post/how-databricks-is-turning-video-into-searchable-actionable-intelligenc.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Justin Monaldo, Kacey Hertan, Yvan Aquino
- Published: Jun 26, 2026
- Type: Problem & solution

Databricks presents video analysis as a data engineering problem for organizations with terabytes of footage that is difficult and expensive to review manually. An app accepts a video and a natural-language prompt, then triggers a Lakeflow job on Serverless GPU Compute to run Meta’s SAM3 segmentation model frame by frame and retain matching moments. Those clips preserve original timestamps and are sent through the Databricks Foundation Model API for summaries that can be written to tables or passed into downstream workflows. In one example, 26 minutes of traffic footage became one minute and 55 seconds of relevant video. The model-agnostic pipeline uses MLflow signatures to support interchangeable or custom models, while event-driven execution and independent serverless GPUs allow concurrent processing without cluster management or idle GPU costs.


### [Achieve state-of-the-art inference latencies with speculative decoding](https://yomu.fyi/post/achieve-state-of-the-art-inference-latencies-with-speculative-decoding.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 24, 2026
- Type: Problem & solution

Modal launched Auto Endpoints, combining speculative decoding with Blackwell GPUs, SGLang engine, and Modal Servers to achieve low inference latency for production deployments. The decode phase, where each token requires a full forward pass loading model weights, is the primary latency source, making speculative decoding the most impactful optimization. Speculative decoding runs a separate DFlash speculator model in parallel with the target model using KV projections and a custom Triton kernel, achieving near-linear speedup proportional to acceptance length. In the Decagon voice case study, Modal cut 100ms from a 290ms baseline by mid-training a custom speculator on synthetic data, beating proprietary providers by 60ms. Modal plans to automate speculator training with continuous improvement on production data, moving toward autospec for Auto Endpoints.


### [Genesis Workbench: A blueprint for industry AI in life sciences, powered by Databricks and NVIDIA](https://yomu.fyi/post/genesis-workbench-a-blueprint-for-industry-ai-in-life-sciences-powered.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Mark Lee, Srijit Nair
- Published: Jun 23, 2026
- Type: Explainer

Genesis Workbench is an open blueprint for a Databricks-native life-sciences application that brings computational drug-discovery stages into one governed workbench. It combines Unity Catalog governance, MLflow tracking, Model Serving, serverless GPU compute, Databricks AI Search, and NVIDIA technologies including CUDA-X libraries, Parabricks, BioNeMo tools, GenMol, and Proteina-Complexa. Independent modules cover genomics, single-cell analysis, large- and small-molecule workflows, and model fine-tuning, with handoffs spanning gene-to-sequence resolution, structure prediction, docking, ADMET, and candidate ranking. A point-and-click React interface supports bench scientists, while declarative workflow generation and MCP exposure let pipelines and external clients use the workbench; inference runs on GPU endpoints inside the governed workspace without runtime external API dependencies. The stated aim is to let teams move from disease hypotheses to ranked therapeutic candidates on their own data, with a roadmap for automated workflow generation, BioNeMo Skills integration, and additional MCP services.


### [ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)](https://yomu.fyi/post/parallelkernelbench-frontier-llms-can-t-write-fast-multi-gpu-kernels-y.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jun 23, 2026
- Type: Benchmark

ParallelKernelBench (PKB) measures whether language models can replace PyTorch + NCCL implementations with fast CUDA kernels for multi-GPU workloads, where communication can account for over 20% of inference latency. Its 87 problems come from real codebases and span tensor, context, data, expert, sequence, and FSDP/ZeRO parallelism, with hardware topology supplied and symmetric memory used for direct GPU communication over NVLink. In zero-shot evaluation, GPT-5.5 solved 28 problems and produced 22 faster-than-baseline solutions; sampling three attempts raised those figures to 36 and 27, but the best fast 1@3 result was only 31%. Models performed best on familiar collective primitives, tensor-parallel GEMMs, and Ulysses-style context parallelism, while failures often involved rank coordination, data partitioning, collective ordering, and underuse of TMA or NVLS.


### [Databricks and NVIDIA: Building for the Agentic Era](https://yomu.fyi/post/databricks-and-nvidia-building-for-the-agentic-era.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Hanlin Tang, Tejas Sundaresan
- Published: Jun 17, 2026
- Type: Announcement

Databricks and NVIDIA describe a full-stack partnership for enterprise AI, spanning governed model training, inference, agent infrastructure, and industry-specific AI. Databricks AI Runtime brings NVIDIA Hopper GPUs and Quantum InfiniBand to governed data for distributed training, with Blackwell preparation, Free Edition GPU support, and planned NGC container support. Model Serving uses NVIDIA hardware and Triton Inference Server for low-latency, high-throughput production inference, while NVIDIA Vera targets agent harnesses, tool calls, analytics, and multi-step orchestration with up to 3x faster SQL queries and 80% faster agentic performance. The collaboration also places NVIDIA Agent Toolkit on Databricks Apps, integrates GPU debugging through Genie Code, and brings frameworks such as BioNeMo, Parabricks, cuOpt, and Isaac Sim to Databricks workflows. The stated vision is an end-to-end stack in which GPUs handle models and purpose-built CPUs handle surrounding agentic workloads, with governance supplied by the Databricks platform.


### [What’s coming next to Free Edition](https://yomu.fyi/post/what-s-coming-next-to-free-edition.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Will Valori, Trang Le
- Published: Jun 17, 2026
- Type: Announcement

Databricks is expanding Free Edition, launched twelve months ago to help people learn modern data and AI, after providing more than $10M in credits to over 500,000 learners. The five additions are Genie Code, serverless GPUs subject to availability, Lakebase, Agent Bricks, and Lakeflow Designer. Genie Code can generate, run, interpret, and refine code autonomously for tasks such as dataset analysis, pipeline cleaning, and visualization, while GPUs support deep-learning workloads including training, fine-tuning, inference, and large-dataset processing. Lakebase adds a fully managed Postgres-compatible database for data apps and AI agents; Agent Bricks supplies composable tools, memory, orchestration, and evaluation; and Lakeflow Designer enables visual pipeline construction. Together, these additions are presented as a complete free toolkit covering data engineering, analytics, machine learning, app development, and AI agents, allowing users to build end-to-end data and AI projects in Free Edition.


### [What’s New in the AI Platform: Agents for ML Engineering, Our Deep Learning Platform, and New Capabilities for Real-Time ML](https://yomu.fyi/post/what-s-new-in-the-ai-platform-agents-for-ml-engineering-our-deep-learn.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Tejas Sundaresan, Mike Del Balso
- Published: Jun 17, 2026
- Type: Announcement

The announcement presents three additions to the Databricks AI Platform: Genie Code support for ML engineering, AI Runtime’s serverless GPU environment, and expanded real-time ML capabilities. Genie Code integrates with Unity Catalog, Feature Store, training, serving, monitoring, and MLflow, assisting with feature engineering, model training, deployment, evaluation, and production operations. AI Runtime provides on-demand serverless NVIDIA A10 and H100 GPUs, supports high-performance multinode training with RDMA and high-performance data loading, and adds Lakeflow Jobs, DABs, MLflow, and Unity Catalog integration. For real-time ML, the platform adds declarative feature engineering, streaming features, online feature serving on Lakebase, and enhanced Model Serving targeting 300K+ QPS with under 10ms p99 latency overhead. Reported customer examples include faster workflows, lower infrastructure costs, and production scaling beyond 100K QPS.


### [Making FlashAttention-4 faster for inference](https://yomu.fyi/post/making-flashattention-4-faster-for-inference.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Charles Frye
- Published: Jun 11, 2026
- Type: Explainer

FlashAttention-4, originally designed for pre-training, required modifications to efficiently handle LLM inference workloads dominated by the memory bandwidth-limited decode phase. Modal’s contributions include adjusting parallelism strategies from query parallelism to key/value parallelism, implementing split KV to parallelize across KV tiles, and adding a heuristic for optimal splits. They added support for FP8 inputs, arbitrary KV page sizes via cp.async loads, and reduced wasted work for small query sizes by using a single query tile. The kernel also gained extended GQA packing for irregular head ratios, using cp.async for query tiles to bypass TMA alignment restrictions. These changes, measured in throughput improvements up to 3x for single-token decode and up to 2.4x for small page sizes, demonstrate the leverage of flexible parallelism and tile-based programming for inference optimization.


[Newer posts](https://yomu.fyi/topic/gpu/page/3.md) · [Older posts](https://yomu.fyi/topic/gpu/page/5.md)