Loading…
GPU
12 posts about GPU. Every summary links to the original.
NVIDIA ·
How to Size GPUs for AI Inference and TCO Without Overspending
GPU sizing for AI inference depends on use case, token patterns, latency targets, concurrency, cache hit rate, model choice and deployment strategy rather than raw hardware specifications alone. The post proposes a core-and-flex capacity model, combining baseline on-prem or reserved cloud capacity with elastic spot or on-demand GPUs for surges, launches and experiments. It also recommends matching GPU memory to prompt lengths and concurrency, while using quantization, pruning and distillation to reduce footprint and operating cost. A NeMo example prunes Qwen3-8B from 36 to 24 layers or narrows its dimensions toward a roughly 6B student before teacher-student distillation. In the illustrative run, width pruning reached lower final validation loss than depth pruning, 3.21 versus 3.60, while depth pruning converged faster.
Elizabeth GoodmanNVIDIA ·
Run NVIDIA BioNeMo NIM Microservices for Protein Structure Prediction in Claude Science
Claude Science is used with the NVIDIA BioNeMo Agent Toolkit and NVIDIA NIM microservices to investigate how the predicted Seh1 structure changes alone versus with proposed partner C1HCX1. The workflow retrieves UniProt sequences, generates unpaired and species-paired multiple-sequence alignments with MSA Search, and sends those inputs to OpenFold3 and Boltz-2 for independent monomer and heteromer predictions. Using MSA input, heteromer iPTM reached 0.85 for OpenFold3 and 0.82 for Boltz-2, versus 0.14 and 0.19 without it; five-sample runs had standard deviations no greater than 0.006. Both models placed partner β-strands at Seh1’s WD40 propeller closure, while core Cα-RMSD remained 0.68 Å and 0.65 Å, respectively, but the predicted interaction remains experimentally unverified.
Michelle HortonFine-Tuning SOTA Object Detection Models on Real-World Datasets
This practical comparison examines how YOLO12, YOLO26, and RF-DETR behave when moved from COCO to specialized real-world object-detection datasets, and how fine-tuning changes the outcome. The authors first evaluate six pretrained checkpoints on COCO val2017, then use cable-damage, bone-fracture, and soda-bottles datasets from RF100-VL for zero-shot testing, fine-tuning, and accuracy/latency comparison. Because the model implementations require incompatible dependencies, the workflow uses one PyCharm project with three isolated uv environments and remote GPU execution; the reported latency stays in native frameworks rather than TensorRT. All models perform well on COCO, but direct performance on specialized data falls close to zero, while 10 fine-tuning epochs adapt the detectors well to cable damage and soda bottles. Bone-fracture remains difficult despite fine-tuning, with RF-DETR performing best, indicating that domain shift may require more data, longer training, or domain-specific pretraining.
Arina BelovaRed Hat ·
Stop wasting GPU allocation in Kubernetes with GPU-pruner
GPU-pruner addresses idle GPU allocation in Kubernetes by identifying workloads whose hardware activity remains below a configured threshold, rather than relying on browser sessions, network traffic, or pod lifetime. It queries Prometheus for NVIDIA Data Center GPU Manager metrics, including engine activity and GPU utilization, and treats peak performance below 1% throughout a default 35-minute observation window as idle. After tracing an idle pod through ownerReferences or KServe labels, it scales the parent resource to zero, preserving workload metadata for later restoration. Slack notifications, grace periods, lookback checks, and dry-run mode provide safeguards, while Prometheus label handling must preserve the workload namespace and pod identity in DCGM telemetry.
Fahim UddinHow we keep GPUs reliable across Databricks AI
Databricks AI describes GPU reliability as a fleet-scale problem involving crashed jobs, silent slowdowns, and numerical corruption during distributed training. Its approach combines stress testing with demanding reinforcement learning, agentic coding, and document intelligence workloads, plus gpu-monitor, a multi-stage service that checks nodes during bootstrap, continuously while active, and periodically between workloads. The checks cover compute burn-in, peer connectivity, NCCL correctness and bandwidth, RDMA, ECC and HBM memory, PCIe integrity, throttling, thermal conditions, XID errors, and fabric health. A production incident showed why cumulative InfiniBand port downtime matters: a single outage exceeded the lower-level NCCL_IB_TIMEOUT, killing a collective before the longer PyTorch watchdog timeout. Nodes that fail checks are quarantined, while inter-node probes sweep message sizes from 8 bytes to 2 GiB and apply latency or BusBW criteria; the system is continuously expanded as new failure modes emerge.
Steven Chen, Feng Wang, Bhavik Soni, Chengguang Yang, Albert Zhong, Naren Loganathan, Harsh Panchal, Jianwei XieHow Databricks is turning video into searchable, actionable intelligence
Databricks presents video analysis as a data engineering problem for organizations with terabytes of footage that is difficult and expensive to review manually. An app accepts a video and a natural-language prompt, then triggers a Lakeflow job on Serverless GPU Compute to run Meta’s SAM3 segmentation model frame by frame and retain matching moments. Those clips preserve original timestamps and are sent through the Databricks Foundation Model API for summaries that can be written to tables or passed into downstream workflows. In one example, 26 minutes of traffic footage became one minute and 55 seconds of relevant video. The model-agnostic pipeline uses MLflow signatures to support interchangeable or custom models, while event-driven execution and independent serverless GPUs allow concurrent processing without cluster management or idle GPU costs.
Justin Monaldo, Kacey Hertan, Yvan AquinoGenesis Workbench: A blueprint for industry AI in life sciences, powered by Databricks and NVIDIA
Genesis Workbench is an open blueprint for a Databricks-native life-sciences application that brings computational drug-discovery stages into one governed workbench. It combines Unity Catalog governance, MLflow tracking, Model Serving, serverless GPU compute, Databricks AI Search, and NVIDIA technologies including CUDA-X libraries, Parabricks, BioNeMo tools, GenMol, and Proteina-Complexa. Independent modules cover genomics, single-cell analysis, large- and small-molecule workflows, and model fine-tuning, with handoffs spanning gene-to-sequence resolution, structure prediction, docking, ADMET, and candidate ranking. A point-and-click React interface supports bench scientists, while declarative workflow generation and MCP exposure let pipelines and external clients use the workbench; inference runs on GPU endpoints inside the governed workspace without runtime external API dependencies. The stated aim is to let teams move from disease hypotheses to ranked therapeutic candidates on their own data, with a roadmap for automated workflow generation, BioNeMo Skills integration, and additional MCP services.
Mark Lee, Srijit NairDatabricks and NVIDIA: Building for the Agentic Era
Databricks and NVIDIA describe a full-stack partnership for enterprise AI, spanning governed model training, inference, agent infrastructure, and industry-specific AI. Databricks AI Runtime brings NVIDIA Hopper GPUs and Quantum InfiniBand to governed data for distributed training, with Blackwell preparation, Free Edition GPU support, and planned NGC container support. Model Serving uses NVIDIA hardware and Triton Inference Server for low-latency, high-throughput production inference, while NVIDIA Vera targets agent harnesses, tool calls, analytics, and multi-step orchestration with up to 3x faster SQL queries and 80% faster agentic performance. The collaboration also places NVIDIA Agent Toolkit on Databricks Apps, integrates GPU debugging through Genie Code, and brings frameworks such as BioNeMo, Parabricks, cuOpt, and Isaac Sim to Databricks workflows. The stated vision is an end-to-end stack in which GPUs handle models and purpose-built CPUs handle surrounding agentic workloads, with governance supplied by the Databricks platform.
Hanlin Tang, Tejas SundaresanWhat’s coming next to Free Edition
Databricks is expanding Free Edition, launched twelve months ago to help people learn modern data and AI, after providing more than $10M in credits to over 500,000 learners. The five additions are Genie Code, serverless GPUs subject to availability, Lakebase, Agent Bricks, and Lakeflow Designer. Genie Code can generate, run, interpret, and refine code autonomously for tasks such as dataset analysis, pipeline cleaning, and visualization, while GPUs support deep-learning workloads including training, fine-tuning, inference, and large-dataset processing. Lakebase adds a fully managed Postgres-compatible database for data apps and AI agents; Agent Bricks supplies composable tools, memory, orchestration, and evaluation; and Lakeflow Designer enables visual pipeline construction. Together, these additions are presented as a complete free toolkit covering data engineering, analytics, machine learning, app development, and AI agents, allowing users to build end-to-end data and AI projects in Free Edition.
Will Valori, Trang LeWhat’s New in the AI Platform: Agents for ML Engineering, Our Deep Learning Platform, and New Capabilities for Real-Time ML
The announcement presents three additions to the Databricks AI Platform: Genie Code support for ML engineering, AI Runtime’s serverless GPU environment, and expanded real-time ML capabilities. Genie Code integrates with Unity Catalog, Feature Store, training, serving, monitoring, and MLflow, assisting with feature engineering, model training, deployment, evaluation, and production operations. AI Runtime provides on-demand serverless NVIDIA A10 and H100 GPUs, supports high-performance multinode training with RDMA and high-performance data loading, and adds Lakeflow Jobs, DABs, MLflow, and Unity Catalog integration. For real-time ML, the platform adds declarative feature engineering, streaming features, online feature serving on Lakebase, and enhanced Model Serving targeting 300K+ QPS with under 10ms p99 latency overhead. Reported customer examples include faster workflows, lower infrastructure costs, and production scaling beyond 100K QPS.
Tejas Sundaresan, Mike Del BalsoReliable LLM Inference at Scale
Databricks describes an inference platform serving open-source and proprietary frontier models at more than 125T tokens per month, with reliability as the central challenge under spiky, multi-tenant demand. It identifies failures and latency risks from expensive GPU configurations, variable input and output lengths, long-context requests, multimodal preprocessing, and engine-level hangs. Its architecture combines rate limiting, capacity management, model units, Dicer load-aware sticky routing, and autoscaling; model units estimate multidimensional request cost and allocate capacity more predictably. For runtime reliability, prioritized black-box health checks trigger Kubernetes liveness restarts for silent hangs, completing detection and recovery in less than five minutes while eliminating false probe failures. Image-processing changes, including Torchvision processors and OMP_NUM_THREADS configuration, removed CPU throttling and increased completed requests per second by more than threefold with the same replicas and load; bursty workloads also achieved over 80% GPU savings versus static peak provisioning.
Ying Chen, Wendy Hu, Ankit Mathur, Mike Eastham, Pei-Lun Liao, Wai Wu, Arjun DCunhaHow Superhuman and Databricks built a 200K QPS inference platform together
Superhuman replaced a DIY vLLM-based serving stack for its grammar-correction model with Databricks model serving, targeting peak traffic above 200,000 QPS, sub-second P99 latency, four-nines reliability, and no quality regression. The platform combines an Endpoint Discovery Service with power-of-two-choices load balancing, concurrency-based autoscaling, and asymmetric scale policies to reduce hotspots and handle rapid traffic ramps. Lazy-loading container images changed startup from several minutes to a few seconds by fetching metadata first and retrieving 4MB blocks on demand. Runtime work on H100 GPUs raised per-pod throughput from 750 to 1,200 QPS: FP8 weight quantization contributed up to 30%, while a multiprocessing server added 20%, alongside C++ and asynchronous scheduling optimizations. The teams validated changes through shadow testing and evaluation harnesses, and Superhuman is migrating additional models and workflows to the platform.
Myke Troianovskyi, Christoph Stüber, Wai Wu, Arjun DCunha, Amine El Helou, Tian Ouyang, Jarek Odzga, Alex Coleman
