Loading…
Topic
812 posts about Performance. Every summary links to the original.
See the idea
Bare metal gives engineers direct control over GPU, network, and virtualization layers, but it shifts elasticity and lifecycle work onto the team. This trade-off map shows when that bargain pays off.
Bun packs a runtime, package manager, bundler, and test runner into one binary. Here is how those four parts fit together and where the seams still show.
A five-step how-to for deciding when an AMD Instinct GPU should stay dedicated and when SR-IOV virtual functions, VMs, or sidecar isolation can safely share it.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Critical Manufacturing faced performance and scaling bottlenecks while processing millions of factory floor events using a legacy Microsoft SQL Server analytics stack. Increasing data volumes caused historical queries to take hours, straining real-time dashboards and escalating operational maintenance. The engineering team replaced this architecture by migrating analytics workloads to ClickHouse connected directly to streaming Kafka topics. They implemented an ELT pattern alongside ReplacingMergeTree engines to deduplicate late-arriving sensor records and utilized TTL policies for automated lifecycle management. The resulting platform delivers sub-second analytical queries across historical and real-time operational data, expands into OpenTelemetry observability storage, and maintains embeddings for retrieval-augmented generation workloads.
Ramp describes how it benchmarks large language models (LLMs) used in financial products against day-to-day tasks such as invoice extraction, financial-statement OCR, policy review, accounting autocoding, compliance detection, and fund routing. These benchmarks combine task-specific metrics with cost, latency, reasoning effort, human decisions, historical context, and ground-truth datasets where available. For contextual invoice OCR, perfect extraction requires every field to match the user's final bill, while financial-statement OCR uses a 1% relative-error threshold against transcriptions from over 500 real P&L documents. Results show application-specific trade-offs: Gemini 3 Flash is described as a cost-efficient leader for several visual and financial tasks, while Claude models lead some high-accuracy or low-miss-rate settings, and behavior can differ within one provider. The framework emphasizes Pareto trade-offs and continuous testing, with plans to expand from benchmarking current performance to hill climbing on capabilities.
Replo built an analytics pipeline on ClickHouse to serve more than 4,000 Shopify merchants processing 3,000 to 5,000 events per second. The initial single-table architecture recalculated session metrics on the fly, leading to poor query efficiency as volume scaled. To speed up dashboard queries, the team implemented a mark-and-unmark pattern with SummingMergeTree and refreshable materialized views, but real-time deduplication and fractional attribution caused runaway query backlog loops. Replo resolved the issue by limiting real-time recomputations to a specialized table holding only the last 40 minutes of purchase-related events. This architecture allowed processing over 100 billion records while keeping query times fast and ingestion lag to approximately one minute.
Ulysses Sequence Parallelism addresses the memory and compute pressure of training language models on sequences extending beyond tens of thousands of tokens, where attention remains quadratic in sequence length even with FlashAttention. It shards tokens across GPUs, projects local queries, keys, and values, then uses two all-to-all operations to exchange data so each GPU computes a subset of attention heads before restoring sequence-sharded outputs. The post describes integrations with Accelerate, the Transformers Trainer, and TRL's SFTTrainer, including variable-length handling and weighted loss aggregation for uneven valid-token counts. In reported four-GPU tests, SP=4 reduced per-GPU memory sufficiently for 96K-token training on H100 80GB GPUs, reached 13,396 tokens per second at 64K, and matched non-SP canonical token-normalized loss within 0.000005 maximum absolute difference.
LeRobot v0.5.0 is a major release that expands supported hardware, policies, datasets, simulation, and its software foundation, following more than 200 merged PRs and more than 50 new contributors since v0.4.0. It adds full Unitree G1 humanoid support for locomotion, manipulation, teleoperation, and whole-body control, alongside OpenArm, Earth Rover, CAN-bus motor controllers, and other integrations. The policy additions include Pi0-FAST's autoregressive VLAs with FAST action tokenization, Real-Time Chunking for continuously blended predictions, Wall-X, X-VLA, SARM, and PEFT fine-tuning. For data and simulation workflows, streaming video encoding removes waits between recording episodes, while EnvHub loads environments from the Hugging Face Hub and IsaacLab-Arena supplies GPU-accelerated simulation. The codebase now requires Python 3.12, uses Transformers v5, supports installable third-party policy plugins, and adds remote Rerun visualization plus installation and documentation improvements.
Grab’s Android app cached runtime images with DiskLruCache, but its 100 MB size limit did not remove stale content while the cache remained below capacity, contributing substantially to device storage usage. The team developed Time-Aware Least Recently Used (TLRU), which combines LRU size eviction with time-based expiration using last_accessed metadata and a TTL, while a minimum cache-size threshold protects a baseline of cached images. TLRU retains the 100 MB maximum and evicts entries when current_time minus last_accessed exceeds the TTL; the implementation also supports reading existing LRU journal files by ignoring timestamp suffixes, enabling safe rollback. Configuration was tuned through experiments using cache hit ratio, with a maximum acceptable decline of three percentage points. After production rollout, P95 total app size fell approximately 50 MB, while hit ratio stayed within target and infrastructure costs did not increase.
Deploying Vision–Language–Action policies on embedded robots is constrained by compute, memory, power, and real-time control requirements; synchronous inference can leave the arm idle and cause oscillatory corrections. NXP presents dataset-recording practices for a tea-bag placement task, including fixed cameras, controlled lighting, a gripper-mounted view, calibration backups, and diverse training and validation splits with recovery episodes. The example uses 120 episodes, three 640×480 cameras at 30 fps, batch size 8, and checkpoint selection based on training and validation success rather than loss alone. For deployment, the VLA is decomposed into vision, LLM backbone, and action-expert stages, with selective quantization and asynchronous scheduling constrained by inference time being shorter than action execution. On the i.MX 95, optimized ACT latency reached 0.32 seconds with 0.89 global accuracy, while ONNX FP32 ACT reached 0.96 global accuracy at 2.86 seconds.
Supabase Storage has shipped updates live in all regions for projects using Storage, targeting performance at scale, security vulnerabilities, and reliability issues. As projects reached tens of millions of objects, concurrent uploads exposed write bottlenecks, certain SQL deletes could orphan files, and a crafted path could escape the file backend’s storage root. The object-listing rewrite removes the prefixes table, six triggers, and twelve helper functions, derives folder structure from the objects table with a hybrid skip-scan algorithm, and uses cursor-based pagination; deep pagination on 60 million rows is up to 14.8 times faster without a write penalty. Direct SQL deletes are blocked by default through a statement-level trigger, while a configurable 30-second timeout supports cancellation of disconnected clients via native Postgres protocol. Idempotent migrations, TUS lock-race prevention, orphan-scanner fixes, OpenTelemetry metrics, and adapter and API bug fixes are included, with no code changes required.
PRX Part 3 reports a 24-hour speedrun for training a text-to-image diffusion model, combining techniques that performed well in earlier experiments under a budget of about $1,500 on 32 H200 GPUs. The recipe uses x-prediction in pixel space with 32-pixel patches and a 256-dimensional initial bottleneck, followed by LPIPS and DINOv2 perceptual losses, TREAD token routing, REPA alignment with DINOv3, and Muon for matrix parameters while Adam handles the rest. Training ran for 100,000 steps at 512px with batch size 1,024, then 20,000 steps at 1024px with batch size 512 and no REPA, across three synthetic datasets. The resulting model is described as usable, with strong prompt following and consistent aesthetics, though texture glitches, anatomy issues, and difficulty with hard prompts remain. The post attributes these limitations mainly to undertraining and limited data diversity and presents the recipe as a foundation for further scaling.
Shopify’s SimGym addresses the difficulty of evaluating storefront changes when merchants lack enough traffic for conventional A/B tests, by simulating shopping sessions with LLM-guided robots. Each robot runs in a cloud Chromium browser with a persona, budget, and intent, while a counterpart tests the alternate theme; the system uses real page representations rather than a mocked DOM. To support bursty, sequential workloads with 89K–127K accumulated tokens per session, Shopify self-hosted gpt-oss-120b on 48 NVIDIA B200 GPUs, combining MXFP4 expert quantization, FP8 KV-cache, FlashInfer kernels, async scheduling, and guided decoding. Production measurements report 80K to 88K tokens/second per B200 from async scheduling, a 57% reduction in HTTP/gRPC overhead, and MIG experiments yielding about 20% lower latency and 12% higher daily throughput without quality regression; speculative decoding is validated but queued.
Mixture of Experts (MoEs) in Transformers let a Transformer retain its backbone while replacing selected feed-forward layers with routed expert sub-networks, allowing total model capacity to exceed the active parameters used per token. The post explains why sparse models can reduce inference cost and memory pressure, then focuses on making them first-class in Transformers. Its weight-loading refactor converts independently serialized expert tensors into packed runtime layouts through composable WeightConverter operations and lazy materialization. The Experts Backend supplies eager, batched_mm, and grouped_mm execution strategies, while expert parallelism shards experts across devices and aggregates partial outputs. Training support uses grouped GEMM and custom Triton plus LoRA kernels, with reported improvements of approximately 12× speed, over 35% VRAM reduction, and roughly 6× longer context.
Shopify describes a foundational generative recommender that predicts next products, and in some contexts ads, from buyer event sequences spanning searches, views, add-to-carts, favorites, and purchases. The system uses an autoregressive model with causal masking, time-aware attention combining RoPE-inspired rotary encoding with relative attention bias, and sampled softmax with shared and positive-aware hard negatives. Built on HSTU, it combines richer activity data, ensemble-focused training for incremental recall, and optimized CUDA kernels; the full training pipeline reached up to 7.3x the speed of a baseline implementation. In an online test of an August model version, relative shop orders rose 0.94%, high-quality click-through rate rose 5%, conversion rate rose 0.71%, and final served product recall at 2 rose 4.8%.
Writing CUDA kernels that integrate with diffusers and transformers involves GPU-specific optimization, library integration, PyTorch registration, and environment compatibility. It presents a cuda-kernels agent skill that packages architecture guides, integration patterns, kernel templates, troubleshooting material, benchmarks, and Kernel Hub examples for coding agents. Claude and Codex used it to generate RMSNorm, RoPE 3D, GEGLU, and AdaLN kernels for LTX-Video in diffusers, and an RMSNorm kernel for Qwen3-8B in transformers, including CUDA source, PyTorch bindings, build configuration, and benchmark scripts. On H100 80GB HBM3, the LTX-Video RMSNorm tests averaged 1.88x speedup and the optimized pipeline with torch.compile reached 1.43x the listed baseline, while Qwen3-8B RMSNorm averaged 1.94x and reached 2.47x at 8192 tokens. The skill supports packaging kernels for precompiled distribution through the Hub, where get_kernel selects a matching binary across Python, PyTorch, and CUDA variants.
Transformers.js v4 is now available on NPM, marking a major release after a year of development. It introduces a C++-rewritten WebGPU runtime tested across about 200 supported model architectures, enabling the same JavaScript code to run WebGPU-accelerated models in browsers, Node, Bun, Deno, and desktop applications. Specialized ONNX Runtime Contrib Operators support new export strategies for large language models, and the post reports a roughly fourfold speedup for BERT-based embedding models using MultiHeadAttention. The release also restructures the repository as a pnpm-workspaces monorepo, modularizes model definitions, moves examples to a separate repository, and switches from Webpack to esbuild, reducing build times from two seconds to 200 milliseconds and shrinking transformers.web.js by 53%. New APIs cover model asset inspection, caching, environment fetch controls, logging, and standalone tokenization, while added architectures include Mamba, MLA, and MoE models compatible with WebGPU.
Modern edge AI deployments require immediate real-time decision-making, local privacy, and offline operation while retaining the analytical scale of central cloud infrastructure. To address these hybrid demands, MongoDB and ObjectBox partnered to release the ObjectBox MongoDB Sync Connector as part of ObjectBox 5.0. ObjectBox operates as a lightweight on-device database with local vector search, multi-language bindings, and high-throughput I/O suited for resource-constrained hardware. In environments such as industrial factory floors and restaurant point-of-sale terminals, devices capture and evaluate operational data onsite without depending on active network connections. The native connector subsequently synchronizes critical operational datasets to MongoDB Atlas for long-term storage, fleet-wide monitoring, and machine learning model retraining.
The second part of a series on training efficient text-to-image models from scratch documents training-design ablations for the PRX model. The authors establish a Flow Matching baseline (PRX-1.2B, 100K steps, 256×256, AdamW, GemmaT5, RoPE, 1M MidJourneyV6 synthetic images) and benchmark interventions against FID, CMMD, DINO-MMD, and throughput across four buckets: Representation Alignment, Training Objectives, Token Routing and Sparsification, and Data. REPA representation alignment boosts early convergence when used as a burn-in, while the Muon optimizer lowered FID from 18.20 to 15.55 and improved CMMD and DINO-MMD. Storing denoiser weights in BF16 instead of using BF16 autocast raised FID to 21.87, and token routing methods such as TREAD/SPRINT were minor at 256² but became major wins at higher resolutions. Long captions proved critical, synthetic versus real data shifted texture versus structure, and upcoming work releases the full training code alongside a 24-hour end-to-end speedrun.
Engineers evaluated the GPT-OSS model for agentic reinforcement learning across multi-step tasks such as ReTool and GSM8K using the verl framework. Initial training runs suffered from exploding gradient norms, increasing KL divergence, and stagnating rewards. Investigation revealed that non-deterministic expert routing in the Mixture of Experts architecture caused log-probability mismatches between forward passes, which falsely triggered Proximal Policy Optimization clipping. The team resolved this by substituting detached current log-probabilities for old log-probabilities during on-policy updates. To address further training instabilities and memory bottlenecks, the implementation integrated an attention-sink-aware backward pass into FlashAttention v3 and applied sequence parallelism using all-to-all communication around attention layers.
Shopify uses SkyPilot to run machine-learning training workloads across GPU clusters on Nebius and GCP, addressing scarce hardware, fragmented capacity, and provider-specific interfaces. Persistent Kubernetes clusters remain managed by Shopify, while SkyPilot launches jobs and a company plugin routes requests, validates labels, injects configuration, and applies organizational policies. H200 workloads go to Nebius for InfiniBand-equipped distributed training, while L4 and CPU-only jobs go to GCP; labels also connect jobs to cost ownership, Kueue quotas, priority classes, and development-environment rules. The plugin automatically configures InfiniBand access, shared caches, and temporary storage, reducing manual setup and speeding repeated workloads. The resulting YAML-based interface preserves direct resource control while hiding cloud-selection decisions, giving teams common workflows as providers, prices, and GPU availability change.
Large Docker images for Airflow and Spark Connect caused minute-scale downloads, delaying cold starts and weakening autoscaling responsiveness. The post describes testing eStargz and Seekable OCI (SOCI) with containerd remote snapshotters, which expose registry-backed layers through a virtual lowerdir and fetch file content on demand. In production, SOCI delivered 30–40% faster startup times for Airflow and Spark Connect, while benchmark results showed SOCI matching standard images’ 5.0-second Airflow application startup; eStargz took 25.0 seconds. Configuration tuning increased download concurrency, unpack concurrency, and chunk size, reducing fresh-node image download time from 60 seconds to 24, a 60% improvement. Grab chose SOCI because Bottlerocket supports it natively, while noting that lazy loading redistributes rather than removes download work and requires workload-specific tuning.
Differential Transformer V2 introduces an architectural refinement to differential attention that pairs and subtracts query heads within the same grouped-query attention group. The architecture doubles query heads while preserving key-value head counts, which avoids increasing key-value cache demands and removes the need for custom kernels. DIFF V2 replaces the per-head RMSNorm of its predecessor with a per-token, per-head projected lambda passed through a sigmoid function. This formulation expands the context root-mean-square range to between zero and root two, preventing massive gradient magnifications, mitigating attention sinks, and stabilizing large learning rate pretraining. Ablations confirm that sharing keys and values across subtracted heads and constraining lambda scaling are necessary to prevent training instability and elevated loss.
The best engineering reads, once a week.
