Loading…
Topic
812 posts about Performance. Every summary links to the original.
See the idea
Bare metal gives engineers direct control over GPU, network, and virtualization layers, but it shifts elasticity and lifecycle work onto the team. This trade-off map shows when that bargain pays off.
Bun packs a runtime, package manager, bundler, and test runner into one binary. Here is how those four parts fit together and where the seams still show.
A five-step how-to for deciding when an AMD Instinct GPU should stay dedicated and when SR-IOV virtual functions, VMs, or sidecar isolation can safely share it.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
The Grounded Reasoning Cup evaluated whether AI agents could generalize grounded reasoning improvements from OfficeQA to a newly released, enterprise-style document benchmark. Eleven academic teams had about two months to build agents with a partner lab’s model family, then applied them live to OfficeQA Pro V2, based on U.S. Treasury Accounts of Receipts and Expenditures, after a 36-hour release window. Stanford won with 63.3% accuracy, while the top systems combined document preprocessing, targeted or hybrid retrieval, specialized tools, parallelism, and answer verification; 18.8% of questions remained unsolved by every team. The results show that benchmark gains did not reliably transfer, and that parsing, retrieval, tool use, verification, and operational infrastructure can matter as much as model choice in end-to-end enterprise reasoning.
PostgreSQL 19, still in beta when this was written, adds monitoring and observability changes spanning lock contention, logging, WAL, wait events, remote messages, and multixacts. The default for log_lock_waits changes to on, while log_min_messages gains per-process-type levels and autoanalyze receives its own duration setting. A new wal_fpi_bytes counter exposes full-page-image volume cluster-wide, per session, per query, and in VACUUM and ANALYZE logs; new wait events expose WAL write, flush, and replay stages plus COPY file, pipe, and program I/O. Remote server messages now use normal PostgreSQL logging with origin prefixes, and pg_get_multixact_stats() surfaces multixact usage that previously required filesystem inspection and educated guessing. The post also lists upgrade work for parsers, metric collectors, wait-event dictionaries, configuration UIs, and alert thresholds, while noting that final release notes may differ.
Sentence Transformers v6.0 adds MultiVectorEncoder, bringing ColBERT-style late-interaction retrieval into the library’s existing API for dense, sparse, and reranker models. The encoder retains one projected, typically 128-dimensional vector per token instead of pooling a document into one vector, allowing documents to be encoded independently and indexed offline. At scoring time, MaxSim finds each query token’s highest dot-product match among document tokens and sums those maxima; contextual embeddings support both paraphrase matching and exact-token evidence. The guide covers checkpoint loading, encoding, scoring, semantic search, retrieval and reranking, indexing, token pooling, inference speed, evaluation, and retrieval from page images, audio, and video. It also quantifies the storage trade-off: 4,874 passages produced 608,414 token vectors and a 311.5 MB float32 index, while fast-plaid compression reduced that example to 92 MB.
Databricks Feature Store addresses the lag between batch-computed machine-learning features and decisions that depend on events from seconds earlier. It lets a feature definition drive both offline batch processing and online streaming pipelines, orchestrating Kafka ingestion, Spark Real-Time Mode (RTM), Lakebase, and Model Serving. For a rolling 10-minute transaction sum, RTM updates per entity state in local RocksDB, expires contributions per event, and writes revised values to Lakebase through a streaming JDBC sink. The described path reports end-to-end p99 200ms latency from Kafka arrival to online feature availability, while Model Serving retrieves declared dependencies automatically at inference. Tumbling and sliding windows remain options when fewer, cheaper updates are acceptable.
The post presents a constraint-aware GPU allocator as a response to utilization losses caused by fixed real-time reservations and FIFO ordering under contention. It compares the allocator with FIFO on identical hardware and workloads across seven benchmark scenarios, modeling batch-like jobs as contiguous GPU blocks and real-time inference as a timestep-by-timestep demand curve with bounded GPU swaps. Across five contention scenarios, utilization improved from 52–85% to 72–88%, while priority-weighted value increased 24.6–105.1%; the strongest case gained 33 percentage points of utilization and 105% value, and the scale test gained 15.9% value at identical utilization and throughput. The scheduler optimizes a 24-hour horizon but commits only the current timestep, re-running every 30–60 minutes so fresh observations absorb forecast error while preserving running work.
The post argues that the “prototyping tax”—delay between an AI-powered idea and a working MVP—causes initiatives to lose momentum, often because organizational context is fragmented across teams, tools, and codebases. Traditional encapsulation and siloed domain knowledge can leave agents with syntax and contracts but not business intent, making production codebases harder to navigate than personal projects. The proposed response is agentic development grounded in business semantics and governed data, with alignment occurring through working builds while CI/CD, code review, and production rigor remain unchanged. The post recommends tracking time-to-prototype, first-pass acceptance rate, and PoC-to-production rate to distinguish faster demos from durable delivery. It cites a benchmark of 401 data tasks and Abacus Insights results: 77% accuracy at roughly half the cost for a platform-native agent, plus about 40% less manual effort and onboarding to first value in roughly half the time.
NVIDIA’s post presents a workflow for developing an NVFP4 checkpoint of the Nemotron 3.5 Lightning 30B-A3B model, targeting lower memory use and higher inference throughput without sacrificing accuracy. The method starts with post-training quantization (PTQ), producing a W4A16 student from the BF16 teacher, then applies quantization-aware distillation (QAD), where simulated quantization and KL-divergence loss align student logits with the frozen teacher. For the selected four_over_six recipe, 32K calibration performed best, and the recipe using W4A16 Mamba linears offered the best reported trade-off between accuracy degradation and inference performance; the checkpoint is 22 GB versus 66 GB and delivers up to 4x faster throughput. The workflow includes Model Optimizer examples and launchers for Megatron-LM and Megatron-Bridge, covering calibration, distillation, evaluation, and export.
AI context architecture defines the boundaries, data access, memory, and fallback behavior that constrain AI agents and make their outputs more predictable. The discussion distinguishes architecture as system design, infrastructure as the storage and delivery of context, and engineering as implementation; MCP is treated as an architectural protocol, while RAG spans all three layers. Examples include restricting retrieval to car or sports-car tires, preserving agent history, and routing incomplete or low-trust knowledge to subject-matter experts through a human-in-the-loop flow. The buy-versus-build discussion notes that coding a custom architecture is possible, but buying can provide accumulated experience with data treatment, edge cases, and recurring operational problems, while good architecture also reduces token costs and aims for consistent, predictable answers.
On August 12, a total solar eclipse crossed the North Atlantic and Europe, passing directly over Iceland, northern Spain, and Portugal. Cloudflare Radar analyzed HTTP request volumes in five-minute intervals across affected countries, comparing the data against a baseline derived from the previous three Wednesdays. Traffic reductions aligned closely with peak solar obscuration, with totality regions recording drops between 15% and 30% and specific areas shifting by up to -46.7%. Once the sun reappeared, request volumes rebounded to normal levels within minutes as users resumed device activity. The observed traffic declines were driven by aggregate human behavior rather than network failures.
Unity AI Gateway’s Smart Routing, now in Beta, addresses the cost and choice overload of assigning coding tasks to increasingly diverse models and harnesses. It uses task-aware routing: a small, low-latency model classifies the initial task description and metadata, then a policy selects a cheaper, medium, or frontier model; Omnigent can also choose the coding harness and route sub-agents. The design keeps consecutive turns on one model to preserve cache-hit rates, while escalating complex work and delegating simpler tasks to lower-cost options. On internal coding workloads, Smart Routing achieved 35% savings and outperformed any single model at 65% of Opus 5’s cost; public benchmarks reported matched Opus 5 performance with 56% savings. The team identifies mid-session reassessment, real-session evaluation, and affordable switching as continuing challenges.
This walkthrough presents a streaming data loop for Strands Robots that records robot demonstrations, stores them in Hugging Face Storage Buckets, trains from the growing dataset, and deploys a resulting policy back to hardware. It uses the LeRobot on-disk format throughout: Robot("so100") records through DatasetRecorder, sync_dataset_to_bucket writes changed bytes to a mutable bucket, and stream_dataset reads the dataset from the Hub without a full local download while decoding camera video on the fly. The agent can decide which episodes to keep and when to retrain, while the same Robot object records and streams data; deployment changes the robot mode keyword from simulation to real. The default simulation path runs with a mock policy that produces a valid dataset but not a useful one, while hardware, bucket, and training paths require additional dependencies.
ClickHouse’s Grafana plugin has shipped a set of changes aimed at making common observability tasks—finding log lines, narrowing to services, and detecting deployments—faster without requiring SQL. Version 4.20 adds single-table configuration and compact query mode, where users search logs, apply type-aware filters, review query history, and see or edit generated ClickHouse SQL. Investigation can proceed by clicking through collapsible OpenTelemetry Resource, Scope, and Log attributes, filtering values or selected message text, while dashboards can discover Map keys and values across tables. The release also includes interactive Logs Explorer, Traces Explorer, and per-service OpenTelemetry dashboards, plus guided variable and annotation editors that generate editable SQL for distinct values and change markers such as deployments and rollbacks. Additional changes cover schema-introspection caching, trace-ID lookup, Grafana 13 compatibility, SQL validation, backend OpenTelemetry instrumentation, and forwarding Grafana identity to ClickHouse; compact mode initially remains limited to single-table data sources.
In Text2SQL systems, generating SQL on the fly with frontier foundation models can cause response times of 25 to 30 seconds due to 60,000-token prompts and model latency. To solve this without degrading accuracy, an intelligent caching layer was built using parameterized SQL query templates paired with question embeddings. When an incoming question matches a cached template via semantic search above a confidence threshold, named entity recognition extracts values to populate template placeholders, bypassing the initial SQL generation step. In production, this approach reached a 60% cache hit rate after two weeks, cutting per-hit token consumption by roughly 90% and reducing overall latency by 80% to under 5 seconds.
HeyGen and Google Cloud ported Avatar IV, an 18B-plus-parameter diffusion stack for streamed talking-head video, to an eight-chip Trillium (v6e) host. The pipeline uses two diffusion transformers and a VAE decoder, with FSDP weight sharding and Ulysses sequence parallelism across the mesh, and runs through torchax, JAX, XLA, and shape-specific Pallas kernels. Optimization targeted exposed all-to-all collectives, frame-misaligned sparse-attention blocks, serial softmax work, layout copies, scheduler choices, and custom-kernel cost estimates; changes were measured per video chunk against output-quality gates. The resulting pipeline was 1.86× faster than the first working TPU version, comparable in performance to the 8×H100 production setup and up to 25% more cost efficient per generated-video minute, while shipped changes preserved byte-identical output or passed defined similarity review.
Databricks’ serverless platform launches tens of millions of VMs daily, and each VM needs network configuration before serving customer workloads. The original design synchronously aggregated data from multiple upstream services on the cluster-creation critical path, producing 5,000-ms p99 RPC latency and availability bottlenecks as request volume reached billions per day. The replacement uses change events through a message queue, asynchronous per-workspace recomputation, partition-local snapshots, and periodic reconciliation to recover from missed events. The serving path now performs a single snapshot-store read without upstream calls, while events carry workspace and resource identifiers rather than customer data. After rollout, p99 RPC latency fell to 125 ms, server success rate rose to 99.99%, upstream call volume dropped 86%, and the legacy synchronous framework was deprecated.
Alibaba’s Qwen3.8-2.4T-A95B (Qwen3.8-Max) is an open-weight 2.4T-parameter model with 95B activated per token, designed for reasoning and agentic workloads. Its fine-grained mixture-of-experts architecture combines full and linear attention, supporting context of up to one million tokens and output lengths up to 128K while using bounded recurrent state in linear-attention layers. NVIDIA describes multinode serving on the 72-GPU GB300 NVL72, where a 130 TB/s NVLink domain supports expert communication; in FP8, the model reportedly exceeds 4K tokens per second per GPU and 350 tokens per second per user without additional tuning. Developers can control reasoning depth with low, high, or xhigh modes, choose SGLang, vLLM, NVIDIA Dynamo, or NIM for serving, and use NeMo AutoModel for full SFT or LoRA post-training.
OlmoEarth Studio now supports custom computation and export of embedding vectors, compact representations of Earth-observation data generated by open-source OlmoEarth foundation models. Users can select an area, a one-to-twelve-month time span, an encoder variant, spatial resolution, and Sentinel-2 L2A or Sentinel-1 RTC imagery, then receive a Cloud-Optimized GeoTIFF with one int8 band per embedding dimension. The post demonstrates similarity search, few-shot segmentation, change detection, and PCA-based exploration; a logistic regression trained on 60 labeled pixels produced a weighted F1 of 0.84 for three land-cover classes in Ca Mau, Vietnam. Embeddings provide a fast, shareable starting point, while supervised fine-tuning can improve performance for applications needing task-specific results; cloud cover, atmospheric artifacts, and missing observations can affect quality.
AI infrastructure spans compute, networking, storage, orchestration, and applications, so a symptom at one layer may originate elsewhere. The post proposes selecting observability tools by failure domain, mapping components to telemetry sources, and reducing signals to a top-k alert set in a unified Prometheus/Grafana triage dashboard. For a DGX cluster using InfiniBand, BCM, and Slurm, it recommends Redfish/IPMI, DCGM, NVSM, UFM, and BCM, while deferring NetQ, NMX, Run:ai, and NIM until their corresponding capabilities are needed. The framework emphasizes DCGM for GPU metrics, NVSM for DGX system health, UFM for InfiniBand, and BCM for cluster and job aggregation, with specialized tools retaining low-level telemetry. An elevated InfiniBand bit error rate illustrates how retransmissions can slow one rank during NCCL all-reduce, causing synchronized training throughput to follow the slowest rank; alerts should connect concise signals to owners and remediation actions.
iFood’s security incident response team needed high-volume log ingestion, long-term retention, and rapid access to months of history, but its Databricks-based platform became costly to scale beyond 130 TB and imposed query timeouts. The team ran a production-data proof of concept and rebuilt its tooling around ClickHouse Cloud on AWS: logs land in S3 and flow through ClickPipes into ClickHouse, with Querybook preserving notebook-based investigations and ClickStack planned for operational metrics. With ClickHouse Cloud, iFood reports 9–16x faster queries, expected cost reductions of 40–50%, and data freshness improving from hourly batches to 2–10 minutes; alerts that previously ran hourly can now run about every 10 minutes. That foundation also supports agentic threat hunting: multiple sub-agents query more than 30 TB in parallel, reducing a hunt from about a week to roughly two hours, while Langfuse is used to inspect and tune agent analyses.
JetPack 7.2.1 adds support for PyNvVideoCodec 2.2, NVIDIA's Python library for hardware-accelerated video encoding and decoding on GPUs, alongside foundational agentic video skills for Jetson workflows. These skills inspect the live target, query codec capabilities, generate Video Codec SDK or PyNvVideoCodec recipes, execute runs, and return measurements, warnings, and reproducible evidence. PyNvVideoCodec keeps frames in GPU-resident device memory through DLPack and CUDA device buffers, with multi-mode frame sampling and a ThreadedDecoder that pre-decodes frames in a background thread. The release also enables emulation of recently announced Jetson T3000 performance on a Jetson T5000 module in the Jetson Thor AGX Developer Kit, while the video skills are limited to Video Codec SDK and PyNvVideoCodec stages.
The best engineering reads, once a week.
