Activation Steering: One Dial, Two Costs
Activation steering edits a model's internal activations to add or remove a concept at inference time. This explainer maps its one control knob, its two costs, and when it beats prompting.
Loading…
Topic
LLMs
655 posts about LLMs. Every summary links to the original.
- 655 posts
- 49 companies
- 5 explainers
See the idea
LLMs explainers
Antibodies as a Prediction Pipeline: Where AI Models Break Down
Antibody discovery is a three-stage prediction pipeline, and AI models that excel on familiar targets degrade sharply on unfamiliar ones. This explainer maps where that gap appears and what it costs.
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How Databricks rolls out frontier models to 14,000 employees on Day 1
Databricks provides employees Day 1 access to new AI models using Unity Gateway for governance and cost management. The company uses a three-step pipeline: immediate experimental availability, per-user budget constraints, and data-driven promotion decisions. Unity Gateway CLI distributes model configurations to employee laptops, tagging new models as experimental. Per-user budgets include a monthly maximum, daily runaway limit, quality frontier budget, and experimental budget to control costs. Within three days of the Opus 5.5 and GPT-6 Sol launches, Databricks gathered benchmark, user feedback, and cost data to confirm they were on the efficiency frontier and promoted them, with Opus 5.5 showing a 29% cost reduction per session over Opus 4.8 and GPT-6 Sol showing a 48% reduction over GPT-5.6 Sol.
How to rank fraud detection models using custom cost metrics
Accuracy is misleading for fraud detection because a model labeling everything legitimate can score 96% while catching zero fraud when fraud is only 4% of transactions. The post runs a hand-built Python grid search across algorithm, hyperparameters, feature set, class weight, and decision threshold, ranking by a custom cost metric balancing fraud savings against a US 6 analyst cost per flag. On a 480-transaction test split from a 2,400-transaction dataset with 4% fraud, the winning logistic regression caught 15 of 19 frauds, saving US 3,578 over a baseline that caught none. The winner’s evidence is passed to a local 1-billion-parameter LLM that drafts analyst notes, keeping the LLM downstream of the decision to avoid hallucination. The same workflow scales to Red Hat OpenShift AI by swapping local components for enterprise services like AutoML with AutoGluon and model serving via vLLM.
Automated merchant review summary system with integrated feedback
The system uses representative sampling, large language model (LLM) summarization, automated quality checks, and consumer feedback to generate concise summaries from written merchant reviews. Consumers can scan an overall summary or focus on specific topics such as taste, packaging, and portion size, while merchants can identify recurring themes without reading every comment. The pipeline calculates the merchant’s rating distribution, uses a sentiment model to score reviews, and applies stratified sampling to select a representative subset before sending them to an LLM for summary generation. A separate LLM validates that each claim in the generated summary is supported by the source reviews, and summaries that fail this automated assessment are withheld from publication to reduce unsupported claims. The system regenerates summaries when new reviews or negative consumer feedback reach configured thresholds, and the feedback-driven regeneration includes anti-abuse controls to maintain trustworthiness.
Accelerating vision-language models with LFM2.5-VL-DSpark
Liquid AI released an experimental DSpark draft model for their vision-language model LFM2.5-VL-3B. The drafter adds a speculative decoding path that trades a minimal increase in memory footprint for a larger speedup without changing output quality. The 280M-parameter drafter, a 4-layer attention-only transformer, captures the target model’s hidden states at fixed tapped layers to draft a block of candidate tokens. On an M5 Max, decoding runs 2.30x to 3.13x faster by task, with end-to-end latency improvements of 1.56x to 2.62x. The drafter ships with day-one support for llama.cpp, MLX-VLM, and SGLang, and is available on Hugging Face in Safetensors and GGUF formats.
Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine
Modal and CMU built Quail, a query-aware inference layer for AI-SQL that combines a query planner with an inference engine to process structured LLM transformations of database entries. On a multi-join query, Quail achieves over a billion tokens per minute per H100 GPU, over 10x faster than vLLM, costing under 6 cents per billion tokens on Modal. The gain comes from using the query plan to order requests for better KV cache management, including a revised Hydragen-style cascade attention and batching small requests to reduce host overhead. On a new AI-SQL benchmark, Quail runs 1.84x faster than vLLM, but falls behind on agent trace processing. The authors see this as part of a trend toward custom inference engines for specific workloads, with a reusable core like databases post-DataFusion, and note that much work was done by prompting coding agents.
Reproducing OLMo 3 7B Pre-training in MaxText: case study of large scale training on TPUs
The post describes the reproduction of AI2’s OLMo 3 7B pre-training using MaxText on Google Cloud TPUs, covering both stages, matching the reference loss curve and held-out metrics. PyTorch weights were converted to JAX with logit parity, KL divergence 1.5e-3 and 98.75% top-1 agreement, and held-out evaluations caught a data loader bug that made incorrect gains appear. The run demonstrated reliability via exact checkpoint-resume with zero loss delta, resilience to a 75% capacity loss via resize with 1% throughput preservation, and seamless TPU generation change. Performance optimizations achieved 44.5% MFU on Ironwood, and head reshaping from 32 heads by 128 to 16 by 256 yielded 12.4% faster throughput at identical parameters and FLOPs. The post emphasizes that faithful reproduction requires verification on generalization metrics and provides instructions for readers to reproduce the run with open sourced code.
Introducing NV-Reason-CT Open 3D CT VLM for Radiologist Chain-of-Thought Reasoning
NVIDIA released NV-Reason-CT, a vision language model for 3D CT analysis that processes volumes as true 3D input rather than independent 2D slices. The model generates structured diagnostic reports covering 30 chest and 29 abdominal abnormalities and emulates radiologist chain-of-thought reasoning with systematic anatomical reviews and uncertainty articulation. Its architecture pairs a Qwen3.5-4B language model with a full 3D ViT encoder that processes volumes resampled to 192³ voxels at 2mm isotropic resolution using 8x8x8 patch tokens and 3D MRoPE for spatial token relationships. Training uses supervised fine-tuning on roughly 550,000 QA examples from CT-RATE, NIH CT, and CancerVerse followed by Group Relative Policy Optimization that refines reasoning quality based on diagnostic accuracy. NV-Reason-CT is an open research foundation not a cleared clinical product and sits within NVIDIA’s Medical AI ecosystem alongside NV-Reason-CXR, NV-Segment-CTMR, and NV-Generate-CTMR.
How I massively improved my AI inference performance without buying new hardware
Default inference configurations for large language models often leave significant performance unused, even on expensive hardware like 16 NVIDIA H200 GPUs across two nodes. The author tested this setup with vLLM and Red Hat AI Inference 3.4 on a CoreWeave Kubernetes cluster, using enterprise RAG workloads with 2,000-token inputs and 100 concurrent users. By systematically comparing aggregated and prefill-decode (PD) disaggregated architectures, and tuning tensor parallelism (TP) and PD split, latency was dramatically reduced. For Qwen3-32B, time-to-first-token (TTFT) on the best aggregated configuration (2xTP8) dropped from 995ms to 304ms, and with optimized routing to 287ms — a 71% improvement without hardware changes. For Llama-70B, TTFT fell from 1,292ms to 463ms, a 64% gain. The author also built ServeIt Studio, an open-source tool that automates the full optimization pipeline on Kubernetes.
How to Guide Your Language Flow
Probe guidance, a novel method for guiding flow matching models, is introduced. It uses the frozen internal states of an existing diffusion model to construct a guidance signal, operating on a similar principle as autoguidance but removing the need for an additional forward pass at inference time. The method provides a reliable path ensuring the weak and strong models share similar dynamics. Probe guidance sets a new state-of-the-art performance on unconditional generation when benchmarked on continuous diffusion language models. Applied to a 1.7B diffusion language model, it consistently improves multiple choice question answering benchmarks.
Introducing Support for Local AI Models in the Antigravity SDK
The Antigravity SDK now supports local AI models, starting with Gemma 4 26B A4B via Google AI Edge’s LiteRT, enabling offline agentic assistance on local GPU and RAM with a recommended machine of 24GB VRAM or unified memory. Local execution eliminates API costs and rate limits while keeping code and requests on-device, offering privacy and offline resiliency. The SDK supports hybrid workflows where a cloud model like Gemini 3.8 Flash plans while local Gemma 4 instances execute, as in a code auditing task where 97.2% of tokens (3,322 of 3,417) ran locally. A built-in example generates a live-updating CLI resource monitor using psutil and rich from a single prompt, with the agent autonomously writing the script, creating requirements.txt, and testing the code entirely offline. The SDK also supports any OpenAI-compatible server like Ollama, LM Studio, or vLLM via LocalOpenAIAgentConfig without changing agent orchestration.
Enabling Private High-Performance Production AI Inference with NVIDIA Confidential Computing
NVIDIA Confidential Computing (CC) enables secure AI inference by using memory-encrypted confidential VMs, confidential GPUs, and encrypted NVLink. Running inference frameworks like TensorRT LLM in a CC environment introduces performance overhead because secure execution changes assumptions about memory movement, timing, and multi-GPU communication. To quantify this overhead, the post presents a controlled methodology comparing CC-on and CC-off performance on a workload with long input context, extended output generation, and low concurrency. With CC-aware adaptations—including pageable memory selection, asynchronous readback, GPU %globaltimer for autotuning, and CC-aware multi-GPU communication—TensorRT LLM retained 96.1–98.2% of baseline throughput on eight NVIDIA B200 GPUs while keeping per-token latency overhead below 5%. The post concludes that security configuration and inference optimization should be treated as a single deployment problem for organizations moving private inference into production.
How UK AISI and EvalEval Are Making Benchmark Results Reproducible
The UK AI Security Institute (AISI) has begun using EvalEval’s Every Eval Ever (EEE) schema and Evaluation Cards platform to openly share LLM evaluation results. This collaboration, stemming from a joint workshop at NeurIPS 2025, aims to make benchmark results more reproducible by providing standardized configuration and context alongside verified scores. AISI released results for five benchmarks—HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro, and Terminal-Bench 2.0—across six frontier models, plus two cyber evaluations using a partially overlapping model set. The release accompanies a paper studying how inference-time compute and evaluation protocol affect performance, revealing that models continued solving tasks when given correctness feedback. By adopting a shared reporting schema, evaluators can compare findings across studies and understand how setup choices influence reported performance, while EvalEval invites model developers, evaluation creators, and researchers to contribute to the shared infrastructure.
How to Evaluate AI Agents From Tool Calls to Task Completion
Standard LLM benchmarks are insufficient for evaluating AI agents because they score static outputs rather than multi-step tool-call sequences against a live environment. The post explains that agent evaluation requires two scoring layers: step-level (process scoring) checks each call’s validity and usefulness, while end-to-end (E2E) scoring checks only the final environment state. A trace logs every step and environment state, and metrics roll up through a hierarchy of benchmark, trial, task, turn, and step. Key metrics include task success rate, consistency, tool-call precision, argument accuracy, steps per success, and cost per success. The post argues that benchmarks are converging on tool use because deployed agents always use tools, and enterprise benchmarks should measure task completion against real APIs and policies.
Benchmarking LLM Inference at Scale with AIPerf
AIPerf is NVIDIA’s successor to GenAI-Perf, intended to benchmark large-scale LLM inference without letting the load client become the bottleneck. Its multiprocess architecture combines worker processes, separate record-processor services, and ZMQ coordination, while supporting 15+ endpoint types, public datasets, and trace-replay formats. The walkthrough serves Qwen3-0.6B through vLLM, using streaming synthetic requests with fixed 128-token inputs and outputs; AIPerf reports TTFT, ITL, request latency, output token throughput, percentile distributions, and optional GPU telemetry in CSV and JSON. A second profile uses Poisson arrivals averaging 10 requests per second, variable input and output lengths, and a fixed random seed, producing wider latency distributions than the static baseline as concurrency and workload variability increase.
Making Local AI Smarter and Faster
Junie Local’s update introduces Qwen3.8-3.6-27B-blend, a merged model that balances speed and capability for local coding agents. The blend, created by merging Qwen3.6 and Qwen3.8 weights in equal proportions, completed 37 tasks in a 100-task internal benchmark, compared to 34 for Qwen3.6 and 39 for Qwen3.8, while generating 71% fewer output tokens than Qwen3.8. The runtime uses multi-token prediction (MTP), with a two-token proposal on M5 MacBook Pro making decoding 60% faster. A four-bit MTP head was kept over an eight-bit one due to similar acceptance rates and memory savings. The update also brings experimental NVIDIA RTX support on Windows for machines with at least 24 GB of VRAM.
How a global fintech scaled coding agent traffic with Dedicated Model Inference
A global fintech runs its coding assistant on GLM 5.2 through Together’s Dedicated Model Inference (DMI), but spiky engineering-hours traffic outpaced static capacity planning. DMI gives the customer’s teams direct control over endpoint provisioning, scaling, configuration, model changes, and performance data through APIs, UI, and CLI. The workload is peak-load and relatively low-TPS, with ISL p50/p90/p95 of roughly 81K/163K/178K tokens and RPS p50/p90/p95 of 3/6/7, so concurrency headroom mattered more than raw throughput; the deployment used dozens of B200s, multiple replicas, and 256K context. When a reshaped endpoint later reached near-100% prefill capacity, requests queued for one to three minutes and decode throughput fell to roughly 5 tokens per second, the team restored cache-session-aware routing and widened max-inflight-per-worker, pushing the change same-day with zero downtime.
Dynamically Scaled Activation Steering
Activation steering has emerged as a way to guide generative models toward desired behaviors, but uniform interventions can reduce performance when steering is unnecessary. DSAS, or Dynamically Scaled Activation Steering, addresses this by separating when to steer from how to steer and adapting intervention strength across layers and inputs. At generation time, it computes context-dependent scaling factors that modify existing steering transformations, and it can also be jointly optimized end-to-end with the steering function. Combined with existing methods, DSAS improves the reported Pareto front between toxicity mitigation and utility preservation, while adding minimal computational overhead; experiments also apply it to a text-to-image diffusion model for modulating specific concepts and show token-level interpretability through the strength and location of steering.
coSTAR: How we ship AI agents at Databricks fast, without breaking things
Databricks describes coSTAR, a methodology for testing and iteratively improving long-running, stochastic AI agents without relying on manual review for every change. It addresses nondeterministic outputs, slow executions, cascading failures, and subjective quality by treating scenarios as fixtures, MLflow traces as a reusable harness, and judges as an automated test suite. Its two coupled STAR loops align judges with human experts using a Golden Set, then use those judges to refine the agent until it passes the scenarios. Agentic judges selectively explore traces with tools instead of receiving entire, potentially thousands-line executions, while persisted traces allow rescoring without rerunning agents. The approach can also reuse judges for production monitoring, but remains limited by manually authored scenarios, judge-calibration costs, coverage gaps, overfitting, and difficult root-cause attribution.
Catching poor LLM performance and accuracy before deployment
Rapid changes in vLLM and the path from upstream releases to Red Hat AI Inference images can introduce performance and accuracy regressions that ordinary CI/CD tests miss. The midstream regression detector benchmarks upstream and release-candidate builds across hardware variants, using GuideLLM at multiple concurrency levels for performance and GSM8K for accuracy. It stores nightly upstream results, compares them with midstream benchmarks against threshold deltas, and feeds confirmed failures into fixes and rebuilds before release. To reduce false positives from limited samples, the workflow adds Wasserstein distance as a secondary check of whole request-response distributions rather than relying only on medians or means. The system also connects benchmark runs with prior JIRA regressions through a custom Claude skill that suggests possible fixes and assesses whether a flagged change is genuine.
Migrating from closed to open source models, Together
The post presents a five-stage playbook for moving production workloads from closed-source models to open-source models: discover, evaluate, adapt, decide, and take the system to production. It argues that managed services, model-agnostic harnesses, gateways, and tooling can reduce migration complexity, potentially making the effort weeks to months rather than months to years. Discovery combines workload definition, relevant benchmark filtering, cost-per-task and runtime analysis, playground testing, and narrowing candidates to two or three models. Evaluation separates accuracy from performance and recommends replaying real traffic against explicit goals for cost, speed, and quality, while adaptation may involve prompts, inference settings, context, and the surrounding harness. The excerpt frames migration as iterative rather than assuming a model passes unchanged.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.