Activation Steering: One Dial, Two Costs
Activation steering edits a model's internal activations to add or remove a concept at inference time. This explainer maps its one control knob, its two costs, and when it beats prompting.
Loading…
Topic
LLMs
648 posts about LLMs. Every summary links to the original.
- 648 posts
- 49 companies
- 5 explainers
See the idea
LLMs explainers
Antibodies as a Prediction Pipeline: Where AI Models Break Down
Antibody discovery is a three-stage prediction pipeline, and AI models that excel on familiar targets degrade sharply on unfamiliar ones. This explainer maps where that gap appears and what it costs.
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Welcome Inkling by Thinking Machines
Thinking Machines Lab has released Inkling and the smaller Inkling-Small family as open multimodal models that accept image, text, and audio inputs. Inkling is described as a roughly 1T-parameter decoder-only Mixture-of-Experts model, with 975B total and 41B active parameters, a 1M-token context window, and training on 45 trillion multimodal tokens. Its architecture combines relative attention, alternating global and sliding-window attention, short convolution, shared experts, hierarchical image patchification, and discretized mel-spectrogram audio processing. The release includes BF16 and NVFP4 checkpoints, speculative MTP layers, day-zero support in Transformers, SGLang, vLLM, and llama.cpp, plus deployment configurations and interactive voice and image demos. Reported benchmarks show Inkling and Inkling-Small achieving strong but varied results across reasoning, coding, vision, audio, factuality, chat, and safety evaluations.
Introducing Real World VoiceEQ: Measuring the human quality of voice AI
Existing voice-AI benchmarks increasingly suggest near-human performance, while real-world conversations still expose failures involving speaker consistency, hesitation, accents, noise, emotion, and natural responses. Real World VoiceEQ measures human quality in voice interaction by testing whether systems recognize, produce, and respond to acoustic information omitted by transcripts, including tone, emotion, speaker identity, and background context. It evaluates more than 40 proprietary and open-source models across 15+ dimensions and 60+ metrics covering ASR, TTS, S2S, and Speech Understanding, using more than 1 million human ratings, including 785,000 TTS and 48,000 STS ratings. Findings indicate that capabilities are specialized, speech-to-speech systems often underuse paralinguistic cues, and traditional benchmarks can overestimate performance in noise and other conversational conditions. The research also reports that human agreement remains essential for subjective acoustic and social judgments because SLM evaluators were less reliable outside clear, verifiable tasks.
Lessons From the Leaderboard: What 5,000+ Kagglers Taught Us About Improving AI Reasoning
The NVIDIA Nemotron Model Reasoning Challenge on Kaggle tasked over 5,000 participants with improving reasoning accuracy on Nemotron-3-Nano-30B using restricted LoRA adapters on Google Cloud G4 VMs. Top-performing competitors approached reasoning as an end-to-end engineering workflow by auditing synthetic chain-of-thought traces with programmatic solvers before training. Successful teams also compressed reasoning representations into compact signatures and bit-manipulation formats to prevent long chains from exceeding runtime token generation limits. Additionally, leading approaches separated precomputed reusable structures from live computation and conducted granular per-category validation to monitor regression and non-determinism. These results show that reasoning improvements stem from verifiable training data, compact context encoding, and structured validation rather than unconstrained data scaling.
From weeks to a day: how we made LLM evaluation fast enough to iterate on
Airbnb built a four-layer LLM infrastructure framework to reduce iteration and evaluation turnaround from weeks to a single day. By using per-sample caching for generated references and judge scores, they established a deterministic evaluation foundation that separates genuine model drift from measurement noise. This setup enables rapid, bounded model hotfixes using micro LoRA adapters and comprehensive end-to-end validation across component boundaries.
How to Run an Autoresearch Workflow with RL Agent Skills and NVIDIA NeMo
Setting up reinforcement learning workflows and running repetitive experimentation often consumes significant engineering time before meaningful metrics emerge. To address this, an autonomous autoresearch workflow uses a frontier coding agent running Codex with GPT 5.5 to manage NVIDIA NeMo RL and NeMo Gym on an NVIDIA Brev GPU instance. The agent executes full-stack setup, creates a novel NeMo Gym visual counting environment, tunes Qwen3-VL-2B-Instruct accuracy from 25.0% to 96.9%, and implements an off-policy reinforcement learning algorithm directly from a research paper. Reusable skills, including brev-etiquette, session-memory, and autoresearch, enforce local conventions and prevent context drift during long executions. This setup offloads environment configuration, dependency resolution, and experiment tracking while preserving human oversight for goals and strategy.
Post-Train NVIDIA Cosmos 3 in One Day Using Agent Skills
Adapting vision reasoning models for specialized physical AI video tasks often demands days of manual container configuration, training script authoring, and hyperparameter tuning. NVIDIA Cosmos 3 Nano, an omnimodal mixture-of-transformers model, pairs an autoregressive Reasoner tower with an iterative diffusion Generator tower for video question answering. By integrating NVIDIA TAO agent skills with a coding agent like Codex, developers can automate data patching, evaluation, and Low-Rank Adaptation (LoRA) post-training. In experiments using the Woven Traffic Safety dataset, LoRA fine-tuning boosted zero-shot baseline accuracy from 54.41% to 87.14% while using approximately seven times fewer GPU hours than full-parameter supervised fine-tuning. Combining this agent workflow with TAO AutoML further elevated model accuracy to 93.35% before automated deployment via NVIDIA NIM.
Reducing High-Bandwidth Memory Bottlenecks in JAX-Based LLM Training with Host Offloading
Large language model training workloads frequently face GPU high-bandwidth memory (HBM) capacity constraints before compute limits are reached. To alleviate this memory pressure in JAX training workloads, host offloading transfers selected forward-pass activations to pinned host memory over high-speed NVLink-C2C interconnects and reloads them during the backward pass instead of recomputing them. Benchmarks executed on 128 GPUs within NVIDIA GB200 NVL72 systems using MaxText demonstrated significant improvements. For DeepSeek-V3 671B, host offloading combined with the Latency Hiding Scheduler and pipelined offloading reached 908.2 TFLOPs/s/device, surpassing activation rematerialization throughput by 57% and unlocking larger batch configurations. Similarly, QKV activation offloading on Llama 3.1 405B achieved a 2.9% throughput increase when transfer latency was hidden behind computation.
AI Model Co-Design: Hardware-Friendly LLM Design
Efficient large language model serving requires co-designing architectures to balance datacenter token throughput and user interactivity within hardware constraints. Roofline analysis shows that linear layers become memory-bound when hidden or intermediate projection dimensions are small, leaving arithmetic hardware underutilized even at large token counts. For prefill workloads, Chunked Pipeline Parallelism reduces first-token latency on long contexts when models use regular, repeatable layer patterns that split into balanced stages. For latency-critical decoding, decoupling attention and feed-forward network parallelization enables independent optimization, applying expert parallelism for feed-forward networks and Helix parallelism to shard the key-value cache across sequences. Following architectural design rules—such as dimension alignment, width prioritization, and low-precision NVFP4 compatibility—allows models to achieve higher utilization on modern accelerator hardware.
Better tools made Copilot code review worse. Here’s how we actually improved it.
When migrating GitHub Copilot code review from custom navigation tools to shared Unix-style CLI utilities—grep, glob, and view—benchmarks revealed higher review costs and fewer caught issues. Rather than an issue with the underlying tools, trace analysis showed the agent used general-purpose coding assistant instructions that triggered expansive repository browsing loops. In response, the team rewrote the tool guidance to enforce a review-specific workflow anchored to pull request diffs. The updated instructions direct the agent to narrow candidate call sites and files using batched discovery before reading minimal line ranges with view. This workflow tuning reduced average review costs by approximately 20% in production while maintaining review quality.
Synthetic Data Generation for Financial AI Research with NVIDIA NeMo
Real-world financial natural language processing datasets often overrepresent common events like earnings while underrepresenting rarer occurrences such as credit-rating changes. To address this imbalance, an iterative generation pipeline produced over 500,000 unique headlines across 13 categories using NVIDIA NeMo Data Designer, NeMo Curator, and Nemotron 3 Nano. Instead of relying on a single large batch that yielded 65 percent near-duplicates, the workflow executes iterative cycles of category-weighted generation, rule-based filtering, global semantic deduplication, and dynamic category reweighting. Farthest-from-centroid few-shot example selection guided subsequent rounds toward novel semantic spaces across 82 total iterations. Executed over approximately six days on an eight-way NVIDIA B200 node, the resulting corpus enabled fine-tuning compact student language models to achieve 95 percent of teacher model performance.
amazon.science
Capturing token IDs during agentic interactions for better reinforcement learning
Reinforcement learning for agentic tasks requires updating model parameters against the precise token sequences generated during rollouts. Standard agent harnesses frequently modify conversation transcripts through message compaction and tool formatting, causing retokenization drift that degrades policy-gradient training. Turnstile addresses this issue as a Rust-based proxy situated between agent harnesses and inference backends such as SGLang. By intercepting OpenAI-compatible Chat Completions traffic, the proxy records exact token IDs, log probabilities, loss masks, weight versions, and mixture-of-experts routing traces at the moment of generation. Validation across text-based coding and multimodal computer-use agents demonstrated steady reinforcement learning improvements without requiring modifications to agent harnesses.
Automating cross-repo documentation with GitHub Agentic Workflows
Maintaining documentation across separate repositories often leads to severe lag because technical writers must reverse-engineer shipped features weeks after release. To address this in the Aspire project, the team implemented an automated pipeline using GitHub Agentic Workflows to bridge the product and documentation repositories. When product pull requests merge, a bash step maps milestones to docs release branches before an LLM agent evaluates the diff, drafts documentation updates, and emits structured pull request intents. A dedicated safe-outputs handler materializes these drafts via a scoped GitHub App and assigns the original code reviewers to verify accuracy. Across 396 product pull requests, the system generated 82 documentation pull requests that all merged with a median turnaround time of 44.8 hours.
Create a LangChain Deep Agents Harness Profile for NVIDIA Nemotron 3 Ultra to Improve Performance
The tutorial shows how to create and validate a LangChain Deep Agents harness profile for NVIDIA Nemotron 3 Ultra using an evaluation benchmark rather than fine-tuning the model. It starts with a baseline, analyzes failed trajectories, and iterates on prompts, exclusions, or added middleware while rerunning tests to detect regressions. One failure occurred when read_file returned the first page of a large file without signaling that more content existed, so the model answered before paginating to the line containing opal-fox-91. The proposed ReadFileContinuationNoticeMiddleware appends a continuation notice when a full page is returned, raising the reported benchmark score from 94/127 to 96/127 and resolving all three read_file tests. The source also describes automating this propose, verify, and retain-or-reject loop, with safeguards against overfitting and regressions.
Open, convenient and predictable: Introducing Provisioned Throughput
Companies adopting open-weight models for production workloads have faced a trade-off between convenient but best-effort serverless inference and powerful but complex dedicated GPU instances. Together AI’s Provisioned Throughput fills that gap with reserved inference capacity sold as Provisioned Throughput Units (PTUs). Each PTU is priced at $0.05 per minute and guarantees a fixed rate of tokens per minute for a specific model, with a 99% uptime SLA. Input, cached input, and output tokens draw down capacity at different rates, letting teams optimize for their traffic shape. Available now for MiniMax M3 and GLM-5.2 with capacity in North America and EMEA and a one-month minimum term, at full utilization costs can run roughly $0.36 per million input tokens and $2.16 per million output tokens, up to 90% below proprietary alternatives.
NVIDIA Vera CPU Boosts AI Factory Throughput to Accelerate Agentic Workloads
Agentic AI workflows and reinforcement learning rely heavily on sequential CPU tasks, such as tool calls, sandboxed code execution, data processing, and environment rollouts between GPU steps. When CPU performance degrades under full socket load, training time inflates, serving latency grows, and GPU KV-caches face eviction. The NVIDIA Vera CPU addresses these bottlenecks by featuring 88 Olympus cores on a monolithic compute die, paired with power-efficient LPDDR5x memory and the Scalable Coherency Fabric. These cores incorporate a neural branch predictor, a 10-wide decode front end, deep out-of-order execution, and NVIDIA Spatial Multithreading. Consequently, Vera achieves 1.8x faster core execution, 40% lower peak loaded latency compared to x86 CPUs, and up to 1.2 TB/s total memory bandwidth, preventing costly KV-cache recomputations.
Develop Humanoid Robot Policies End-to-End with NVIDIA Isaac GR00T
Humanoid robotics development suffers from fragmented tooling, siloed software ecosystems, and incompatible data formats that hinder reproducible workflows. NVIDIA introduced the Isaac GR00T Development Platform, an open-source, modular humanoid development environment that connects simulation setup, demonstration capture, policy training, evaluation, and hardware deployment. Central to this platform is the Isaac GR00T 1.7 vision-language-action model, which features a 3-billion-parameter base checkpoint licensed under Apache 2.0. The architecture adopts a Cosmos-Reason2-2B backbone and supports full pipeline export to ONNX and TensorRT for deployment on Jetson Thor. In a reference pick-and-place manipulation task, developers capture teleoperated demonstrations with Isaac Teleop, convert trajectories into LeRobot format, post-train GR00T 1.7, and evaluate the closed-loop policy inside Isaac Lab-Arena.
Building an Analysis AI Agent for Industrial Alarm Management with NVIDIA Nemotron
Industrial machinery produces high volumes of alarms and sensor readings that exceed manual triage capacity. To address this bottleneck, an end-to-end GPU-accelerated AI agent is designed using NVIDIA NeMo libraries, Nemotron open models, and the NVIDIA OpenShell runtime behind a single HTTP endpoint. When triggered with an alarm payload and sensor frame, the agent gathers historical context using cuDF, cuVS, and NeMo Retriever, queries data warehouses via Text-2-SQL, and executes specialist signal checks with tools like cuFFT and cuML. Nemotron 3 Nano manages orchestration tasks while Nemotron 3 Super synthesizes the gathered evidence. The workflow outputs a structured package containing an observation, root-cause hypothesis, remedy, and recommended action within a seconds-long latency budget.
Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism
Large-scale LLM training jobs spanning thousands of GPUs frequently encounter transient hardware interruptions that bottleneck tightly coupled tensor parallel groups. Nonuniform Tensor Parallelism addresses this bottleneck by dynamically adapting the tensor parallelism degree within affected scale-up domains rather than dropping entire data replicas. When a GPU drops out, the system reconfigures the remaining functional devices to process larger shards and resumes execution from the latest checkpoint. To prevent the downsized replica from stalling global synchronization, advanced rack designs temporarily power-boost the active GPUs to increase clock frequencies and match baseline throughput. Overlapping the necessary tensor resharding with backward computation and parameter synchronization reduces overhead on healthy replicas to under one percent.
Introducing AI dashboard and workflow generation in ClickStack
ClickStack now supports AI-generated dashboards and linked investigative workflows from natural-language prompts, targeting the effort required to understand telemetry, write queries, choose visualizations, and iterate. Rather than asking an LLM to return a finished artifact, ClickStack uses an AI Notebook with the same MCP tools available to external agents, where queries, visualizations, findings, and reasoning are captured in a persistent, editable investigation. The system explores telemetry, validates assumptions, assembles dashboards, and can connect overview and service-level dashboards through dashboard actions; generated charts remain editable. The same MCP server exposes these capabilities to agents such as Claude, Cursor, and Codex, so users can create, refine, and extend dashboards through one toolchain.
Hardware-Rooted AI Security That Won’t Slow You Down
NVIDIA Confidential Computing secures enterprise data, proprietary model weights, and the model itself during active AI inference. The architecture integrates silicon-level private signing keys fused during manufacturing, encrypted multi-GPU NVLink, and remote attestation through the NVIDIA Remote Attestation Service alongside CPU Trusted Execution Environments. To mitigate overhead from secure work submission latency and encrypted host-to-device transfers, software optimizations incorporate CC-safe autotuner timing in FlashInfer, asynchronous copy workers in SGLang, and piecewise CUDA graph replay. Benchmarks on an HGX B300 system running the Qwen 3.5 397B-A17B model at FP8 precision show inference performance reaches up to 98% of non-confidential baselines across tested concurrency levels.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.