Activation Steering: One Dial, Two Costs
Activation steering edits a model's internal activations to add or remove a concept at inference time. This explainer maps its one control knob, its two costs, and when it beats prompting.
Loading…
Topic
LLMs
648 posts about LLMs. Every summary links to the original.
- 648 posts
- 49 companies
- 5 explainers
See the idea
LLMs explainers
Antibodies as a Prediction Pipeline: Where AI Models Break Down
Antibody discovery is a three-stage prediction pipeline, and AI models that excel on familiar targets degrade sharply on unfamiliar ones. This explainer maps where that gap appears and what it costs.
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How do you turn AI coding chaos into a repeatable playbook?
Software development dynamics are shifting as code generation costs decrease, requiring engineering organizations to adapt their leadership and processes. Snowflake addressed this transformation across its software lifecycle by first encouraging widespread, unrestricted experimentation with coding agents before formalizing successful practices. The engineering organization codified these practices into fourteen distinct AI design patterns, including strategies such as planning in markdown and isolating parallel agents. This structured approach, combined with dedicated focus weeks and a progress scale, allowed engineering teams to accelerate both inner and outer loops of development. Consequently, a team of three engineers leveraged coding agents to deliver a 40x improvement to Snowflake's query compiler while significantly shortening release validation times.
ClickHouse Agents is now available for Managed Postgres
ClickHouse Agents is now available for all Managed Postgres services in ClickHouse Cloud, extending its plain-English agent interface to operational data. The beta service, powered by Claude and built on LibreChat, combines an agent builder and chat interface with a sandboxed code interpreter; it is read-only by default and supports MCP-compatible systems. Agents can inspect Postgres schemas, run read-only queries, return the SQL used, and reason across Postgres and ClickHouse by querying each engine independently before merging results. Additional skills guide migration from Postgres to ClickHouse through discovery, access-pattern analysis, schema design, type mapping, CDC ingestion, and validation, while other workflows use metrics, Query Insights, and EXPLAIN to rank performance fixes. Planned extensions include scheduled recommendations and safe automated remediation, but the current release focuses on exploration, analysis, monitoring, and guided database work.
Claude Fable 5 and Claude Mythos 5
Claude Fable 5 and Claude Mythos 5 are presented as a joint launch of the same underlying model, with Fable 5 intended for general use and Mythos 5 initially reserved for selected cyberdefenders and infrastructure providers. Fable 5 uses conservative safeguards that route some sensitive queries to Claude Opus 4.8, while Mythos 5 lifts safeguards in selected areas through Project Glasswing and a planned trusted-access program. The post reports strong results in software engineering, knowledge work, vision, long-context tasks, cybersecurity, protein design, molecular biology, and genomics, including a reported 50-million-line Ruby migration completed in a day. Both models are priced at $10 per million input tokens and $50 per million output tokens, with Fable 5 available through the Claude API and staged subscription access.
Mastering Agentic Techniques: AI Agent Reinforcement Learning
Language model agents often fail across long-horizon workflows, repeat tool-call errors, and output invalid schemas when relying solely on prompting or retrieval. Reinforcement learning with verifiable rewards provides a practical training signal for domain-specific tasks by scoring model rollouts against deterministic verifiers such as schema checkers, unit tests, and execution sandboxes. Teams can implement group relative policy optimization to sample multiple trajectories per prompt and update policy weights based on relative performance within the group. Establishing environments with explicit harness, state, and verifier definitions allows agents to learn multi-step tool use, whereas static datasets fail to capture sequential dependencies. Inspecting checkpoint rollouts and testing against held-out tasks prevents reward hacking while continuously turning operational failures into verifiable benchmark environments.
Reading the agent traces is how you make the call your eval can't
Automated evaluations often miss unknown failure modes in AI agents because developers only write tests for anticipated errors. While building an itinerary agent for AI Engineer World's Fair, an open-weight model routed a speaker query to a tool returning only tracks, leading the model to hallucinate famous speakers and falsely claim the tool provided that data. Because a tool was executed, the trace appeared grounded until inspectable tool outputs revealed the discrepancy. To resolve this, the author retained the inexpensive model, corrected query routing, tightened system prompts against ungrounded entities, and added a custom entity-diffing eval in CI. Ultimately, directly reading agent telemetry traces alongside errors and logs is essential for diagnosing root causes and deciding trade-offs between model costs, prompt adjustments, and routing fixes.
Multi-token Residual Prediction
Multi-Token Residual Prediction (MRP) adapts multi-token prediction for diffusion language models by training a small module to predict the residual between adjacent denoising steps rather than the full next-step distribution. This residual target is easier to learn, allowing a tiny 3-layer transformer to accurately predict corrections across multiple steps. In the static regime, MRP enables lossless speculative decoding with up to 1.56× throughput in SGLang, or direct decoding with a small quality cost. In the dynamic regime, MRP recovers up to +16 accuracy points lost by aggressive low-threshold decoding through a remasking mechanism. The module is trained on a frozen backbone and composes cleanly with existing inference methods.
Why intent prediction needs more than an LLM
Traditional large language models rely on next-token prediction trained over broad text sequences, excelling at conversational synthesis and code generation while struggling with forecasting future user behavior. Predicting intent inherently requires modeling decision-making under conditions of uncertainty, a capability not directly produced by general text pretraining. Dedicated behavioral foundation models address this limitation by training representations directly on proprietary, sensitive behavioral data, such as anonymous browsing sessions and product identifiers. By learning base representations of behavior, these models generalize to unseen predictive tasks, such as optimizing ad campaigns for products absent from the original training dataset. Implementing these behavioral models also necessitates privacy-centric machine learning techniques, ranging from differential privacy concepts to homomorphic machine learning.
How to Govern Autonomous Agents in Enterprise AI Factories
Autonomous AI agents require governed environments to safely inspect code, execute tests, query enterprise systems, and handle sensitive business data. The NVIDIA Secure Agent Workspace Reference Design shifts agent execution away from user laptops to dedicated, single-user virtual machines managed on Red Hat OpenShift Virtualization or Microsoft Azure. Perimeter defenses enforce enterprise single sign-on access broker connections, default-deny network egress rules, and human approval gates for critical state-changing actions. Inside the workspace virtual machine, active sandboxing through NVIDIA OpenShell evaluates signed security policies before each tool invocation while credential proxies prevent direct access to raw secrets. GitOps workflows continuously reconcile infrastructure profiles and network configurations while all operational and security telemetry streams to centralized SIEM platforms in OCSF format.
Agents hate friction: early thoughts on building for agents
Designing software for LLMs and agents requires more than extending human developer experience: a polished text-first interface can still produce an AX that is merely functional. The post frames an agent as a combination of model, harness, context, and skills, with behavior varying across providers, models, and harnesses; it focuses attention on commonly used combinations while testing beyond them. In a test setup, agents receive goals and different amounts of context, and model-harness combinations are evaluated by how they reason, which tools they call, how many steps they take, and how consistently they repeat a task. The resulting design questions include friction, context and token efficiency, familiarity with established tools, and the trade-off between fewer steps and higher context or cost; the central conclusion is that agents may abandon a product when friction persists, making testing necessary to expose that behavior.
Deploy a Production-Ready NVIDIA AI-Q Blueprint on Oracle Cloud Infrastructure
The NVIDIA AI-Q Blueprint operates as an open-source reference architecture for long-horizon agents built upon LangChain Deep Agents and the NVIDIA NeMo Agent Toolkit. Deploying this multi-agent system on Oracle Cloud Infrastructure requires coordinating foundational cloud infrastructure with containerized application workloads across Kubernetes. The deployment methodology utilizes Terraform to provision network isolation, an enhanced OKE cluster, a flexible public load balancer, and an AES-256 encrypted OCI Vault. Helm manages the application layer on Kubernetes by pulling official images from the NGC registry to run a FastAPI agent backend, a Next.js web frontend, and PostgreSQL. This configuration establishes an extensible routing architecture where user queries route automatically between shallow search agents and deep planning sub-agents while maintaining clean infrastructure teardown commands.
Creating the NVIDIA Nemotron 3 Ultra NVFP4 Checkpoint with NVIDIA Model Optimizer
Moving large model weights efficiently requires quantization, but mapping weights to NVFP4's eight positive values risks severe precision loss or outlier-induced underflow. To address this, NVIDIA quantized the 550B Nemotron 3 Ultra model using NVIDIA Model Optimizer with a multi-precision strategy rather than quantizing all layers uniformly. The recipe applies four-over-six adaptive block scaling to Mixture of Experts routed weights, while assigning FP8 to shared experts and retaining BF16 for sensitive attention linears and embeddings. This process reduced the checkpoint size from 1,121 GB down to 352.3 GB and enabled single-checkpoint execution across Hopper using W4A16 and Blackwell using native W4A4. As a result, the model achieves up to 5.9x higher inference throughput than the GLM-5.1 754B FP4 model on decode-heavy workloads while matching BF16 benchmark accuracy.
How I hunt for vulnerabilities with AI
The author describes using AI-assisted code review to hunt for vulnerabilities in ClickHouse, a large C++ codebase, despite having limited bug-bounty experience. GitHub Copilot, Claude, ChatGPT, and Gemini models were used to inspect code, generate vulnerability hypotheses, explain unfamiliar subsystems, and identify older or newly introduced attack surfaces. Promising ideas were recorded, then investigated with Python scripts against a locally running ClickHouse Docker container, followed by manual review and reproduction to distinguish genuine findings from hallucinations and misconfigurations. The work produced a few real vulnerabilities that were reported to ClickHouse’s security team and bug bounty program, while the broader conclusion is that AI can accelerate exploration and validation but still requires persistent questioning and human verification.
Q&A: How KRAFTON Built PUBG Ally, a Co-Playable Character Powered by NVIDIA ACE
KRAFTON developed PUBG Ally, an on-device AI co-playable character for PUBG: BATTLEGROUNDS, using NVIDIA ACE tooling and models. Conventional non-playable characters rely on fixed dialogue, but PUBG Ally processes live game states and player voice input in real time. The interactive pipeline combines automatic speech recognition, a quantized Mistral-NeMo-Minitron-2B small language model, and a custom text-to-speech model running locally on client GPUs with at least 8GB of VRAM. To minimize latency, the architecture separates reactive gameplay into a behavior-tree System 1 layer while routing strategic decisions and speech generation to the language model. KRAFTON tuned the non-deterministic system across multi-layer evaluations and large-scale playtests with over a thousand players.
How we used DSPy to turn AI evaluations into better responses in Dash chat
Dropbox improved its Dash chat agent by establishing an automated optimization loop powered by DSPy and LLM-as-judge evaluations. Engineers first calibrated their LLM judges against human-annotated interaction traces, then used those judges to systematically optimize the agent's system prompts via offline counterfactual replay. This automated workflow reduced incomplete responses by 26% while decreasing overall token consumption.
Modal's serverless Servers
Modal introduces Servers, a low-latency HTTP serving solution designed for applications like LLM inference where milliseconds matter. Servers reduce p50 latency from 39ms to 6ms compared to Modal Web Functions by removing control-plane lookups and queuing from the hot path. The architecture uses a stateless proxy (fprs) for domain association and load balancing, with Envoy for TLS termination and HTTP/2 multiplexing. Two design principles governed development: maximizing resource sharing while minimizing interference, and making no network calls on the request path. The system supports autoscaling, regionalized routing, proxy authentication, and mirroring for A/B testing and continual learning.
How Visa went from multi-day reporting to conversational analytics agents with ClickHouse Cloud and LibreChat
Visa used ClickHouse Cloud to modernize reporting on Authorize.net payments data, replacing multi-day analyst-led turnaround with natural-language queries for internal teams. The solution pairs LibreChat, an open-source conversational interface, with the ClickHouse MCP server and runs on ClickHouse Cloud on AWS, allowing agents to explore and query a roughly 40 PB data footprint containing millions of rows. ClickHouse materialized views pre-compute and incrementally update results, reducing scanned data and bringing example analyses down to 100–200 milliseconds, with reported speedups of roughly 30–70 times. The platform also uses encrypted transfers, Amazon S3, VPC PrivateLink, change data capture, and federated access controls, while Visa reports identifying millions of dollars in revenue risk and reclaiming 8–10 hours per user each week.
Achieve state-of-the-art inference latencies with speculative decoding
Modal launched Auto Endpoints, combining speculative decoding with Blackwell GPUs, SGLang engine, and Modal Servers to achieve low inference latency for production deployments. The decode phase, where each token requires a full forward pass loading model weights, is the primary latency source, making speculative decoding the most impactful optimization. Speculative decoding runs a separate DFlash speculator model in parallel with the target model using KV projections and a custom Triton kernel, achieving near-linear speedup proportional to acceptance length. In the Decagon voice case study, Modal cut 100ms from a 290ms baseline by mid-training a custom speculator on synthetic data, beating proprietary providers by 60ms. Modal plans to automate speculator training with continuous improvement on production data, moving toward autospec for Auto Endpoints.
Announcing the Managed ClickStack MCP Server
Managed ClickStack in ClickHouse Cloud now includes the ClickStack MCP server, extending its observability investigation capabilities to external agents and AI workflows. Unlike the generic ClickHouse MCP server’s SQL-oriented interface, it offers semantic tools for log patterns, outlier correlation, slow traces, and repeatable investigations across logs, metrics, and traces. For event patterns, the server samples matching events, processes them with Drain3 in the MCP layer, and extrapolates and ranks counts, reducing output cardinality and making query overhead fixed rather than linear in the full dataset. Internal benchmarks report 25% fewer tool calls, 2.5x greater consistency, and an almost 20% increase in evaluation scores versus the standard ClickHouse MCP. Managed deployments also support full-fidelity telemetry, dedicated compute for agent workloads, and orchestration such as persisted dashboards, while native SQL remains available for unusual or deeper questions.
Maximize AI Factory Energy Efficiency Through Full-Stack Inference and Training Optimizations
Power costs comprise up to 40% of AI factory operating expenses, making performance per watt critical within fixed site power limits. Operators maximize throughput by combining architectural enhancements, narrow-precision formats like NVFP4, and mixture-of-experts model designs that activate fewer parameters per token. During distributed LLM training in Megatron-LM, tuning individual GPU speeds along critical execution paths eliminates idle power waste without extending total training time. Additionally, the NVIDIA DSX platform orchestrates dynamic power allocation, 45°C direct-to-chip liquid cooling, and grid integration across racks and facilities. Together, these full-stack optimizations enable performance-optimized AI factories to deliver up to 2.6 times more tokens per second per megawatt.
Build Trust in Agentic AI: From POC to Production
Transitioning autonomous agentic AI systems from sandbox environments to enterprise production exposes a trust gap driven by hallucination, reasoning errors, and financial liability. To mitigate these risks, organizations can implement a proactive four-step framework covering foundations, verification, governance, and business outcomes. The architecture grounds agents using retrieval-augmented generation, short-term memory, long-term procedural memory, and observability traces consolidated on a unified data layer. Active verification computes an Agent Confidence Score alongside a deterministic Business Risk Score to produce an Agent Decision Score governing autonomy levels. Aggregated execution telemetry feeds business observability dashboards that monitor unit economics and enable analytical agents to recommend operational interventions.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.