Activation Steering: One Dial, Two Costs
Activation steering edits a model's internal activations to add or remove a concept at inference time. This explainer maps its one control knob, its two costs, and when it beats prompting.
Loading…
Topic
LLMs
673 posts about LLMs. Every summary links to the original.
- 673 posts
- 49 companies
- 5 explainers
See the idea
LLMs explainers
Antibodies as a Prediction Pipeline: Where AI Models Break Down
Antibody discovery is a three-stage prediction pipeline, and AI models that excel on familiar targets degrade sharply on unfamiliar ones. This explainer maps where that gap appears and what it costs.
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference
Speculative decoding accelerates large language model inference by having a smaller draft model propose multiple tokens and a target model verify them in parallel, preserving the standard output sequence under its acceptance criteria. It reduces decoding iterations and increases target-model arithmetic intensity, but speedup depends on acceptance length, verification time, draft latency, batch size, and hardware behavior. For attention-heavy workloads, the recommended starting point is D = 128/G - 1, where G is query heads per KV head; larger D values should align G × (1 + D) with 128-token kernel tiles. The remaining guidance covers compute-bound GEMMs, low-latency draft-cost tradeoffs, and choosing among external models, EAGLE-3, MTP, DFlash, DSpark, and suffix or n-gram methods; SPEED-Bench, TensorRT LLM, and NVIDIA/Model-Optimizer support measurement, training, and quantization workflows.
LLM quantization guide: How to do it, and how it helps
LLM quantization addresses the gap between rapidly growing model sizes and GPU memory by storing weights and activations at lower precision than BF16. It explains how quantization targets linear layers in transformer blocks, distinguishes weight quantization for memory-bound decoding from activation quantization for compute-bound prefill, and maps formats such as GGUF, FP8, NVFP4, AWQ-INT4, and MLX to their inference engines. Calibrated methods including GPTQ, AWQ, and SmoothQuant protect important weights using representative data, while reported NVFP4 results recover about 99% of BF16 accuracy for 70B–235B models and 95–98% for 7B–14B models. In a Llama 3 70B benchmark on two H100s, FP8 weights-and-activations increased throughput from 158 to 474 input tokens/sec and reduced time to first token under load from over 30,000ms to about 4,800ms.
BenchMIRT: What are LLM benchmarks actually measuring?
BenchMIRT is a method for auditing LLM benchmarks at the individual-prompt level, where questions may measure capabilities beyond a benchmark’s stated goal. It applies multidimensional Item Response Theory (MIRT) to model and question results, estimating model abilities plus each item’s difficulty and discrimination across dimensions. Trained on 100 LLMs, 16 benchmarks, and more than 34K questions, it independently recovered safety and general reasoning as dominant dimensions, repeatedly producing the same result. The analysis found, for example, that BBQ and WMDP aligned more strongly with reasoning than safety, while HarmBench combined safety- and reasoning-associated question groups; retaining 10% of items generally preserved capability rankings, and held-out correctness prediction reached 79% versus 70% for a baseline. BenchMIRT’s dimensions depend on the benchmark set, and its item-level transparency can also enable evaluation trimming.
The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents
The post presents data-agent-mnist, an open harness for measuring analytical agents on correctness, cost, and speed against a warehouse-shaped workload rather than a single text-to-SQL query. The authors extracted 201 production analytics questions from DWAINE, curated and anonymized them, reconstructed a deterministic synthetic warehouse with 18 tables and 865 columns, and evaluated 28 proprietary and open-weight models. The benchmark preserves schema discovery, multi-turn querying, planning, and multi-hop joins across curated marts and a dimensional CRM layer, with provider-diverse ground truth and judging described as part of the workflow. On the authors’ warehouse, Claude Fable 5.1 ranked first for correctness at 76.6%, while the full run cost $1 with DeepSeek V4 Flash versus $52 with Fable 5.1, an 11-percentage-point correctness difference. The harness is intended to identify the best production fit for a specific warehouse, not a universal winner.
How to Size GPUs for AI Inference and TCO Without Overspending
GPU sizing for AI inference depends on use case, token patterns, latency targets, concurrency, cache hit rate, model choice and deployment strategy rather than raw hardware specifications alone. The post proposes a core-and-flex capacity model, combining baseline on-prem or reserved cloud capacity with elastic spot or on-demand GPUs for surges, launches and experiments. It also recommends matching GPU memory to prompt lengths and concurrency, while using quantization, pruning and distillation to reduce footprint and operating cost. A NeMo example prunes Qwen3-8B from 36 to 24 layers or narrows its dimensions toward a roughly 6B student before teacher-student distillation. In the illustrative run, width pruning reached lower final validation loss than depth pruning, 3.21 versus 3.60, while depth pruning converged faster.
Developing LLM guardrail configs locally with NeMo Guardrails
The article presents a local workflow for developing and testing NeMo Guardrails configurations in a Jupyter notebook, avoiding repeated Kubernetes deployments and requiring no LLM, GPU, or cloud resources. It defines rails, configurations, and servers, then starts with a case-insensitive regex rail that blocks the exact phrase “ignore all previous instructions”; the test catches a direct attack but allows an accented “Ignóre” variant. A broader regex configuration is followed by a DeBERTa-based prompt-injection classifier and a manually assembled set of 13 prompts, producing 12 correct decisions out of 13, with one subtle attack allowed. The article recommends iterating locally, potentially layering targeted regexes before classifiers for fail-fast, defense-in-depth filtering, while noting that 13 prompts cannot meaningfully measure performance and pointing to standardized EvalHub evaluations as a next step.
From Leaderboards to Model Profiles: A Deep Dive Evaluation of LLMs for Agentic Coding
Resolve rate alone can hide how coding agents reach the same outcome. This evaluation compares Claude Opus 4.7, Gemini 3.5 Flash, GPT-5.5, and Qwen 3.6 27B FP8 through a Junie pipeline across four benchmark datasets, combining test results with trajectory, cost, efficiency, patch quality, and process measures. In one private benchmark, Opus and Gemini solved the same number of tasks, but Opus averaged 184 steps at USD 2.79 per run versus Gemini's 271 steps at USD 1.24. Across 523 tasks, Opus resolved 267 (51.1%) and Gemini 254 (48.6%); shared outcomes masked differences in navigation, validation, redundancy, hallucination, and patch scope. The wider comparison found GPT-5.5 had the highest resolve rate and always ran an executable check, Opus led patch-quality metrics, and Qwen delivered lower cost with lower resolution, while the authors caution that profiles depend on the Junie scaffold and LLM judges are diagnostic rather than ground truth.
What is an AI Copilot?
An AI copilot is an assistant embedded in a software application that uses workflow context to offer suggestions, generated content, or approved actions while keeping the human in control. Unlike a standalone chatbot, it combines an LLM with contextual grounding, retrieval-augmented generation (RAG), access to enterprise data, and action layers that can call APIs, execute code, trigger workflows, or update records. The explainer surveys code, productivity, data and analytics, customer-facing, and domain-specific copilots, including uses such as SQL generation, dashboard creation, email drafting, and case summarization. Reported benefits include faster routine work, broader data access, more consistent outputs, and less context switching; a cited GitHub study found Copilot users completed a controlled coding task 55% faster. It also stresses hallucinations, privacy, bias, overreliance, integration complexity, cost, and the requirement for human review, governance, and accountability.
Beyond answers: New Genie One features to turn insights into action
Genie One’s latest updates target the manual work that follows an AI-generated answer, including accuracy checks, report formatting, sharing, and repeated workflows. The release adds a native macOS desktop app in beta with a global launcher, persistent conversations across web and desktop, and access to documents, skills, scheduled tasks, Genie Agents, and Databricks Apps. Genie One can create editable documents with live visualizations, comments, version history, sharing, PDF export, and external-document editing through MCP writes. Users can also turn conversations into reusable, shareable Genie Agents or distribute read-only chat links containing questions, answers, visualizations, and context. Additional capabilities include ontology snippets, private uploads of supported files, and governed MCP actions, with Unity Catalog, source authentication, and Unity Gateway governing access across assets and connected tools.
The State of Django 2026: Boring is so back
The fifth annual Django Developers Survey, produced by Django Software Foundation and PyCharm, draws on responses from nearly 3,500 developers in more than 40 countries. It reports that Django’s core remains stable, with PostgreSQL used by 76–79% of respondents across five years, templates around 80%, and 43% already on Django 6.0. At the same time, AI is routine but fragmented: 58% of AI users work with it daily, while 56% use it only for chat and advice and just 27% use autonomous multi-step tasks. The survey also describes rapid consolidation around uv and Ruff, growing adoption of type hints, and a testing and CI/CD base that can verify agent-generated changes, although 19% write no automated tests. Django supports both server-rendered and API/frontend patterns, while its stable foundation lets developers change surrounding tools and workflows without replacing the framework.
GLM-5.3 vs. GLM-5.3 Flash on DeepSWE: Cost, Coding, and Routing
The GLM-5.3 Flash, a distilled variant of the full GLM-5.3, achieves a pass@1 of 63.4% on the DeepSWE benchmark at $0.24 per rollout, 17 times cheaper than the full model’s 69.0% at $3.99. The pass@4 gap narrows to 2.6 points (87.6% versus 85.0%). Distillation removed consistency rather than capability: the Flash solves 93 of the 99 tasks the full model solves, but its ability to convert extra effort into a win is lost (46% win rate on flaky tasks versus 61%). A cascade that runs Flash first and escalates to the full model only when the answer is rejected solves 80.9% of tasks at $1.70 each, 57% lower cost than using the full model alone.
LLMs Are Not (Consistently) Bayesian: Quantifying Internal (In)consistencies of LLMs’ Probabilistic Beliefs
LLMs are deployed in medicine, science, and law, where observed evidence may not determine a single correct answer and systems must represent and update uncertain beliefs. The work treats LLMs as information processing rules and measures an information processing gap, defined as deviation from Bayes updates, to quantify internal consistency in probabilistic belief updates. Extensive experiments compare multiple ways of incorporating evidence: some yield nearly Bayesian updates, while others rely on learned heuristics. Although exact Bayesian updating is optimal information processing, the non-Bayesian heuristic updates often achieve better downstream task performance, suggesting that the LLMs’ probabilistic models of the world are misspecified. The measure also serves as a diagnostic for issues in LLM-powered inferential systems.
Agent Seer: Synthesizing Scenarios from Specification Understanding
Agent Seer addresses the problem of creating realistic, scalable test scenarios for AI agents that use external tools, where hand-built cases require domain expertise and static benchmarks cannot track changing APIs. It synthesizes scenarios from a single Model Context Protocol (MCP) specification containing function names, natural-language descriptions, and typed parameter schemas, without examples, live tool access, or domain-specific tuning. The pipeline enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock-data-grounded, multi-turn dialogues, then evaluates tool-calling correctness and conversational coherence across seven MCP specifications spanning domains and tool-suite sizes. It achieves strong quality across all domains and complete tool coverage on small and medium specifications; parameter schema complexity correlates most strongly with quality variation, while tool-suite size has a smaller orthogonal role. Among imperfect scenarios, argument value accuracy is the dominant failure mode, which coarse-grained name-match metrics do not reveal.
Differential Privacy for Hugging Face Trainers – Without Rewriting Your Training Loop
Training large language models on sensitive data risks memorization and membership inference attacks, yet standard high-level transformer tools lack native differential privacy capabilities. JetBrains Research created and open-sourced DPTrainer, a library that integrates PyTorch's Opacus framework with the Hugging Face Trainer ecosystem. The library provides differential privacy through differentially private stochastic gradient descent without requiring manual training loops or source code modifications. DPTrainer extends base training arguments with a PrivacyArguments dataclass, managing per-sample gradient calculation, gradient clipping, noise injection, and Poisson-sampled data loading. A dynamic patching mechanism injects differential privacy into specialized subclasses like DPOTrainer and SFTTrainer, while built-in privacy accountants track the cumulative privacy budget across training steps and checkpoints.
The CLion Roadmap: What’s Coming Between Now and Late 2026
The CLion development roadmap outlines planned updates for minor releases and the stable 2026.3 release. Development priorities center on agentic workflows, embedded development support, and debugger enhancements. Planned AI features include agent-assisted project setup and a specialized skill for ARM Cortex-M HardFault debugging with register, memory, and disassembly access. Debugger enhancements introduce JSON-based debug profiles, import capabilities from tools such as VS Code, a dedicated Lauterbach TRACE32 profile, and LLDB 21 for Windows MSVC users. Additional additions span QNX project support, default resolve configurations for unlinked files, ISPC language support, bundled toolchain upgrades, and improvements to IDE responsiveness and Dev Container workflows.
llm-d flow control: Priority queuing for shared GPU inference
Shared GPU model pools allow platform teams to maximize utilization, but mixed workloads often contend for serving capacity with differing service requirements. Generally available in Red Hat AI Inference 3.5, llm-d flow control introduces explicit admission control prior to backend routing. An Endpoint Picker sits in the request path and evaluates active priority bands from highest to lowest integer values whenever pool saturation is detected. Saturation metrics combine per-replica queue depth and key-value cache pressure to determine when dispatch gates should engage. Once a priority band is admitted, tenant fairness policies distribute dispatch turns, and ordering rules select the next request to forward to a vLLM replica via Envoy.
From Preferences to Principles: Rubric-Based Alignment for Grounded Knowledge Answers
Designing reward signals for open-domain question answering remains challenging because high-quality answers must satisfy multiple distinct criteria that holistic scalar objectives fail to capture. To address this limitation, researchers developed a rubric-based reward framework that produces query-specific rubrics grounded in retrieved evidence and decomposed into specific quality dimensions. This structured reward setup provides fine-grained supervision during the post-training phase. Across the evaluation axes of composition, grounding, and instruction-following, the multi-dimensional grounded rubric method outperforms instruction-tuned baselines by 6.5% and flat rubric variants by 4%. Grounding rubrics in retrieved evidence bolsters factual support, while dimension decomposition enhances organization, coherence, and requirement adherence.
SKT partnership announcement
SK Telecom (SKT), Korea’s largest mobile operator, has become a commercial partner and strategic investor in Anthropic. The companies will develop a multilingual large language model customized for telecommunications applications such as customer service, marketing, sales, and interactive consumer experiences, with support planned for Korean, English, Japanese, Spanish, and more. Anthropic will fine-tune Claude using SKT’s telecommunications expertise: SKT experts will provide feedback on responses, and that feedback will help train industry-specific solutions. SKT has invested an additional $100 million in Anthropic, following an earlier investment by SK Telecom Venture Capital, while the companies describe the partnership as a way to promote AI innovation and develop safer, more reliable technology for telcos worldwide.
Introducing 100K context windows
Claude’s context window has expanded from 9K to 100K tokens, or about 75,000 words, enabling businesses to submit hundreds of pages and sustain conversations for hours or days. The announcement describes using the larger window to digest documents, synthesize information across multiple files, answer questions, and work with full codebases, with the claim that complex questions may perform substantially better than vector-search approaches. In a test, Claude-Instant processed the 72K-token text of The Great Gatsby after one line was altered, identified the change correctly, and responded in 22 seconds; 100K tokens is also described as roughly six hours of audio. The capability is available through the API, with examples including financial analysis, legal review, developer-documentation search, podcast summarization, and rapid prototyping.
Gallup scales real-time coaching for thousands with Amazon Bedrock
Gallup needed to centralize decades of proprietary workplace performance research to provide proactive, real-time coaching for organizational leaders directly within their daily workflows. To solve this challenge, Gallup developed Gallup AI inside its Gallup Access platform using Amazon Bedrock and Anthropic Claude models. The multi-tenant serverless architecture leverages Amazon Bedrock Knowledge Bases and Amazon Kendra for retrieval-augmented generation, while AWS Lambda with FastAPI powers real-time streaming responses. Amazon ElastiCache Serverless caches recent conversation history for sub-millisecond session retrieval, Amazon RDS for MySQL provides durable transactional storage, and Amazon Bedrock Guardrails enforce content safety. Since its June 2024 launch, the system has achieved sub-second time-to-first-byte streaming latency while prompt volume grew approximately sevenfold across thousands of organizations.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.