Activation Steering: One Dial, Two Costs
Activation steering edits a model's internal activations to add or remove a concept at inference time. This explainer maps its one control knob, its two costs, and when it beats prompting.
Loading…
Topic
LLMs
658 posts about LLMs. Every summary links to the original.
- 658 posts
- 49 companies
- 5 explainers
See the idea
LLMs explainers
Antibodies as a Prediction Pipeline: Where AI Models Break Down
Antibody discovery is a three-stage prediction pipeline, and AI models that excel on familiar targets degrade sharply on unfamiliar ones. This explainer maps where that gap appears and what it costs.
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Choosing a low-latency infrastructure layer for conversational AI
Conversational AI call-center latency can feel like dead air, and the delay may come less from LLM inference than from the infrastructure moving audio among telephony, media servers, ASR, the agent, and TTS. The described approach uses a unified voice path: ConversationRelay provides a WebSocket API to stream live call audio, handle speech-to-text and text-to-speech in one connected pipeline, connect with a backend agent, and support barge-in. It is model-agnostic, allowing teams to retain their LLM, knowledge bases, AI tools, and third-party voice providers, while Conversation Orchestrator, Conversation Memory, and Agent Connect address context and integration. Twilio’s internal benchmarks report median latency below 0.5 second and below 0.725 second at the 95th percentile, though relevance and context also matter alongside speed.
How Agentforce Achieves 100% Deterministic Rendering for AI Agent UX
Agentforce faced a production problem: LLM-driven rendering could recognize a required form or regulated disclosure yet choose a different output because its reasoning was stochastic. The Agentforce Connections layer addresses this by separating probabilistic reasoning and orchestration from deterministic presentation: builders associate an action with a connection-specific response format through the render: directive, and the platform applies that format instead of asking the LLM to select a component. Multi-action, long-running, and supervised flows required the system to identify the configured action and render only after approval and execution. The same abstraction supports Lightning Web Components, third-party interfaces, and headless structured JSON outputs. Per-action, per-session event logs record the action, format, connection, and conversation point, providing auditable evidence that required rendering occurred.
Speeding up LLM inference with P-EAGLE in vLLM Speculators
P-EAGLE (Parallel EAGLE), implemented in Speculators v0.6.0, extends EAGLE-3 with parallel drafting so a draft model predicts multiple future tokens in one forward pass rather than sequentially. Speculative decoding pairs this small draft model with a verifier model, which checks the proposed sequence in parallel and accepts only its correct prefix while preserving the verifier’s output distribution. The post adds Conditional Drop-token (COD) sampling, learnable mask embeddings and hidden states, and a custom flex-attention mask to support multi-depth prediction within the EAGLE-3 framework. The post presents an end-to-end workflow covering data preparation, vLLM hidden-state extraction, training and deployment, including a Qwen3-8B example that takes about 50 minutes on four H100s. A published Qwen3-a8B draft model reports an average length of 3.044 across the listed datasets.
Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps
This guide presents a low-cost recipe for improving structured-output compliance in Liquid AI’s LFM2.5-350M, a task where parseable, schema-adherent responses determine whether an LLM can feed downstream systems. It fine-tunes the model with Group Relative Policy Optimization (GRPO) through TRL, using about 500 samples, 100 steps, three weighted rewards for format, field count, and JSON Schema validation, and a LoRA adapter training roughly 6M parameters. Prompt augmentation covers fenced-code-block instructions and top-level-array tasks, while evaluation uses IFStruct with llama.cpp on an OpenAI-compatible local server. On 2,000 IFStruct samples, the BF16 base model passes 22.6%, rising to 29.7% after tuning. JSON improves from 18.0% to 31.9%, bare-list compliance from 16.6% to 29.7%, while YAML changes little.
How we make AI coding more cost efficient without sacrificing task quality
GitHub Copilot’s AI coding efficiency work focuses on completed task cost rather than tokens in individual tool calls, because shorter responses can trigger recovery work. Using agentic coding benchmarks and controlled online experiments, the team evaluated selective output compression, removal of unused line-number prefixes, prompt shortening, and direct delivery of completed background results. The shipped compressor preserves source-like and arbitrary output, reorganizes search results without dropping content, and compresses repetitive install, build, test, and progress output while retaining a recovery path to originals. Removing line-number prefixes cut offline model-inference cost roughly 5% and online daily cost about 3%; direct background-result delivery reduced AI Credit usage about 2.3%, while prompt changes required workload-specific validation.
Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference
Speculative decoding accelerates large language model inference by having a smaller draft model propose multiple tokens and a target model verify them in parallel, preserving the standard output sequence under its acceptance criteria. It reduces decoding iterations and increases target-model arithmetic intensity, but speedup depends on acceptance length, verification time, draft latency, batch size, and hardware behavior. For attention-heavy workloads, the recommended starting point is D = 128/G - 1, where G is query heads per KV head; larger D values should align G × (1 + D) with 128-token kernel tiles. The remaining guidance covers compute-bound GEMMs, low-latency draft-cost tradeoffs, and choosing among external models, EAGLE-3, MTP, DFlash, DSpark, and suffix or n-gram methods; SPEED-Bench, TensorRT LLM, and NVIDIA/Model-Optimizer support measurement, training, and quantization workflows.
LLM quantization guide: How to do it, and how it helps
LLM quantization addresses the gap between rapidly growing model sizes and GPU memory by storing weights and activations at lower precision than BF16. It explains how quantization targets linear layers in transformer blocks, distinguishes weight quantization for memory-bound decoding from activation quantization for compute-bound prefill, and maps formats such as GGUF, FP8, NVFP4, AWQ-INT4, and MLX to their inference engines. Calibrated methods including GPTQ, AWQ, and SmoothQuant protect important weights using representative data, while reported NVFP4 results recover about 99% of BF16 accuracy for 70B–235B models and 95–98% for 7B–14B models. In a Llama 3 70B benchmark on two H100s, FP8 weights-and-activations increased throughput from 158 to 474 input tokens/sec and reduced time to first token under load from over 30,000ms to about 4,800ms.
BenchMIRT: What are LLM benchmarks actually measuring?
BenchMIRT is a method for auditing LLM benchmarks at the individual-prompt level, where questions may measure capabilities beyond a benchmark’s stated goal. It applies multidimensional Item Response Theory (MIRT) to model and question results, estimating model abilities plus each item’s difficulty and discrimination across dimensions. Trained on 100 LLMs, 16 benchmarks, and more than 34K questions, it independently recovered safety and general reasoning as dominant dimensions, repeatedly producing the same result. The analysis found, for example, that BBQ and WMDP aligned more strongly with reasoning than safety, while HarmBench combined safety- and reasoning-associated question groups; retaining 10% of items generally preserved capability rankings, and held-out correctness prediction reached 79% versus 70% for a baseline. BenchMIRT’s dimensions depend on the benchmark set, and its item-level transparency can also enable evaluation trimming.
The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents
The post presents data-agent-mnist, an open harness for measuring analytical agents on correctness, cost, and speed against a warehouse-shaped workload rather than a single text-to-SQL query. The authors extracted 201 production analytics questions from DWAINE, curated and anonymized them, reconstructed a deterministic synthetic warehouse with 18 tables and 865 columns, and evaluated 28 proprietary and open-weight models. The benchmark preserves schema discovery, multi-turn querying, planning, and multi-hop joins across curated marts and a dimensional CRM layer, with provider-diverse ground truth and judging described as part of the workflow. On the authors’ warehouse, Claude Fable 5.1 ranked first for correctness at 76.6%, while the full run cost $1 with DeepSeek V4 Flash versus $52 with Fable 5.1, an 11-percentage-point correctness difference. The harness is intended to identify the best production fit for a specific warehouse, not a universal winner.
How to Size GPUs for AI Inference and TCO Without Overspending
GPU sizing for AI inference depends on use case, token patterns, latency targets, concurrency, cache hit rate, model choice and deployment strategy rather than raw hardware specifications alone. The post proposes a core-and-flex capacity model, combining baseline on-prem or reserved cloud capacity with elastic spot or on-demand GPUs for surges, launches and experiments. It also recommends matching GPU memory to prompt lengths and concurrency, while using quantization, pruning and distillation to reduce footprint and operating cost. A NeMo example prunes Qwen3-8B from 36 to 24 layers or narrows its dimensions toward a roughly 6B student before teacher-student distillation. In the illustrative run, width pruning reached lower final validation loss than depth pruning, 3.21 versus 3.60, while depth pruning converged faster.
Developing LLM guardrail configs locally with NeMo Guardrails
The article presents a local workflow for developing and testing NeMo Guardrails configurations in a Jupyter notebook, avoiding repeated Kubernetes deployments and requiring no LLM, GPU, or cloud resources. It defines rails, configurations, and servers, then starts with a case-insensitive regex rail that blocks the exact phrase “ignore all previous instructions”; the test catches a direct attack but allows an accented “Ignóre” variant. A broader regex configuration is followed by a DeBERTa-based prompt-injection classifier and a manually assembled set of 13 prompts, producing 12 correct decisions out of 13, with one subtle attack allowed. The article recommends iterating locally, potentially layering targeted regexes before classifiers for fail-fast, defense-in-depth filtering, while noting that 13 prompts cannot meaningfully measure performance and pointing to standardized EvalHub evaluations as a next step.
From Leaderboards to Model Profiles: A Deep Dive Evaluation of LLMs for Agentic Coding
Resolve rate alone can hide how coding agents reach the same outcome. This evaluation compares Claude Opus 4.7, Gemini 3.5 Flash, GPT-5.5, and Qwen 3.6 27B FP8 through a Junie pipeline across four benchmark datasets, combining test results with trajectory, cost, efficiency, patch quality, and process measures. In one private benchmark, Opus and Gemini solved the same number of tasks, but Opus averaged 184 steps at USD 2.79 per run versus Gemini's 271 steps at USD 1.24. Across 523 tasks, Opus resolved 267 (51.1%) and Gemini 254 (48.6%); shared outcomes masked differences in navigation, validation, redundancy, hallucination, and patch scope. The wider comparison found GPT-5.5 had the highest resolve rate and always ran an executable check, Opus led patch-quality metrics, and Qwen delivered lower cost with lower resolution, while the authors caution that profiles depend on the Junie scaffold and LLM judges are diagnostic rather than ground truth.
What is an AI Copilot?
An AI copilot is an assistant embedded in a software application that uses workflow context to offer suggestions, generated content, or approved actions while keeping the human in control. Unlike a standalone chatbot, it combines an LLM with contextual grounding, retrieval-augmented generation (RAG), access to enterprise data, and action layers that can call APIs, execute code, trigger workflows, or update records. The explainer surveys code, productivity, data and analytics, customer-facing, and domain-specific copilots, including uses such as SQL generation, dashboard creation, email drafting, and case summarization. Reported benefits include faster routine work, broader data access, more consistent outputs, and less context switching; a cited GitHub study found Copilot users completed a controlled coding task 55% faster. It also stresses hallucinations, privacy, bias, overreliance, integration complexity, cost, and the requirement for human review, governance, and accountability.
Beyond answers: New Genie One features to turn insights into action
Genie One’s latest updates target the manual work that follows an AI-generated answer, including accuracy checks, report formatting, sharing, and repeated workflows. The release adds a native macOS desktop app in beta with a global launcher, persistent conversations across web and desktop, and access to documents, skills, scheduled tasks, Genie Agents, and Databricks Apps. Genie One can create editable documents with live visualizations, comments, version history, sharing, PDF export, and external-document editing through MCP writes. Users can also turn conversations into reusable, shareable Genie Agents or distribute read-only chat links containing questions, answers, visualizations, and context. Additional capabilities include ontology snippets, private uploads of supported files, and governed MCP actions, with Unity Catalog, source authentication, and Unity Gateway governing access across assets and connected tools.
The State of Django 2026: Boring is so back
The fifth annual Django Developers Survey, produced by Django Software Foundation and PyCharm, draws on responses from nearly 3,500 developers in more than 40 countries. It reports that Django’s core remains stable, with PostgreSQL used by 76–79% of respondents across five years, templates around 80%, and 43% already on Django 6.0. At the same time, AI is routine but fragmented: 58% of AI users work with it daily, while 56% use it only for chat and advice and just 27% use autonomous multi-step tasks. The survey also describes rapid consolidation around uv and Ruff, growing adoption of type hints, and a testing and CI/CD base that can verify agent-generated changes, although 19% write no automated tests. Django supports both server-rendered and API/frontend patterns, while its stable foundation lets developers change surrounding tools and workflows without replacing the framework.
GLM-5.3 vs. GLM-5.3 Flash on DeepSWE: Cost, Coding, and Routing
The GLM-5.3 Flash, a distilled variant of the full GLM-5.3, achieves a pass@1 of 63.4% on the DeepSWE benchmark at $0.24 per rollout, 17 times cheaper than the full model’s 69.0% at $3.99. The pass@4 gap narrows to 2.6 points (87.6% versus 85.0%). Distillation removed consistency rather than capability: the Flash solves 93 of the 99 tasks the full model solves, but its ability to convert extra effort into a win is lost (46% win rate on flaky tasks versus 61%). A cascade that runs Flash first and escalates to the full model only when the answer is rejected solves 80.9% of tasks at $1.70 each, 57% lower cost than using the full model alone.
LLMs Are Not (Consistently) Bayesian: Quantifying Internal (In)consistencies of LLMs’ Probabilistic Beliefs
LLMs are deployed in medicine, science, and law, where observed evidence may not determine a single correct answer and systems must represent and update uncertain beliefs. The work treats LLMs as information processing rules and measures an information processing gap, defined as deviation from Bayes updates, to quantify internal consistency in probabilistic belief updates. Extensive experiments compare multiple ways of incorporating evidence: some yield nearly Bayesian updates, while others rely on learned heuristics. Although exact Bayesian updating is optimal information processing, the non-Bayesian heuristic updates often achieve better downstream task performance, suggesting that the LLMs’ probabilistic models of the world are misspecified. The measure also serves as a diagnostic for issues in LLM-powered inferential systems.
Agent Seer: Synthesizing Scenarios from Specification Understanding
Agent Seer addresses the problem of creating realistic, scalable test scenarios for AI agents that use external tools, where hand-built cases require domain expertise and static benchmarks cannot track changing APIs. It synthesizes scenarios from a single Model Context Protocol (MCP) specification containing function names, natural-language descriptions, and typed parameter schemas, without examples, live tool access, or domain-specific tuning. The pipeline enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock-data-grounded, multi-turn dialogues, then evaluates tool-calling correctness and conversational coherence across seven MCP specifications spanning domains and tool-suite sizes. It achieves strong quality across all domains and complete tool coverage on small and medium specifications; parameter schema complexity correlates most strongly with quality variation, while tool-suite size has a smaller orthogonal role. Among imperfect scenarios, argument value accuracy is the dominant failure mode, which coarse-grained name-match metrics do not reveal.
Differential Privacy for Hugging Face Trainers – Without Rewriting Your Training Loop
Training large language models on sensitive data risks memorization and membership inference attacks, yet standard high-level transformer tools lack native differential privacy capabilities. JetBrains Research created and open-sourced DPTrainer, a library that integrates PyTorch's Opacus framework with the Hugging Face Trainer ecosystem. The library provides differential privacy through differentially private stochastic gradient descent without requiring manual training loops or source code modifications. DPTrainer extends base training arguments with a PrivacyArguments dataclass, managing per-sample gradient calculation, gradient clipping, noise injection, and Poisson-sampled data loading. A dynamic patching mechanism injects differential privacy into specialized subclasses like DPOTrainer and SFTTrainer, while built-in privacy accountants track the cumulative privacy budget across training steps and checkpoints.
The CLion Roadmap: What’s Coming Between Now and Late 2026
The CLion development roadmap outlines planned updates for minor releases and the stable 2026.3 release. Development priorities center on agentic workflows, embedded development support, and debugger enhancements. Planned AI features include agent-assisted project setup and a specialized skill for ARM Cortex-M HardFault debugging with register, memory, and disassembly access. Debugger enhancements introduce JSON-based debug profiles, import capabilities from tools such as VS Code, a dedicated Lauterbach TRACE32 profile, and LLDB 21 for Windows MSVC users. Additional additions span QNX project support, default resolve configurations for unlinked files, ISPC language support, bundled toolchain upgrades, and improvements to IDE responsiveness and Dev Container workflows.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.