Activation Steering: One Dial, Two Costs
Activation steering edits a model's internal activations to add or remove a concept at inference time. This explainer maps its one control knob, its two costs, and when it beats prompting.
Loading…
Topic
LLMs
648 posts about LLMs. Every summary links to the original.
- 648 posts
- 49 companies
- 5 explainers
See the idea
LLMs explainers
Antibodies as a Prediction Pipeline: Where AI Models Break Down
Antibody discovery is a three-stage prediction pipeline, and AI models that excel on familiar targets degrade sharply on unfamiliar ones. This explainer maps where that gap appears and what it costs.
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding
Sequential token generation in autoregressive LLMs limits GPU utilization and serving throughput, while conventional speculative decoding remains constrained by autoregressive draft models. DFlash addresses this bottleneck by employing a lightweight block-diffusion drafter that predicts an entire block of masked candidate tokens in a single forward pass. The architecture incorporates target hidden-state conditioning and key-value injection into draft model projections while relying on the target model for parallel verification. Evaluated on NVIDIA Blackwell GPUs across TensorRT-LLM, vLLM, and SGLang, DFlash increases gpt-oss-120b throughput by more than 15x at high interactivity targets and delivers up to 5.8x speedups on Gemma 4 31B compared to autoregressive decoding.
Your AI shipped a backend that boots. That is the whole problem.
Backend security vulnerabilities often emerge from permissive default settings in web frameworks, a risk amplified when AI agents generate code focused solely on passing tests. Common failure points include unmetered request body parsing, unvalidated schemas, prototype pollution, and wildcard cross-origin resource sharing policies. To eliminate these foot-guns, the TypeScript framework DaloyJS enforces secure-by-default behavior across its entire routing and middleware model. It provides built-in request timeouts, streamed body size caps, prototype-stripping JSON revivers, and unified Zod schemas that simultaneously validate payloads and generate OpenAPI contracts. While secure defaults lower the risk of common infrastructure exploits, developers still remain responsible for application-level authorization and business logic correctness.
The 2026 Developer Survey is now open (for human developers only)!
The 2026 Developer Survey is open to gather data on software builders' tools, working lives, and emerging practices. Marking fifteen years since its initial 2011 release, the annual survey focuses heavily on the role of artificial intelligence and autonomous agents across the software development lifecycle. Past survey results indicated that while developer adoption of AI and agents doubled, user trust declined alongside escalating concerns regarding generated code quality. In addition to modern AI trends and financial return considerations, the questionnaire maintains traditional inquiries regarding day-to-day technologies and developer workflows. The survey is hosted on the Qualtrics platform, which requires users to pause third-party ad blockers and adheres to regional access restrictions across multiple countries.
Build an AI Scientist for Life Science Discovery with NVIDIA BioNeMo Agent Toolkit
General-purpose AI agents often struggle with biomolecular discovery because scientific workflows require precise model selection, strict input formatting, and artifact interpretation. The NVIDIA BioNeMo Agent Toolkit bridges this gap by providing BioNeMo Skills and Model Context Protocol wrappers on top of accelerated NVIDIA NIM microservices. These skills instruct agents on tool capabilities, required inputs, expected file formats like CIF and SMILES, and specific failure modes. Developers can route requests through hosted endpoints for testing or deploy local NIM microservices on GPU nodes to lower warm per-call latency during iterative candidate generation loops. Internal benchmarks using Codex CLI demonstrate that equipping agents with BioNeMo Skills improves task completion rates from 57.1% to 100% while doubling token efficiency across passing assertions.
End-to-End RAG Workflow: How Retrieval Augmented Generation Works
Retrieval Augmented Generation (RAG) connects a large language model to external knowledge at inference time, addressing outdated training data and access to proprietary or real-time information. The workflow ingests and normalizes sources, splits documents into chunks, embeds them in a vector store, retrieves context, assembles a prompt, and generates an answer. Semantic search can be combined with BM25 keyword search through reciprocal rank fusion, while reranking can improve precision; the same embedding model must be used during ingestion and querying. The guide presents evaluation and deployment considerations, including separate measurement of retrieval precision and generation faithfulness, versioning, monitoring, and containerized components. It identifies poor retrieval as the most common failure mode and explains that RAG reduces, but does not eliminate, hallucinations.
How Telcos Build Autonomous Networks with Agentic AI
Telecom operators currently operate mostly within Levels 2 and 3 of the TM Forum autonomous networks taxonomy, relying on predefined scripts. Progressing to Levels 4 and 5 demands autonomous systems capable of intent comprehension, real-time telemetry sensing, and cross-domain action coordination. To address this, an autonomy platform architecture combines domain-specific reasoning models, agent harnesses, and isolated sandboxed runtimes. Workflows separate responsibilities among on-demand agents, long-running execution loops, and deep-research multi-agent systems that evaluate ranked remediation paths or design wireless algorithms. Initial experiments in simulated SR-MPLS routing and wireless physical or MAC layer design show autonomous discovery can deliver measurable gains over standard operational and engineering baselines.
Introducing Modal Auto Endpoints: Optimized inference you actually own
Modal Auto Endpoints is a self-serve service for production-grade LLM inference that gives teams ownership of the entire serving stack, not just an API. Unlike proprietary providers, it exposes the inference code, engine flags, and GPU selection, and provides detailed metrics like speculative decoding acceptance length and per-replica token latency quantiles in a dashboard. Built on Modal’s infrastructure, it uses autoscaling to handle traffic spikes by spinning up additional replicas automatically, global GPU capacity without capacity management, and Modal Servers for ultra-low-latency routing with only 5ms overhead. For each supported model, it offers a pre-configured deployment using speculative decoding with the DFlash block-diffusion drafter from Z Lab, and teams can benchmark latency and throughput tradeoffs with a click after deployment. The platform is designed with a declarative interface based on workloads and SLOs, with future automation levels including autoinference, autospec, autodistill, and autoresearch for continuous improvement of inference performance.
Shipping huggingface_hub every week with AI, open tools, and a human in the loop
huggingface_hub, the Python client underlying libraries such as transformers, datasets, diffusers, and sentence-transformers, moved from releases every four to six weeks to a weekly cadence. A single manually triggered GitHub Actions workflow now computes versions, manages branches, tags and publishes both huggingface_hub and the hf CLI, creates downstream RC test branches, and handles post-release tasks. An open-weights model drafts structured release notes and Slack announcements using PR metadata and documentation diffs, while OpenCode, HF Inference Providers, PyPI Trusted Publishing, and pinned, checksum-verified tooling provide the stack. Deterministic manifest validation checks missing or extra PR references, then a human reviews and publishes the drafts. The reported result is weekly shipping at about $0.25 per full release, with more consistent notes, earlier downstream breakage detection, and shorter contributor feedback loops.
What's new in ClickStack - May 2026
ClickStack’s May and early-June release focused on making observability dashboards more useful during investigations and expanding access to related workflows. Dashboard table actions can open searches against selected sources with templated WHERE clauses or link to other dashboards, while scoped filters prevent source-specific conditions from affecting unrelated tiles. Number visualizations gained fixed or threshold-based colors and automatic text scaling, and the service map added latency percentiles, throughput metrics, server-side filtering, and focus mode. The release also includes a Browser RUM dashboard template, experimental PromQL support through the ClickHouse TimeSeries Engine, expanded MCP tools, AI Notebooks beta for Managed ClickStack, and ClickStack Cloud in private preview.
Measuring What Matters with Jules
Proactive coding agents need evaluation beyond task completion because existing public benchmarks such as SWE-Bench do not measure goal-oriented insight policies: deciding what matters, supporting evidence, and whether to interrupt or stay silent. To create preliminary ground truth, researchers clustered 705 bugs and 1,178 CLs from internal Google codebases using temporal proximity and semantic similarity, reverted each codebase to its pre-fix state, and gave the agent up to three exploration rounds. An LLM scored predicted insights from 1 to 5 against clustered bug targets, while success used average top score and Hit@K. With one exploration round, the agent averaged 4.5 out of 5; increasing the budget from two to three rounds raised Hit@5 from 33% to 57%, although the authors describe the findings as preliminary and are expanding the evaluation to public GitHub data and richer context streams.
We got local models to triage the OpenClaw repo for FREE!*
OpenClaw receives hundreds of issues and pull requests daily, creating a triage problem for maintainers who need near-real-time filtering without consuming a paid closed-model quota. The system uses Gemma and Qwen local models inside a Pi-based agent harness: each item is normalized, enriched with GitHub context, classified into finite topic labels, and required to return structured output. For pull requests, the agent can inspect the repository through reposhell, a read-only command-restricted shell, while localpager, SQLite, a worker queue, and deterministic notification policy handle orchestration and Discord delivery. Tests described for gemma-4-26b-a4b and qwen3.6-35b-a3b report hundreds of tokens per second and good zero-shot classification accuracy without fine-tuning, while an LLM audit loop monitors errors; the authors present the pattern as agentic classification for high-throughput triage.
Dispatches from O'Reilly: From capabilities to responsibilities
High-stakes AI agents capable of mutating external state often face governance failures when relying on system prompts or manual Human-in-the-Loop approval queues that quickly degrade into alert fatigue. The Responsibility-Oriented Agent architecture addresses this operational bottleneck by shifting system design from open-ended capability framing to deterministic, contract-enforced responsibilities. Under this model, underlying orchestration frameworks like LangChain operate in User Space with their side-effecting tools removed, isolating the agent to epistemic reasoning. The agent expresses its intended action exclusively by emitting a structured policy proposal to a privileged Kernel Space runtime. The runtime deterministically evaluates the proposal against versioned YAML contracts registered in an agent registry, ensuring that only genuine policy exceptions are escalated to human supervisors.
Speculation Is All You Need
Speculative decoding is presented as the most important optimization for LLM inference, delivering speedups of 2x-3x rather than small percentage gains from kernel tuning. The technique accelerates the decode phase by using a lightweight draft model to generate candidate tokens that the target model processes in parallel, with lossless resampling. The authors released DFlash speculators for multiple Qwen models, achieving an additional 5-20% speedup over existing baselines, and worked with SGLang to integrate them. On a B200 node, the Qwen 3.5 122B-A10B model with speculation reaches over 1000 tok/s at concurrency 1, compared to 250 tok/s without speculation, demonstrating a 4x speedup. Training these speculators is simplified because the data-generating process is the target model itself, making data collection trivial, and the post envisions a flywheel where custom speculators and distillation enable continuous improvement in latency, cost, and quality for self-hosted inference.
Why everyone is talking about real-time analytics (a note from “the yellow company”)
Real-time analytics is presented as a foundational workload for customer-facing analytics, observability, fraud detection, operational applications, and AI agents rather than a niche use case. The discussion argues that performance claims require open, transparent, reproducible benchmarks that disclose methodology, datasets, queries, results, and operating conditions, including continuous ingestion, concurrency, memory pressure, and fresh data. A real-time database should support queryable data within milliseconds to a couple of seconds, incremental transformations, bounded tail latency as writes increase, configurable read-time versus write-time work, unified support for text, vector, JSON, and structured analytics, and efficiency as workload scales. The article connects renewed attention to agentic workloads, whose continuous, high-volume querying makes low latency, concurrency, and fresh context increasingly necessary, and calls for shared standards beyond a single latency number.
MosaicLeaks: Can your research agent keep a secret?
MosaicLeaks examines privacy leakage in deep-research agents that combine private enterprise documents with external web retrieval. Queries that appear harmless individually can reveal sensitive facts when an observer combines them over multiple hops, producing intent, answer, or full-information leakage. The benchmark contains 1,001 multi-hop chains interleaving local and web questions, and compares baseline behavior with task-only reinforcement learning and Privacy-Aware Deep Research (PA-DR). Task-only training raises strict chain success from 48.7% to 59.3% but increases answer/full-information leakage from 34.0% to 51.7%, whereas PA-DR reaches 58.7% success with 9.9% leakage by rewarding stage-specific decisions and penalizing query-level privacy risk. The evaluation is controlled, using synthetic enterprise documents, a fixed web corpus, three company contexts, and one multi-hop question-answering harness.
Data Engineering for AI: A Practical Guide for Data Professionals
Data engineering for AI extends traditional ETL by adding requirements for model-ready data, unstructured inputs, feature engineering, governance, and production monitoring. The guide addresses data professionals building or scaling AI infrastructure, covering ingestion, architecture, data quality, privacy compliance, generative AI, and career development. It emphasizes shared feature schemas, pipeline data contracts, lineage, statistical validation, drift detection, and PII stripping or anonymization as responsibilities shared across data engineering and data science. For generative AI systems, it describes preparing RAG pipelines by ingesting and chunking documents, creating vector embeddings, and indexing them for semantic retrieval, while evaluating vector databases for latency, scale, and integration. It concludes that reliable AI depends on fresh, accurate, compliant data and ongoing operational and architectural review.
The new bottleneck
AI coding tools have significantly lowered the cost of generating software, yet many engineering organizations fail to realize overall delivery speed improvements. Applying the Theory of Constraints reveals that eliminating code production as a bottleneck shifts inventory directly into surrounding legacy processes that remain unadjusted. New friction points consistently emerge across underspecified requirements, prolonged design handoff gates, senior engineer review capacity, and external sign-offs from legal or security. Organizations can address these blockers by interrogating legacy agile ceremonies checkpoint by checkpoint to determine if their original constraints still exist. Practical remedies include adopting real-time co-development between product and engineering, treating initial designs as fluid starting points, and restructuring processes around running rapid, high-volume experiments.
Beyond LoRA: Can you beat the most popular fine-tuning technique?
Parameter-efficient fine-tuning (PEFT) reduces the memory required to adapt open models, enabling fine-tuning with a fraction of the memory and, in some cases, quantized models. LoRA is dominant because it freezes base weights and trains a small set of added parameters, but its popularity may also reflect ecosystem momentum and uneven comparisons. The post presents shared-condition benchmarks for language-model math reasoning and image generation, using identical models, datasets, code, hardware, and measures including test performance, VRAM, forgetting or drift, runtime, and checkpoint size. The results show that LoRA performs well but is not consistently best: other techniques can outperform it on one or more axes, especially in the image-generation benchmark. The recommended choice is therefore to compare techniques against a use case’s needs and constraints, including quantization support, layer coverage, adapter merging, and downstream serving compatibility.
The open ecosystem around ClickHouse
ClickHouse’s open ecosystem spans clients, dashboards, ingestion pipelines, projects built on its engine, and agents that query it, with 2,600+ contributors, 240,000+ commits, and hundreds of millions of downloads cited. It describes open standards including OpenTelemetry for observability, Model Context Protocol (MCP) for AI tooling, and Iceberg and Delta Lake for lakehouse data, plus Iceberg REST catalogs, Apache Polaris, and Unity Catalog. The surrounding stack includes ClickStack, chDB, Langfuse, LibreChat, official language clients, a Kubernetes operator, Postgres bridges such as PeerDB and pg_clickhouse, and integrations with Airflow, dbt, Grafana, Fivetran, and Sigma. Specific updates include full Iceberg v2 support, Iceberg writes in 25.8, Paimon in 25.10, chDB v4’s DataStore API, and a Python client v1.0 with native async and 4x faster cold starts. The post frames this openness as supporting flexible tooling and community-led growth, reporting more than 200 integrations while noting that the ecosystem’s next direction remains unknown.
AI agents are a confused deputy with the keys to your kingdom
Attackers recently compromised over twenty thousand Instagram accounts by manipulating Meta's AI support assistant to rebind recovery email addresses without verifying account ownership. This incident illustrates the classic confused deputy security problem, where a privileged process is persuaded by an unprivileged user to perform unauthorized operations. Because large language model interfaces operate purely on natural language and cannot distinguish instructions from data, the model itself cannot serve as an authorization boundary. Securing AI agents requires verifying caller identity through external policy checks against authenticated sessions rather than relying on chat context or prompt engineering. Teams must enforce least privilege with short-lived scoped credentials, place irreversible actions behind hard policy gates or human approvals, and maintain audit trails of agent actions.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.