Activation Steering: One Dial, Two Costs
Activation steering edits a model's internal activations to add or remove a concept at inference time. This explainer maps its one control knob, its two costs, and when it beats prompting.
Loading…
Topic
LLMs
673 posts about LLMs. Every summary links to the original.
- 673 posts
- 49 companies
- 5 explainers
See the idea
LLMs explainers
Antibodies as a Prediction Pipeline: Where AI Models Break Down
Antibody discovery is a three-stage prediction pipeline, and AI models that excel on familiar targets degrade sharply on unfamiliar ones. This explainer maps where that gap appears and what it costs.
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Closing the Binary Blind Spot: Domain-Knowledge Injection for LLM Malware Triage in npm Packages
Supply chain attacks on the npm registry are growing, and packages that bundle a binary payload are rare but disproportionately hard for LLMs to triage. The authors built a scanner that combines static analysis with LLM reasoning and introduced capability profiles, which encode malicious techniques as weighted conditions injected into the prompt before the model decides. They tested nine models across seven providers on five adversarial payload types, including credential stealers and reverse shells, at varying obfuscation levels. Without profiles, the budget-tier models failed to reach a malicious verdict on 50 percent of tests, with failures following distinct model-specific patterns rather than random error. With profiles enabled, the failure rate dropped to 4 percent, and 91 percent of prior failures recovered to a detection.
Advancing Generative Recommendation Inference: our collaboration with NVIDIA on TensorRT LLM
Spotify has been applying generative AI to personalize recommendations across surfaces, using LLMs for podcast discovery with beam search and candidate scoring. Their initial inference stack relied on vLLM for beam search, which suffered from CPU-side bottlenecks and KV block recomputation, and SGLang for efficient multi-item scoring, but no single engine supported all required features. To unify these capabilities, Spotify collaborated with NVIDIA to extend TensorRT LLM with efficient beam search implemented in the sampler, multi-item scoring via the Python API, and soft token personalization using out-of-tree model plugins. On a 1.7B Qwen3 model, the resulting implementation delivered up to 54% cost reduction for beam search workloads and 7% for scoring workloads compared to previous setups. Spotify plans to continue optimizing inference with quantization and NVIDIA Blackwell-specific techniques to support increasingly sophisticated personalization at scale.
Welcome Beam: a 501B open model you can own, customize, and scale
Reflection released Beam, a 501B-parameter mixture-of-experts model with 23B active parameters per token, under the Apache 2.0 license, targeting coding and agentic use cases. The model supports a 1M-token context window using hybrid attention with 43 sliding-window and 9 global layers, and a residual stream that scales sublayer outputs by 1/sqrt(k) to maintain constant magnitude. Its MoE router uses a group-limited scheme with 256 experts, 8 routed per token, and the BF16 checkpoint occupies about 1 TB. Benchmarks show strong performance: 81.6% on SWE-bench Verified, 97.7% on AIME 2026, and 90.6% on GPQA Diamond, with minimal degradation in FP8 and NVFP4 quantizations. The model ships with a Rust-based harness, day-0 support in transformers, vLLM, and SGLang, and includes SFT and RL recipes, with a $10,000 Hub credits competition for community projects.
MCP standardized tools. Now we need the infrastructure around them.
MCP standardized how AI models connect to tools, with clients like Claude Code and Cursor adopting it and SDK downloads approaching half a billion monthly. However, production use reveals that the protocol solves only the interface problem, not the operational concerns around authentication, rate limiting, idempotency, and observability. Teams repeatedly build duplicate infrastructure for deployment, secret handling, health checks, and access control because the protocol does not address these. Context efficiency also remains a challenge: a benchmark showed reducing context from 220,393 to 27,352 tokens (87.6%) while maintaining test pass rates, highlighting the need for deliberate context selection. The author argues that MCP’s success forces the question of building a production-grade infrastructure layer, similar to how HTTP led to CDNs and API gateways.
Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint
Granite Switch 4.1 models combine chat, general and specialized decisions, and speech-to-text from a single hosted endpoint. The 8B and 30B models include a new Decision aLoRA for general-purpose decision-making and a Guardian aLoRA that outperforms Jev 1.13 and other closed models on safety tasks. They also embed Granite-Speech-5.0-Turbo-CTC, the fastest ASR model on the OpenASR Leaderboard, enabling efficient transcription. All components are composed into one model using cache-efficient Activated LoRAs, taking up about the same GPU VRAM as the base model alone, and the KV cache from base generation can be reused for any subsequent adapter decisions. Granite Switch 4.1 can switch between chat, decision, and spoken requests quickly, and a 3B variant achieved 60-100 decisions per second playing Doom, compared to Jev’s 10, while also narrating and answering questions.
How FactSet cut model training and deployment from days to hours with Databricks AI Runtime
FactSet’s Text-to-Formula (TTF) system converts natural-language investment queries from professionals into correct FactSet Query Language (FQL) formulas. Initially built on retrieval-augmented generation (RAG), the approach hit a quality ceiling as queries became more specialized and formulas more complex. FactSet adopted fine-tuned open models within a Compound AI architecture, reducing end-to-end latency by about 60 percent and improving quality for complex formulas. Fine-tuning runs on Databricks AI Runtime with serverless GPU infrastructure, where engineers choose compute based on prompt length and output needs, and the pipeline integrates with MLflow, the Lakehouse, and Lakeflow Jobs for a unified workflow. This setup cut model setup, training, and deployment from days to hours, enabling the team to retrain every two to three weeks as new data and models become available.
Scale Agentic Coding Workflows with Meta’s Muse Spark Open on NVIDIA GB300 NVL72
Meta launches Muse Spark Open, an 816B-parameter mixture-of-experts model with 42B active parameters and a 1M token context window, trained on NVIDIA GB300 NVL72 for agentic coding workflows. It plans, acts, observes, and self-corrects across multistep tasks like multifile refactoring, using tool calls and subagent coordination while holding an entire project in context without breaking it into pieces. Its 1.8B perception encoder natively processes interleaved text, images, video frames, and audio in a single pass. On GB300 NVL72 with speculative decoding, it achieves over 50K tokens per second per GPU and over 600 tokens per second per user. Sparse MoE routing activates only 42B parameters per token, and NVFP4 quantization reduces memory footprint for larger KV cache. The model is available via Hugging Face and NVIDIA NIM, with support from SGLang, vLLM, and TensorRT LLM for serving, and fine-tuning through NeMo AutoModel and Megatron Bridge.
Apertus 1.5 Release Notes
Apertus 1.5 extends the 8B and 70B open-weight models with image and audio understanding, a thinking mode, and a 262,144-token context window, released under Apache 2.0. The models are a continued pretraining of Apertus 1.0, adding 4 trillion tokens to the 8B model and 2 trillion to the 70B model, for totals of 19T and 17T respectively. Multimodality uses an early-fusion tokenizer with Emu3.5 for images and WavTokenizer for audio, keeping the architecture unchanged and training at text-only speed. Post-training includes SFT, long-context SFT, reinforcement learning with verifiable rewards, DPO, and charter alignment via SDPO, with all data and recipes publicly documented. Evaluation results show competitive performance on reasoning, coding, and multilingual tasks, with the 8B model matching GPT-OSS-20B on translation and outperforming Claude Haiku 4.5, reported transparently via the EvalEval Coalition’s EEE schema.
Reducing Qwen training time by 6x on AI Runtime
The post describes systematic performance optimization to reduce fine-tuning time for the Qwen 3.8 27B model on AI Runtime from 7 hours 44 minutes to 1 hour 17 minutes, a 6x improvement. The baseline showed 98% GPU utilization, but torch traces revealed the CPU was stalling the GPU with too many small operations. Installing manually fused kernels such as flash-linear-attention and causal-conv1d reduced the number of GPU operations by 15x and improved training speed by 3.7x. Further gains came from switching from 2D padded batches to 1D packed tensors with correct position_ids, which reduced the number of batches by 36% and eliminated dynamic shape recompilation. The post emphasizes measuring what matters beyond GPU utilization and leveraging pre-built community kernels and efficient batching.
Building AI with MongoDB and Patronus AI
Large language models can be unreliable, with serious consequences in regulated industries. Patronus AI, an industry-first automated evaluation platform, enables engineers to score and benchmark LLM performance on real-world scenarios, generate adversarial test cases, and monitor hallucinations and PII. Research using FinanceBench found that state-of-the-art LLMs hallucinated or incorrectly answered up to 81% of financial analysts’ questions, even with RAG and external context. Patronus partnered with MongoDB Atlas to offer a RAG evaluation API and published a 10-minute guide demonstrating evaluation of a RAG system built with LlamaIndex and Atlas Vector Search on an SEC 10-K filing. Developers can improve RAG performance by adjusting indexes, chunking sizes, prompts, or fine-tuning embedding models, and should continuously test to maintain improvements.
Plan or Retrieve First? A Scale Study of Tool Discovery Architectures
Tool-augmented LLM agents must discover relevant APIs from large registries before planning, but finite context windows prevent injecting all tool definitions into a single prompt. Two competing architectures are Retrieve-then-Plan (RtP), which embeds the query and retrieves similar tools via nearest-neighbor search, and Plan-then-Retrieve (PtR), which first generates abstract tool stubs as retrieval queries. Evaluating on ToolBench with 46,984 API endpoints across 50 categories reveals a clear complexity-dependent crossover: RtP outperforms PtR on single-tool queries while PtR outperforms RtP on multi-tool queries. PtR’s stub-hallucination failure mode acts as a double-edged sword—on simple queries it abstracts away proper-noun signals, yet on complex queries it decomposes multi-faceted intent into functional sub-needs. PtR roughly doubles inference cost, which pays off on multi-tool queries but hurts on single-tool ones, motivating complexity-adaptive routing driven by RtP’s top-1 retrieval score.
Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta
Meta released Muse Spark Open, an open-weight multimodal mixture-of-experts model with approximately 816 billion total parameters and around 42 billion active per token. It processes text, images, audio, and video inputs while generating text output, supporting a context window of up to one million tokens. The model uses 256 experts per MoE layer with 8 selected per token, and the BF16 checkpoint requires about 1.5 TB of storage. Meta trained Muse Spark Open for agentic coding tasks including reading codebases, writing and debugging code, and calling tools. The post provides recipes for running the model using transformers, llama.cpp, vLLM, and SGLang, as well as fine-tuning with LoRA on 64 H100 GPUs.
Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
MLX enables running large language models locally on Apple silicon by exploiting the unified memory architecture of Mac Studio. The latest MLX release supports distributed inference across multiple Macs using RDMA over Thunderbolt, with pipeline or tensor parallelization schemes, and uses the jaccl communication collective. Benchmarks compare M3 Ultra and M5 Ultra across three models: Qwen 3.8 (27B) and Mistral Medium 3.5 (128B) fit on a single 512GB Mac, while the newly released Muse Spark Open (816B) requires parallelization across up to four machines at its 4-bit NVFP4 quantization. The M5 Ultra’s GPU Neural Accelerator delivers up to 4.6 times faster time-to-first-token than the M3 Ultra, while its 1.2 TB/s memory bandwidth improves generation throughput by 1.6 times for dense models. On four M5 Ultra Mac Studios with tensor parallelism, Muse Spark Open achieves over 36 tokens per second.
Your next job doesn’t have a title yet: Emerging role skills Q&A with Corey Hymel of Andela
Corey Hymel, head of research at Andela, analyzed nearly 50,000 job postings to identify emerging skill bundles that precede formal job titles. The research found AI is creating niche skill sets, not turning everyone into generalists, with roles like LLM application engineer building on top of APIs, FinOps reliability engineer managing consumption-based costs, and docs-as-code engineer constructing context for AI models. Hymel described “bleed roles” where skills merge between adjacent specialties, such as front-end product engineers stretching into product management. The data also showed developer attitudes toward AI tooling vary, with some hand-coding for accountability while others ship code that passes tests without concern. Hymel advises engineers to understand where trends are heading and invest effort to prepare for roles that are morphing but not yet commonly titled, emphasizing the importance of strategic upskilling.
Run fast, private agents on-device with LFM2.5-24B-A2B
LFM2.5-24B-A2B is a 24-billion-parameter reasoning model with only 2 billion active parameters per token, designed for fast, private on-device agents. It uses a Mixture-of-Experts architecture with 40 layers, 64 experts per block, and top-4 routing, combined with gated short convolutions and GQA. The model produces an explicit chain of thought before answering and supports tool calling and multi-step tasks. Pre-training scaled to 33 trillion tokens, followed by mid-training and a five-stage post-training pipeline including supervised fine-tuning, multi-domain distillation, and agentic reinforcement learning. On benchmarks, it leads or matches larger models on tool use and instruction following, achieving 48.09 on LiveClaw and 96.75 on IFEval, while decoding 184 tokens per second on an Apple M5 Max.
Meta’s Generative Ads Model (GEM): Scaling the Central Model and Compute Engine for Frontier Ads AI
Meta introduced GEM, the industry’s largest recommendation system foundation model, and its 2026 generation shifts from isolated per-domain optimizations on Facebook, Instagram, and Reels to a single central foundation model. The Kunlun architecture, co-designed with the latest GPU hardware, enables 2.5x model complexity and demonstrates LLM-like scaling laws with block-aligned sparse attention reducing sequence complexity. The LoopFM post-training technique transfers GEM’s internal representations to downstream models, doubling intelligence transfer at no additional compute cost or latency. GEM delivered 18.9% conversion improvement and 12% ad click-through rate improvement across all Meta ad surfaces and funnel stages, including 5.8% conversion uplift on Facebook, 4.1% on Instagram, and 1.6% on off-site conversion. Training efficiency doubled to 20-25% model FLOPS utilization while scaling total training FLOPs by 4x across thousands of GPUs using custom kernels and 5D parallelism.
Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings
In data-constrained language model pre-training, two remedies exist: aggressive hyperparameter tuning such as high weight decay, and mixing in data from a high-resource auxiliary language to expand the training distribution. This study systematically compares these levers across four model scales from 150M to 1.43B parameters, using Arabic as the low-resource target and English as the auxiliary, over approximately 1000 pre-training runs. Mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy, and the gap grows with model size. Mixing boosts performance by an amount equivalent to 2–3× the unique target data on validation loss and 2–13× on downstream task accuracy, with the gain scaling steeply with model size. The authors recommend mixing in a high-resource language, prioritizing the mixing ratio over hyperparameter tuning, and transferring hyperparameters from a small proxy model via μP, noting that target-language validation loss systematically underestimates mixing’s value.
How Flo Health built safer, more useful open models with Databricks AI Runtime
Flo Health, a women’s health app with 81 million monthly active users, aimed to improve generative AI responses for medical safety and usefulness while operating at scale. Relying on general-purpose models created cost, dependency, and domain-expertise limitations, so Flo built a post-training pipeline with Databricks AI Runtime. The team defined 26 AI judges across three categories: 12 medical-safety judges acting as hard constraints, 6 micro judges for nuanced behaviors, and 8 usefulness criteria. Using these judges as a reward signal, they generated over 15,000 synthetic questions from privacy-first principles and applied reinforcement fine-tuning with supervised fine-tuning (SFT) and direct preference optimization (DPO). After fine-tuning on 32 NVIDIA H100 GPUs using PyTorch and Fully Sharded Data Parallel (FSDP), medical safety scores rose from 34% to 61%, usefulness from 42% to 76%, and the combined metric improved nearly threefold to 47%.
Can we change an AI model’s architecture without retraining it from scratch?
The Transformer architecture’s memory cost scales with token count, making long-context AI agents expensive. In controlled comparisons of three state space models - Gated KalmaNet, Gated DeltaNet, and Mamba-2 - Gated KalmaNet performed best, offering variable-length inference without retraining. Priming identifies which attention layers can be replaced by state space models using a sliding-window attention proxy, then reuses pre-trained weights to convert them into hybrid architectures without retraining, reducing pretraining cost by over 99%. A 32-billion-parameter hybrid model matches the original Transformer’s reasoning quality while decoding up to twice as fast or reaching correct answers in two-thirds of wall-clock time. The team extended hybrid context to 256K tokens without additional training and released primed models at 8B and 32B scales along with training code, kernels, and a vLLM plugin under Apache 2.0.
From post-training to production: How TinyFish runs its web-native model on Databricks AI Runtime
TinyFish builds infrastructure for enterprises that need agents to work across the live web and is developing its own specialized web-native model. The company uses Databricks AI Runtime to manage GPU compute for post-training with supervised fine-tuning and moving toward Direct Preference Optimization, MLflow for evaluation, and Model Serving for production deployment. TinyFish evaluates its model by measuring full agent task completion rather than individual responses, achieving a 77.6% success rate on its internal go/no-go benchmark where 1,691 of 2,178 tasks were completed successfully. Over a 15-day period, the web-native model handled 294,781 agent runs across 6.4 million tool steps at an estimated serving cost of $0.088 per run or $0.0040 per tool step. By consolidating training, evaluation, serving, and operational data on Databricks, TinyFish reduces infrastructure overhead and maintains a continuous development cycle from agent behavior to deployment.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.