Activation Steering: One Dial, Two Costs
Activation steering edits a model's internal activations to add or remove a concept at inference time. This explainer maps its one control knob, its two costs, and when it beats prompting.
Loading…
Topic
LLMs
660 posts about LLMs. Every summary links to the original.
- 660 posts
- 49 companies
- 5 explainers
See the idea
LLMs explainers
Antibodies as a Prediction Pipeline: Where AI Models Break Down
Antibody discovery is a three-stage prediction pipeline, and AI models that excel on familiar targets degrade sharply on unfamiliar ones. This explainer maps where that gap appears and what it costs.
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How we built a custom vision LLM to improve document processing at Grab
Grab developed a specialized Vision LLM for extracting text and key information from Southeast Asian identity cards, driver’s licenses, and registration certificates, where varied languages and layouts challenged traditional OCR and existing models. It selected Qwen2-VL 2B for its efficient size, SEA language tokenizer coverage, and dynamic native-resolution image handling, while synthetic Common Crawl text images and Documint supplied reviewed training labels for detection, orientation, OCR, and KIE. LoRA fine-tuning improved Latin-script documents but struggled with Thai, Vietnamese, and dense unstructured layouts, leading to two-stage full-parameter training with synthetic SEA text images followed by task-specific document fine-tuning. The fully fine-tuned 2B model raised Thai accuracy by 70 percentage points and Vietnamese accuracy by 40 points over baseline, while a custom roughly 1B-parameter model combining Qwen2-VL 2B’s vision encoder with Qwen2.5 0.5B’s decoder stayed within a 3-point accuracy gap across most document types and achieved lower latency.
Aligning to What? Rethinking Agent Generalization in MiniMax M2
MiniMax M2 addresses the performance gap between agent benchmark scores and real-world deployment across diverse scaffolding frameworks and environments. While initial alignment efforts relied on scaling tool variety, models continued to fail when subjected to slight variations in prompts, scaffolding, and tool responses. To resolve this instability, the team introduced interleaved thinking, which enables models to execute internal reasoning at arbitrary steps throughout long-horizon tasks to adapt dynamically to external tool outputs. Additionally, developers constructed a data pipeline designed for full-trajectory generalization by introducing perturbations across system prompts, user objectives, environments, and tool feedback. In internal evaluations, MiniMax M2 maintained robust tool-calling and instruction-following performance when tested against unfamiliar, cold-start agent scaffolding frameworks.
On the Shifting Global Compute Landscape
United States export controls on advanced artificial intelligence hardware have catalyzed rapid expansion across China's domestic silicon and open-weight model ecosystem. Faced with restricted access to high-end NVIDIA GPUs, Chinese developers accelerated deployments on domestic accelerators, including Huawei Ascend, Cambricon, and Baidu Kunlun. Hardware scarcity spurred architectural and algorithmic innovations in compute efficiency, notably DeepSeek's Multi-head Latent Attention and Group Relative Policy Optimization, alongside substantial post-training cost reductions. Organizations such as Baidu and Ant Group now train foundation models directly on domestic hardware, fostering non-CUDA software stacks and lowering inference costs globally. Consequently, the global artificial intelligence infrastructure is shifting from an exclusively American-focused paradigm toward a dual-ecosystem landscape powered by domestic chips and open-weight architectures.
Granite 4.0 Nano: Just how small can you go?
IBM has released Granite 4.0 Nano, a collection of sub-billion and ~1.5B parameter language models tailored for edge and on-device deployments. Trained on more than 15 trillion tokens under an Apache 2.0 license, the suite includes four instruct models alongside their base counterparts. The lineup introduces hybrid-SSM architectures—specifically the ~1.5B parameter Granite 4.0 H 1B and the ~350M parameter Granite 4.0 H 350M—as well as standard transformer variants suited for runtimes like llama.cpp. According to benchmark results across general knowledge, math, code, and safety domains, these compact models demonstrated strong capability improvements relative to their parameter footprint. Furthermore, evaluation on IFEval and Berkeley's Function Calling Leaderboard v3 indicated competitive performance on instruction following and tool calling tasks.
Platform Thinking: Three Views from Two Sigma Leaders
Two Sigma structures its quantitative investment operations around foundational platforms that balance operational speed with scientific rigor. In data engineering, foundational teams provide both raw and curated datasets using BigQuery, CI/CD-managed transformation pipelines, and reusable data contracts termed ice cubes to serve ninety percent of use cases. Quantitative modeling incorporates open-source large language models trained strictly on point-in-time data to rapidly generate features while preventing temporal leakage. Long-term platform innovation follows an epsilon and omega strategy that couples overarching vision with iterative, learning-focused steps. Platform leadership categorizes artificial intelligence applications into advisory, oracle, operational, and agentic functions designed to enhance human productivity rather than replace researchers.
Building the Open Agent Ecosystem Together: Introducing OpenEnv
Meta and Hugging Face have partnered to launch OpenEnv Hub, an open platform and specification designed for agentic AI environments. Large language models require external tools and APIs to complete tasks, but exposing broad toolsets directly introduces significant safety and semantic challenges. In response, OpenEnv establishes sandboxed environments providing defined APIs, authenticated tool access, and isolation for reinforcement learning post-training and deployment pipelines. The initial release includes the OpenEnv 0.1 RFC series covering component architecture, packaging, communication, and Model Context Protocol tool encapsulation. Creators can run local Docker-based environments using standard step, reset, and close interfaces while integrating with frameworks like TorchForge, TRL, verl, and SkyRL.
Sentence Transformers is joining Hugging Face!
Sentence Transformers is officially transitioning its stewardship from the Ubiquitous Knowledge Processing Lab at TU Darmstadt to Hugging Face. The open-source embedding library was introduced in 2019 by Nils Reimers using a Siamese network architecture to overcome standard BERT limitations on sentence-level semantics. Tom Aarsen of Hugging Face has maintained the project since late 2023 and will continue leading its development. Under Hugging Face, the project retains its Apache 2.0 license and leverages infrastructure such as continuous integration and automated testing. Sentence Transformers currently serves over one million monthly unique users, with more than 16,000 community models publicly hosted on the Hugging Face Hub.
Supercharge your OCR Pipelines with Open Models
Modern vision-language OCR models can transcribe handwritten text, multiple scripts, mathematical expressions, and chemical formulas while also interpreting tables, charts, and images. The guide compares open OCR models and explains how output choices—including DocTags, HTML, Markdown, and structured JSON—affect digital reconstruction, LLM input, and programmatic processing. It also covers locality awareness through anchors or bounding boxes, prompt-based task switching, fine-tuning versus out-of-the-box use, and factors for selecting a model. Practical deployment options include local or managed inference, vLLM, Hugging Face Endpoints, and batch scripts that add OCR results to datasets. The discussion extends document AI beyond transcription to visual retrieval and direct document question answering with vision-language models, especially where converted text can lose layout or visual context.
Unlock the power of images with AI Sheets
Hugging Face released an update to AI Sheets, an open-source tool for transforming and enriching datasets without code. The update adds vision support powered by Inference Providers, enabling users to upload images, extract structured data, generate visuals, and perform image edits in a spreadsheet interface. Each column can execute AI actions configured with custom prompts and selected models. For text extraction and reasoning tasks, AI Sheets supports vision language models including Qwen/Qwen2.5-VL-7B-Instruct and Qwen/Qwen3-VL-235B-A22B-Reasoning, while image transformations use models like Qwen-Image-Edit. Users can iterate on prompts, provide feedback using thumbs-up ratings to create few-shot examples, and export completed datasets to the Hugging Face Hub, CSV, or Parquet.
AI Interview Success: An Interviewer's Inside Guide
Canva conducts AI-assisted programming interviews designed to evaluate candidates' engineering fundamentals amplified by artificial intelligence tools. Successful candidates structure their interviews across planning, development, and discussion phases, clarifying architectural assumptions before writing code and critically reviewing AI-generated output. Interviewers evaluate engineering problem-solving, technical depth, code ownership, and the effective collaboration with AI assistants rather than isolated prompting mechanics or rushed feature completion. Ineffective strategies include treating models as hands-off solution generators, rushing unvetted features, and prioritizing demonstrations of advanced prompting over core technical judgment. Preparation involves setting up project structures in advance, exploring requirements aloud, and remaining ready to justify architectural trade-offs and production scalability.
Google Cloud C4 Brings a 70% TCO improvement on GPT OSS with Intel and Hugging Face
Intel and Hugging Face benchmarked OpenAI GPT OSS text generation to evaluate performance on Google Cloud C4 virtual machines powered by Intel Xeon 6 processors against previous-generation C3 instances. The benchmark evaluated the unsloth/gpt-oss-120b-BF16 Mixture of Experts model across batch sizes from 1 to 64 using bfloat16 precision, static KV cache, and SDPA attention. To eliminate redundant computation, Hugging Face merged an optimization directing each expert sub-network to process only its routed tokens rather than all tokens. Testing demonstrated that C4 instances deliver 1.4x to 1.7x higher throughput per vCPU compared to C3 instances. At a batch size of 64, C4 achieved a 1.7x Total Cost of Ownership improvement alongside lower hourly pricing.
Nemotron-Personas-India: Synthesized Data for Sovereign AI
Most open artificial intelligence datasets reflect Western norms and English-only contexts, limiting model effectiveness across India's multilingual and multi-script landscape. To address this shortage of culturally grounded training data, NVIDIA released Nemotron-Personas-India under a CC BY 4.0 license. The synthetic dataset contains 21 million personas across 3 million records spanning 7.7 billion tokens in English, Devanagari Hindi, and Latin-script Hindi. The generation pipeline utilized NeMo Data Designer, an Apache-2.0 licensed Probabilistic Graphical Model for demographic grounding against 2011 Census distributions, and GPT-OSS-120B for narrative text. All records are entirely synthetic to eliminate re-identification risks while enabling fine-tuning for region-aware AI agents and domain-specific copilots.
Understanding Etsy’s Vast Inventory with LLMs
Etsy’s seller-created inventory spans more than 100 million items and thousands of categories, making it difficult to capture attributes such as color, material, or niche properties without global SKUs. Because sellers often provide descriptions, titles, and photos instead of complete structured fields, Etsy uses LLMs to distill unstructured listing data into attributes for search filters, comparisons, and other experiences. The pipeline combines seller data, expert-selected few-shot examples, taxonomy logic, and category rules; parallel prompts are routed through LiteLLM, then responses are validated with Pydantic dataclasses and monitored with tracing, metrics, alerts, and performance checks. Human-reviewed silver labels support evaluation, while target categories increased complete attribute coverage from 31% to 91%, and inferred attributes improved search-filter engagement and overall post-click conversion.
Arm will be @ PyTorch Conference, Join Us!
Arm announced its schedule and offerings for the PyTorch Conference on October 22–23, focusing on building and deploying AI applications using PyTorch and ExecuTorch. The company is hosting one-on-one workshops with design experts to improve product usability and introduce Yellow Teaming practices for mitigating fairness, transparency, privacy, and security risks. Additionally, Arm is running 30-minute Voice of the Developer sessions to gather feedback on real-world challenges, including migrating from NVIDIA or x86 to Arm, debugging models at scale, and running large language models on edge devices. At Booth P1, attendees can view interactive demonstrations featuring neural graphics training, speech recognition, audio generation, vLLM, Mixture of Experts, and ExecuTorch edge deployments.
BigCodeArena: Judging code generations end to end with code executions
Assessing AI-generated code via static inspection is error-prone and fails to reflect whether code functions correctly across real-world environments. BigCodeArena addresses this evaluation bottleneck as a human-in-the-loop platform that executes model outputs in real time inside isolated sandbox environments across ten programming languages and eight execution frameworks. Users interact directly with live applications, refine implementations through multi-turn conversations, and submit pairwise preference votes to generate Elo ratings via the Bradley-Terry model. In five months of community testing covering over 14,000 conversations, proprietary models including o3-mini and o1-mini led overall Elo rankings, while open models like Qwen2.5 and Llama-3.3-70B trailed closed competitors. The project has open-sourced its full evaluation pipelines, conversation datasets, and related benchmarks to support community-driven code generation assessment.
Smarter AI Search, Powered by MongoDB Atlas and Pureinsights
MongoDB Atlas has announced the general availability of its integration with the Pureinsights Discovery Platform to deliver a unified keyword, vector, and generative search experience. The combined solution pairs MongoDB Atlas Search for standard text matching with MongoDB Atlas Vector Search and Voyage AI embeddings to interpret ambiguous or multilingual queries. Pureinsights provides the orchestration layer that ingests content, coordinates retrieval, and integrates large language models such as GPT-4 to generate cited responses. Users can customize generated outputs according to preferred technical depth, length, language, and role-specific requirements. The architecture enables enterprises to apply retrieval-augmented generation and semantic search across technical documentation, community forums, and internal knowledge bases.
Introducing RTEB: A New Standard for Retrieval Evaluation
Public embedding benchmarks often fail to measure true generalization because training sets overlap with public evaluation data, rewarding memorization over general search quality. The Retrieval Embedding Benchmark (RTEB) beta introduces a standardized evaluation framework designed for real-world enterprise retrieval applications. To prevent overfitting while preserving transparency, RTEB combines fully public datasets with private evaluation sets managed exclusively by MTEB maintainers. The suite spans 20 languages across domains like healthcare, law, finance, and code, measuring ranked retrieval quality using NDCG@10 on datasets of at least 1,000 documents. While currently limited to text-only retrieval and containing repurposed question-answering datasets, the benchmark provides clear signals when models degrade on unseen data.
Top Considerations When Choosing a Hybrid Search Solution
Natural language queries and generative AI workflows like retrieval-augmented generation have exposed the limitations of standalone vector search, particularly for tokens outside an embedding model's training corpus. Hybrid search addresses these gaps by merging traditional lexical precision with dense vector context. Platforms combine these modalities using techniques such as reciprocal rank fusion and relative score fusion, often complemented by downstream cross-encoders or learning-to-rank models. Architecture choices center on the trade-offs between lexical-first systems using inverted indexes and vector-first platforms employing sparse vectors. MongoDB has integrated vector search indexes alongside its existing lexical indexes, releasing native hybrid search functions across Atlas, Community Edition, and Enterprise Server to unify operational data and search retrieval.
Accelerating Qwen3-8B Agent on Intel® Core™ Ultra with Depth-Pruned Draft Models
Local agentic AI workflows generate multi-step reasoning traces that increase token counts and strain real-time inference on personal computers. To accelerate execution, researchers implemented speculative decoding using OpenVINO.GenAI on an Intel Lunar Lake integrated GPU, pairing a 4-bit Qwen3-8B target model with a smaller Qwen3-0.6B draft model. By measuring angular distance across layers, the team pruned six of the draft model's twenty-eight layers to minimize draft latency and fine-tuned it on synthetic data generated from 500,000 prompts. This depth-pruned draft elevated speculative decoding throughput from a 1.3x speedup to a 1.4x speedup over the standalone baseline. Finally, deploying the optimized model alongside the Hugging Face smolagents framework allowed local agents to perform tool invocation and code generation tasks efficiently.
VibeGame: Exploring Vibe Coding Games
Vibe coding games often fails as projects grow because large context sizes degrade model performance. After evaluating Roblox MCP, Unity MCP, and open web libraries with Claude Code, the author found web tooling offered the strongest model proficiency but lacked high-level game abstractions. To bridge this gap, the author created VibeGame, an open-source declarative game engine built on three.js, rapier, and bitecs. VibeGame uses an Entity-Component-System architecture, XML-like markup, and bundled documentation in an llms.txt file to keep project context lean. While effective for basic physics and rendering mechanics, the engine requires further development for complex features like inventories, multiplayer, and interactions.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.