Activation Steering: One Dial, Two Costs
Activation steering edits a model's internal activations to add or remove a concept at inference time. This explainer maps its one control knob, its two costs, and when it beats prompting.
Loading…
Topic
LLMs
648 posts about LLMs. Every summary links to the original.
- 648 posts
- 49 companies
- 5 explainers
See the idea
LLMs explainers
Antibodies as a Prediction Pipeline: Where AI Models Break Down
Antibody discovery is a three-stage prediction pipeline, and AI models that excel on familiar targets degrade sharply on unfamiliar ones. This explainer maps where that gap appears and what it costs.
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How we’re building a data platform for a new user: agents
AI agents are emerging as autonomous database users that generate SQL, execute queries, and iterate in seconds. Because traditional analytics workflows built for human analysts cannot keep up with these rapid loops, ClickHouse is adapting its platform for agentic workloads. The proposed Agentic Data Stack combines LibreChat for conversational interfaces, ClickHouse with Model Context Protocol servers for the data layer, and Langfuse for observability. Supporting developer tooling includes chDB 4 with a Pandas-like API, IDE integrations, and ClickHouse Agent Skills to enforce database design best practices. Platforms like Anthropic and Character.AI already leverage ClickHouse's high-concurrency aggregation and real-time ingestion capabilities to handle massive data scales required by modern AI applications.
Building Biz Ask Anything: From Prototype to Product
Yelp expanded Yelp Assistant to business pages by developing a question-answering architecture that extracts concise, evidence-backed answers from reviews, photos, and structured facts. The production system replaces static snapshots with near-real-time streaming pipelines for reviews and structured properties alongside weekly batch ingestion for menus and website data. For each incoming query, the system evaluates intent, retrieves relevant context through keyword and embedding searches, and constructs targeted prompts with dynamically selected few-shot examples. Synthesized answers are augmented with supporting visuals and citations before being streamed token-by-token to users. Generating suggested questions directly from specific business content rather than generic categories improved user engagement by roughly fifty percent and reduced unanswerable queries by twenty-six percent.
Announcing Fin Apex: The age of vertical models is here
Fin announces Apex 1.0, a custom model trained by its AI Group for customer-service conversations, replacing frontier-lab models as the core answering model for nearly all English-language chat and email traffic. The post says Apex improves resolution, speed, hallucination rates, and cost, citing one gaming customer’s resolution rate rising from 68% to 75% overnight and describing a 22% reduction in unresolved conversations. Its development relies on proprietary evaluations and billions of human and agent interaction data points generated by Fin’s resolution engine, creating a training flywheel for further domain-specific improvement. The launch is presented as evidence that customer-service applications may need to become full-stack AI companies rather than relying on shared general-purpose models. It also argues that frontier labs face pressure to offer cheaper specialized models as post-training and task-specific data become more important.
Intelligent security at ClickHouse speed: How Cogent Security built an AI-native vulnerability management platform
Enterprise security defenders face shrinking exploitation windows alongside massive volumes of vulnerability findings across disconnected tools. To overcome query bottlenecks from traditional relational databases, Cogent Security designed a lakehouse serving architecture using Apache Iceberg and per-customer ClickHouse instances. The architecture relies on query-driven denormalized tables, custom compression codecs, and pre-computed projections to serve hundreds of millions of rows with sub-second latency. On top of this data layer, Cogent implemented an agentic text-to-SQL system that iteratively queries live data, references an in-house Ontology Service for semantic context, and applies majority-voted evaluations. This combination improved text-to-SQL accuracy from 40% to 94% while providing sub-three-second response times across 98.7% of queries at 100 million rows.
Build a Domain-Specific Embedding Model in Under a Day
The tutorial presents a pipeline for adapting a general-purpose embedding model when it misses fine-grained distinctions in domain documents and labeled query–document data is unavailable. It generates synthetic question–answer pairs with NeMo Data Designer, filters them for quality, mines hard negatives with a 95% positive-score margin ceiling, and unrolls multi-hop queries for contrastive training. The recipe fine-tunes Llama-Nemotron-Embed-1B-v2, evaluates base and adapted models with BEIR metrics, and exports the result to ONNX or TensorRT. On the reported datasets, it improved Recall@10 and NDCG@10 by more than 10%; Atlassian increased Recall@60 from 0.751 to 0.951 on JIRA data. The complete workflow can run in roughly 2–3 hours for about 500 documents, using a single 80GB Ampere-or-newer GPU.
Building a Magic Mirror: AI retail experiences with Remix
Shopify’s “magic mirror” is an AI-powered retail installation that uses a display and hidden webcam to deliver personalized messages, animations, and product interactions in under a minute. A Remix route handles the browser and server boundary: a button-triggered POST captures an image, sends it with a curated prompt to a vision-capable language model, and renders a compliment plus a makeup recommendation; server-sent events optionally let a remote device trigger or reset the experience. The prompt combines persona and safety constraints, a plain-text product and inventory table, and a required JSON schema, while strict product-ID lookup provides graceful degradation. The interface uses explicit states, delayed capture, timed response and reset screens, and animated transitions. The production setup keeps brand configuration, shared components, actions, and SSE infrastructure reusable across mirror modes.
Introducing AI observability notebooks for Managed ClickStack in Private Preview
ClickHouse has launched AI Notebooks for Managed ClickStack in private preview to assist site reliability engineers during system investigations. Instead of isolating interactions in a standard chat window, the platform embeds Anthropic's Claude models directly into a cell-based notebook canvas. The system connects the model to internal ClickStack APIs, translating natural language questions into structured queries that leverage ClickHouse optimizations like materialized views and query chunking. Engineers can inspect reasoning, edit generated SQL queries, add manual visualizations, and branch investigations to evaluate multiple hypotheses independently. While currently utilizing a last-write-wins model for shared canvases, the roadmap includes true concurrent editing, customizable context, and external tool integration via an MCP server.
How we optimized Dash's relevance judge with DSPy
Dropbox Dash optimized its LLM-as-a-judge relevance scoring system using DSPy to migrate from expensive proprietary models to cheaper open-weight alternatives. By establishing automated feedback loops based on human agreement and strict JSON format validation, the team systematically generated robust prompts for new models. This reduced human-score disagreement by 45% and slashed model adaptation time from weeks to days while enabling 10x to 100x more data labeling.
Holotron-12B - High Throughput Computer Use Agent
Holotron-12B is a multimodal computer-use model released by H Company and available on Hugging Face. Post-trained from NVIDIA’s Nemotron-Nano-12B-v2-VL-BF16 on proprietary data, it targets agents that must perceive, decide, and act in interactive environments while handling long contexts with multiple images. Its hybrid State-Space Model and attention architecture stores a constant state per layer per generated sequence instead of attention’s per-token key and value activations, reducing memory demands for long-context inference. In a WebVoyager evaluation on one H100 with vLLM, it reached 8.9k tokens/s at concurrency 100 versus 5.1k for Holo2-8B, and its benchmark score rose from 35.1% to 80.5% after training on localization and navigation data.
AI is redrawing the database market
Emerging artificial intelligence workloads are fundamentally transforming data platform requirements across real-time analytics, data warehousing, and observability. Conversational analytics and autonomous agents replace periodic batch queries with rapid bursts of concurrent, low-latency requests, exposing the architectural limits of traditional cloud data warehouses and standalone transactional databases. To resolve these constraints, architectures are converging on engine-level integrations between Postgres for transactions and ClickHouse for analytical scale, while unifying operational observability and business event storage. ClickHouse also incorporates turnkey agent interfaces through its acquisition of LibreChat alongside dedicated execution tracing from tools like Langfuse. These combined capabilities ensure organizations can reliably support interactive agentic workflows without incurring severe latency penalties or compounding infrastructure costs.
AI doesn’t always generate perfect ClickHouse schemas (yet)
LLMs can generate syntactically valid ClickHouse schemas whose advanced choices are poorly matched to a workload. The post examines recurring pitfalls including query-driven partitioning, per-column codecs, premature projections, materialized-view sprawl, JSON configurations, and forced deduplication with OPTIMIZE TABLE ... FINAL. It explains that partitions primarily support data management, incremental materialized views add work to every insert, and ReplacingMergeTree deduplicates during background merges rather than at insertion. The recommended method is to begin with a simple MergeTree or ReplacingMergeTree, a well-chosen ORDER BY, default compression, and no optional structures, then measure production behavior before adding complexity or seeking human expertise for high-scale, interconnected designs.
Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries
At-scale reinforcement-learning training is constrained by long, variable data-generation workloads that leave training GPUs idle, especially for reasoning and agentic models. The surveyed architecture separates inference and training onto distinct GPU pools, connects them through a rollout buffer, and synchronizes weights asynchronously so both stages can proceed concurrently. Sixteen open-source libraries are compared across orchestration, buffering, weight synchronization, staleness, partial rollouts, LoRA, and distributed backends; Ray appears in 8 of 16 orchestration implementations, while NCCL broadcast is the default weight-transfer method. The proposed TRL async trainer favors a lightweight bounded queue with per-token model_version metadata, packed NCCL transfers, and experimental support for prefix-resume or abort-and-retry handling of rollouts interrupted by weight updates.
Financial Benchmarks
Ramp describes how it benchmarks large language models (LLMs) used in financial products against day-to-day tasks such as invoice extraction, financial-statement OCR, policy review, accounting autocoding, compliance detection, and fund routing. These benchmarks combine task-specific metrics with cost, latency, reasoning effort, human decisions, historical context, and ground-truth datasets where available. For contextual invoice OCR, perfect extraction requires every field to match the user's final bill, while financial-statement OCR uses a 1% relative-error threshold against transcriptions from over 500 real P&L documents. Results show application-specific trade-offs: Gemini 3 Flash is described as a cost-efficient leader for several visual and financial tasks, while Claude models lead some high-accuracy or low-miss-rate settings, and behavior can differ within one provider. The framework emphasizes Pareto trade-offs and continuous testing, with plans to expand from benchmarking current performance to hill climbing on capabilities.
Working at the intersection of data and AI with Kenza Boulisfane
Thumbtack software engineer Kenza Boulisfane develops an artificial intelligence tool called the Marketing Analytics Agent to democratize access to complex cross-channel marketing data. Non-technical staff across Data Science, Growth, Brand, and Product can ask natural language performance queries without requiring SQL expertise or manual reporting. A primary technical challenge involves unifying fragmented data across multiple advertising platforms that define metrics differently while maintaining data reliability. The system interprets natural language queries, analyzes underlying performance metrics, and translates results into actionable, data-backed recommendations for next steps. Boulisfane also notes that virtual work environments enhance collaboration by encouraging clearer documentation, intentional communication, and focused individual productivity.
Cursor is now available in JetBrains IDEs
Cursor is now available in IntelliJ IDEA, PyCharm, WebStorm, and other JetBrains IDEs through the Agent Client Protocol (ACP), extending its agent-driven development experience to developers using those environments. The integration lets users choose frontier models from OpenAI, Anthropic, Google, and Cursor, while Cursor provides a custom-built agent harness for each model. Cursor's secure codebase indexing and semantic search are combined with JetBrains IDE code intelligence and tooling to support work on large enterprise codebases. Users install Cursor ACP from the ACP Registry and authenticate with an existing Cursor account; it is free for users on paid plans, and the release is described as a foundation for deeper JetBrains integrations.
2,000 robots walk into a shop: Simulated A/B testing (2026) - Shopify
Shopify’s SimGym addresses the difficulty of evaluating storefront changes when merchants lack enough traffic for conventional A/B tests, by simulating shopping sessions with LLM-guided robots. Each robot runs in a cloud Chromium browser with a persona, budget, and intent, while a counterpart tests the alternate theme; the system uses real page representations rather than a mocked DOM. To support bursty, sequential workloads with 89K–127K accumulated tokens per session, Shopify self-hosted gpt-oss-120b on 48 NVIDIA B200 GPUs, combining MXFP4 expert quantization, FP8 KV-cache, FlashInfer kernels, async scheduling, and guided decoding. Production measurements report 80K to 88K tokens/second per B200 from async scheduling, a 57% reduction in HTTP/gRPC overhead, and MIG experiments yielding about 20% lower latency and 12% higher daily throughput without quality regression; speculative decoding is validated but queued.
Using LLMs to amplify human labeling and improve Dash search relevance
Dropbox Dash uses large language models (LLMs) to amplify human labeling efforts for training its search relevance and ranking models. By validating and optimizing LLM evaluators against a small set of human-labeled internal data, Dropbox creates massive, high-quality training datasets offline for production rankers like XGBoost without incurring high latency or latency costs at query time.
Mixture of Experts (MoEs) in Transformers
Mixture of Experts (MoEs) in Transformers let a Transformer retain its backbone while replacing selected feed-forward layers with routed expert sub-networks, allowing total model capacity to exceed the active parameters used per token. The post explains why sparse models can reduce inference cost and memory pressure, then focuses on making them first-class in Transformers. Its weight-loading refactor converts independently serialized expert tensors into packed runtime layouts through composable WeightConverter operations and lazy materialization. The Experts Backend supplies eager, batched_mm, and grouped_mm execution strategies, while expert parallelism shards experts across devices and aggregates partial outputs. Training support uses grouped GEMM and custom Triton plus LoRA kernels, with reported improvements of approximately 12× speed, over 35% VRAM reduction, and roughly 6× longer context.
Innovating with MongoDB | Customer Successes, February 2026
Modern artificial intelligence startups frequently face operational bottlenecks when deploying autonomous agents and dynamic interfaces on rigid relational schemas or restricted database engines. To address these limitations, platforms including Modelence, Thesys, Emergent Labs, and Heidi integrated MongoDB Atlas as their operational data foundation. Emergent Labs eliminated recursive schema migration failures previously encountered with PostgreSQL by adopting document structures that match native JSON generation, supporting millions of deployed applications. Meanwhile, Heidi migrated from Amazon DocumentDB to utilize MongoDB Vector Search for retrieval-augmented generation pipelines under a single unified API without requiring external search databases. These architectural transitions streamlined continuous agent iterations, eliminated operational scaling downtime, and accelerated deployment cycles across diverse production environments.
Detecting and preventing distillation attacks
Anthropic reports identifying industrial-scale distillation campaigns by DeepSeek, Moonshot, and MiniMax that generated more than 16 million Claude exchanges through approximately 24,000 fraudulent accounts. The campaigns used proxy services and coordinated account networks to extract agentic reasoning, tool use, coding, computer-use, and other capabilities, including chain-of-thought data and reinforcement-learning tasks. Anthropic attributed the operations with high confidence using IP correlation, request metadata, infrastructure indicators, and partner corroboration, while describing hydra cluster architectures that replace banned accounts and obscure patterns. The response combines classifiers, behavioral fingerprinting, coordinated-activity detection, intelligence sharing, stronger account verification, and product, API, and model-level countermeasures. The report argues illicitly distilled models may strip safeguards and undermine export controls, and says coordinated action across AI labs, cloud providers, and policymakers is required.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.