Activation Steering: One Dial, Two Costs
Activation steering edits a model's internal activations to add or remove a concept at inference time. This explainer maps its one control knob, its two costs, and when it beats prompting.
Loading…
Topic
LLMs
648 posts about LLMs. Every summary links to the original.
- 648 posts
- 49 companies
- 5 explainers
See the idea
LLMs explainers
Antibodies as a Prediction Pipeline: Where AI Models Break Down
Antibody discovery is a three-stage prediction pipeline, and AI models that excel on familiar targets degrade sharply on unfamiliar ones. This explainer maps where that gap appears and what it costs.
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
From Rigid Rules to Empathetic Agents: Rebuilding Swiggy’s Order Cancellation Experience with LLMs
Swiggy rebuilt its “I Want To Cancel” (IWTC) support journey because a roughly 150K-line deterministic decision tree treated cancellation keywords alike, repeated canned deterrence, and could leave users stuck without handoff. The replacement is a stateful, multi-turn framework orchestrated with LangGraph: an intent layer identifies needs such as delay help, coupon support, direct cancellation, or human access, while live order signals and deterministic policy checks determine the next action. Full graph state is checkpointed in Redis by order- and session-level thread IDs, allowing conversations to resume across turns. Swiggy evaluated intent reliability using more than 100k historical chats and a pre-deployment suite of about 500 representative queries, then operated the system with reporting and alerts for behavioral and service metrics. The conclusion favors a hybrid design in which LLMs handle language while deterministic systems anchor truth and business decisions.
Explaining Every Rupee: How We Built Reliable LLM Support Bots for Delivery Partners
Swiggy faced close to 9 lakh monthly payout- and incentive-related tickets, where delivery executives needed accurate explanations of earnings, deductions, missed incentives, and credited amounts. The system separates financial truth from language generation: deterministic preprocessing and internal APIs resolve incentive eligibility, payout breakdowns, deductions, and conditions before an LLM explains the result. Two multilingual bots—the Daily Incentive Bot and Payout Bot—use a LangGraph StateGraph with LLM, tool, and validator nodes, Redis checkpointing for multi-turn context, and post-processing translation into the executive’s preferred language. Azure Translator preserve-list controls help keep business-critical terminology consistent, while structured escalation sends out-of-scope cases and context to human agents. Moving retrieval from Databricks SQL queries to an aggregated internal API reduced latency from 8–10 seconds to milliseconds; the bots now resolve about 90% end to end, cut resolution from roughly 20 minutes to seconds, and improved CSAT by 40%.
Beyond code generation: rethinking engineering productivity in the age of AI agents
Dropbox shares how widespread AI code generation shifts software development bottlenecks downstream into code review, CI infrastructure, and validation pipelines. To adapt, they built Nova, an internal coding agent platform that safely automates scoped tasks such as migrations and flaky test remediation. They also evolved their developer productivity framework to measure end-to-end customer impact and code quality rather than simple pull request throughput.
Slack AI: The Path to Multi-Cloud
Slack evolved its Slack AI serving infrastructure across multiple phases to handle enterprise LLM workloads reliably and securely. The initial deployment on AWS SageMaker provided zero-knowledge escrow VPC isolation and FedRAMP compliance, but engineers faced scaling latency, GPU scarcity, and significant operational overhead. Slack then migrated live traffic to Amazon Bedrock to leverage managed Model Units and eliminate model release lag without customer-facing incidents. However, fixed Provisioned Throughput commitments and regional peak traffic variations created persistent underutilization challenges. Consequently, Slack expanded into a multi-cloud orchestration architecture that normalizes disparate provider APIs, integrates unified cross-cloud telemetry, and routes traffic dynamically around latency spikes and outages.
How we cut build times by two-thirds by deleting our CMS
Sentry rebuilt its marketing site and blog of roughly 2,500 pages, replacing a legacy Gatsby setup and traditional headless CMS with Astro, Markdown, Frontmatter, and AI-driven automation. The previous architecture suffered from 14-minute build times, restrictive CMS schema limits, and frequent build failures caused by external API dependencies. A team of 2.5 developers executed the migration in two months using Claude Code agents for data extraction, template translation, and testing. To manage content without a CMS dashboard, Sentry built Claude Skills to guide updates, run local previews, and draft GitHub pull requests. The migration reduced average build times to under 4 minutes, cut broken staging builds by 95 percent, and improved Web Vitals scores from 89 to 97.
Under the River
Shopify describes how its AI agent, River, grew from an infrastructure bet aimed at making a large, fragmented codebase usable by AI. In early 2024, the company adopted a monorepo called World and Nix as a reproducible substrate for development, CI, and production, despite substantial migration costs. River operates in public Slack channels, where it reads code, runs tests, queries data, inspects production traces, and opens pull requests; in a recent 30-day period it handled 59,918 sessions and contributed to 3,536 merged pull requests. Underneath it, Aquifer provides durable sessions, sandboxes, gateways, event logs, credentials handling, and observability, allowing agent profiles to share one platform across interactive, automated, and batch modes. The central design principle is that the session must survive process and machine failure, while public transcripts create a compounding knowledge corpus.
Open House 2026 Day 2: 10 years of open source and what the ecosystem is building next
Open House 2026 Day 2 recaps ClickHouse’s tenth year as an open-source project and the ecosystem forming around its data platform. The event reported more than 2,600 contributors, nearly 48,000 GitHub stars, over 200 integrations, and the launch of the House Mates partner program. Announcements included Fivetran’s generally available ClickHouse Cloud destination, a Google Cloud Dataflow Pub/Sub template, Sigma’s public-beta connector and ETL Cache Layer, and a ClickHouse dbt Fusion adapter available in alpha through the CLI. The recap also describes MCP and agent-oriented observability work, including a ClickStack MCP server that translates investigations into optimized ClickHouse queries. Internal testing reported up to 25% fewer tool calls, 2.5 times greater consistency, and nearly 20% better evaluation results.
Open House observability announcements: MCP server, AI Notebooks, and ClickStack Cloud
At Open House, ClickHouse announced three observability updates: ClickStack Cloud entered private preview, AI Notebooks entered beta for Managed ClickStack, and a ClickStack MCP server was introduced in open-source ClickStack. ClickStack Cloud is a fully managed, serverless platform that accepts OpenTelemetry data through a managed endpoint, while Managed ClickStack is generally available for teams controlling collectors, ingestion, compute sizing, schema design, and tuning. AI Notebooks provide a persistent, shareable investigation workspace in which prompts, queries, charts, reasoning steps, findings, branches, intermediate results, and generated queries remain visible and editable. The MCP server exposes the investigative tools used by Notebooks to external agents, routing requests through optimized searches, aggregations, and visualizations rather than relying entirely on ad hoc SQL generation; it can also persist findings in dashboards. Together, the announcements present a choice between managed convenience, operational control, and programmable observability workflows.
Reliable LLM Inference at Scale
Databricks describes an inference platform serving open-source and proprietary frontier models at more than 125T tokens per month, with reliability as the central challenge under spiky, multi-tenant demand. It identifies failures and latency risks from expensive GPU configurations, variable input and output lengths, long-context requests, multimodal preprocessing, and engine-level hangs. Its architecture combines rate limiting, capacity management, model units, Dicer load-aware sticky routing, and autoscaling; model units estimate multidimensional request cost and allocate capacity more predictably. For runtime reliability, prioritized black-box health checks trigger Kubernetes liveness restarts for silent hangs, completing detection and recovery in less than five minutes while eliminating false probe failures. Image-processing changes, including Torchvision processors and OMP_NUM_THREADS configuration, removed CPU throttling and increased completed requests per second by more than threefold with the same replicas and load; bursty workloads also achieved over 80% GPU savings versus static peak provisioning.
Thank you for building with us
At its 2026 user conference in San Francisco, ClickHouse reports surpassing 4,000 customers and $250 million in annual run-rate revenue, while marking nearly ten years since Alexey Milovidov open-sourced the database. The company attributes its growth to a community of contributors and describes use across AI products, financial services, observability, analytics, and infrastructure. New offerings include ClickHouse Agents, a fully managed agentic analytics service powered by Anthropic’s Claude; CostBench, an open benchmark using each warehouse’s real billing model; and House Mates, a partner program launching with more than 50 partners across six continents. Additional updates include Managed Postgres in public beta, Managed ClickStack and Langfuse for AI-era observability, generally available full-text search, and automatic JOIN optimization, with cost-performance identified as the central constraint for high-concurrency, low-latency workloads.
ClickHouse tops $250M ARR and 4,000 customers, launches Claude-powered agents at Open House 2026
At Open House 2026, ClickHouse announced that its serverless cloud offering exceeded $250 million in annual run-rate revenue and reached 4,000 customers, with ARR more than tripling year-over-year and more than 1,000 net new customers added since January. The company launched ClickHouse Agents, a fully managed, no-code agentic analytics service powered by Anthropic’s Claude, with a chat interface, sandboxed code interpreter, artifacts, skills, memory, multi-agent workflows, and MCP-compatible integrations. It also published CostBench, an open benchmark applying each vendor’s real compute billing model to the same workload; ClickHouse Cloud was reported to have 23x better cost-performance than the nearest cloud data warehouse. Other announcements included Managed Postgres in public beta, managed ClickStack, generally available full-text search, cross-region replication, and the House Mates partner program.
amazon.science
Amazon Research Awards recipients announced
Amazon Research Awards has announced 70 award recipients across 49 universities in 11 countries for its fall 2025 funding cycle. The funded research spans six distinct tracks: AI for Information Security, Agentic AI, Automated Reasoning, AWS Cryptography, Cybersecurity and Anti-Abuse Technologies, and Sustainability. Each selected academic team receives unrestricted financial grants, AWS Promotional Credits to access specialized cloud AI and machine learning services, and access to more than 700 Amazon public datasets. Awardees are paired with an Amazon research contact who provides advisory support, consultation, and access to company training sessions. Furthermore, the program explicitly encourages recipients to publish their discoveries, deliver presentations at Amazon offices worldwide, and release associated software implementations under open-source licenses.
Reachy Mini goes fully local
Reachy Mini’s conversation app can now run its speech stack locally instead of sending audio to a server, avoiding cloud, API keys, and data leaving the machine. The tutorial uses speech-to-speech as a cascaded VAD → STT → LLM → TTS pipeline exposed through a Realtime API-compatible /v1/realtime WebSocket, with Silero VAD v5, Parakeet-TDT 0.6B v3, Qwen3-TTS, and a selectable LLM backend. It shows llama.cpp serving Gemma 4 E4B with parallel slots, 64k context, flash attention, and full sliding-window attention, then covers vLLM, MLX, Transformers, hosted Responses API endpoints, and network connection to a wireless robot. Once the backend and app are connected, conversations run through the local loop, while individual stages can be exchanged to trade latency, quality, language coverage, and hardware choices.
Open House 2026 Day 1: Real-time data without lock-in and what teams can build next
ClickHouse’s Open House 2026 Day 1 announced updates spanning ClickHouse Cloud, Postgres, distributed query execution, observability, AI agents, and lakehouse interoperability. ClickHouse Postgres entered public beta with local NVMe storage, high availability, S3-backed point-in-time recovery, and a native CDC path into ClickHouse; early benchmarks reported more than 5x the transactions per second of AWS RDS. Multi-stage distributed queries repartition intermediate data between stages to scale joins and high-cardinality aggregations, reducing TPC-H SF100 runtime from 117.6 seconds on one node to 54.7 seconds on eight nodes. Additional launches included a Claude-powered ClickHouse Agents public beta, an AI-enhanced query builder, Monitoring v2, and Langfuse V4’s reported 200x query performance improvement, while the wrap-up framed the releases around real-time systems without lock-in.
Beyond the Menu Tree: How Yelp Built a Smarter Customer Success Chatbot with AI
Yelp transitioned from a legacy customer support chatbot that relied on rigid phrase matching and static menus to a retrieval-augmented generation system. The updated architecture classifies incoming queries using a large language model to route users into one of five specialized workflows, including QA, Billing, Refund, Cancel, and Review. For general QA, the system uses FAISS to search an in-memory vectorstore built exclusively from article metadata such as titles, summaries, and headers rather than raw article text. Retrieved articles are supplied to the language model as prompt context, followed by automated validations for safety, character limits, and hallucinated hyperlinks. In production A/B testing, this metadata-driven design doubled the chatbot resolution rate while achieving approximately 94% recall@5.
Shaping Product Understanding with Contrastive Reinforcement Learning
Etsy’s search and recommendation systems need product representations that capture distinguishing details—such as materials, craft, and visual style—beyond categories and titles, despite noisy listing data, varied inventory, and latency constraints. The approach fine-tunes the open-source Qwen3-VL-8B on query, engaged-listing, and non-engaged-listing triplets, generating candidate summaries and scoring them with a frozen two-tower dense retriever. Group Relative Policy Optimization (GRPO) uses the query–engaged versus query–non-engaged similarity margin as a joint reward, encouraging grounded summaries that separate alternatives rather than generic keyword stuffing. Human review found specific, accurate details beyond titles, while adding the summaries to Etsy’s Semantic Relevance model improved macro F1 by 8.7% relative to description n-grams. Online production experiments are planned.
Your agent can't fix what it can't see
Coding agents frequently fail to resolve production bugs when restricted to source code alone, as they lack critical runtime context such as stack traces, breadcrumbs, and environment data. To bridge this gap, Sentry introduces Sentry MCP and the Sentry CLI to supply agents and developers with structured production telemetry. The Model Context Protocol integration supports clients like Claude Code, Cursor, and VS Code, while the CLI enables scripted workflows. When an alert triggers, an agent uses trace and commit metadata to diagnose root causes across service boundaries, generate regression tests, and open draft pull requests. Over six months, the Autofix pull request merge rate increased from 41% to 46%.
amazon.science
Diverse reasoning traces teach LLMs to make better decisions
Large language models typically rely on supervised fine-tuning with a single human-verified reasoning trace per example, which limits reasoning breadth and causes mode collapse when attempting parallel reasoning. Researchers presented set-supervised fine tuning (SSFT) and global forking policy optimization (GFPO) to teach models diverse problem-solving strategies. SSFT represents reasoning as a set of complete solution paths paired with global forking tokens, utilizing bipartite matching to achieve order-invariant training and specialized tokens. GFPO applies lightweight reinforcement learning to select the optimal forking token upfront, detaching gradients from generated rollouts to ensure stable optimization. On standard reasoning and coding benchmarks, the combined framework increased single-shot accuracy by 5% to 7% while improving pass@k diversity without degrading pass@1 accuracy.
Harness, Scaffold, and the AI Agent Terms Worth Getting Right
AI-agent terminology has outpaced shared definitions, leaving terms such as harness and scaffold blurred across frameworks and contexts. The glossary separates the model, which produces text but has no memory or execution loop, from scaffolding, which defines instructions, tools, output parsing, and context management, and the harness, which runs model calls, routes tool use, and decides when to stop. It presents an agent as a model plus the surrounding system that lets it act repeatedly, while distinguishing tools, skills, sub-agents, and orchestrators by their roles. For training, it defines the environment, trainer, rollout, and reward, including verifiable or learned scores and sparse or dense feedback. Because usage is not universal, the glossary offers a practical mental model rather than enforcing one vocabulary.
AI as the Next Abstraction Layer: How I see engineering evolving at Thumbtack
Thumbtack is transitioning its engineering workflow from ad-hoc tool experimentation to end-to-end agentic software development. Unlike previous deterministic abstractions such as compilers or high-level languages, artificial intelligence introduces non-determinism where identical prompts can produce varying code and trade-offs. Because consumer marketplace experiments still require weeks for behavioral signals to stabilize, accelerated implementation primarily helps teams prepare cleaner experiment designs and parallelize hypotheses within experimentation bandwidth constraints. Consequently, day-to-day engineering responsibilities shift upstream from manual coding toward problem framing, system architecture, validation, and co-designing features with product managers. Teams maintain technical sharpness by rigorously reviewing generated code while accepting that long-term specialization above the abstraction layer will become standard across the organization.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.