Activation Steering: One Dial, Two Costs
Activation steering edits a model's internal activations to add or remove a concept at inference time. This explainer maps its one control knob, its two costs, and when it beats prompting.
Loading…
Topic
LLMs
648 posts about LLMs. Every summary links to the original.
- 648 posts
- 49 companies
- 5 explainers
See the idea
LLMs explainers
Antibodies as a Prediction Pipeline: Where AI Models Break Down
Antibody discovery is a three-stage prediction pipeline, and AI models that excel on familiar targets degrade sharply on unfamiliar ones. This explainer maps where that gap appears and what it costs.
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Adding payments to your LLM agentic workflows
Agentic workflows combine large language models with function calling to turn prompts into sequences of actions, but connecting those actions to financial services requires controlled API access. The Stripe agent toolkit integrates Stripe with Vercel’s AI SDK, LangChain, and CrewAI, supports function-calling LLM providers, and is built on Stripe’s Node.js and Python SDKs. It exposes configurable tools for tasks such as invoicing, creating and sharing payment links, and using Issuing virtual cards with spending limits, authorization checks, and deactivation after purchase. Middleware can record input and completion token counts as billing events for usage-based customer billing. Because agent behavior is non-deterministic, the post recommends test mode, evaluations, restricted keys, limited tool sets, and reduced request and response shapes to improve focus and constrain failures.
LLM-assisted vector similarity search
Vector similarity search can struggle with queries containing negation, constraints, or conceptual requirements, even though it efficiently retrieves semantically similar records. The proposed method uses FAISS and OpenAI text-embedding-ada-002 embeddings to shortlist 15 candidates, then asks GPT-4o to rank the candidates against the original query and return the top three. On two GPT-4o-curated synthetic datasets—100 food dishes and 100 Asian tourist spots—the assisted method matched raw search on straightforward queries but produced more relevant results for complex ones, including excluding fish or shrimp and identifying wildlife-related places. The authors note limits from the small datasets and query set, report similar relevance improvements in an internal deployment involving more than 4,500 relational-database rows, and identify shortlist size and added LLM latency as areas for further evaluation.
Leveraging RAG-powered LLMs for analytical tasks
Data analysts at Grab faced an increasing volume of stakeholder data requests that required manually writing and executing repetitive SQL queries with minor parameter changes. To automate routine tasks like metric reporting and fraud investigations, the Integrity Analytics team paired an internal prompt-management platform, Spellvault, with Data-Arks, an in-house Python-based middleware that packages SQL queries and Python functions into APIs. A scheduler triggers automated report summarization, while an analytical bot named A* bot executes relevant investigation queries via RAG and returns summarized findings in Slack. Automated report generation saves an estimated three to four hours per report and shortens multi-query fraud investigations to minutes. Grab favored RAG over fine-tuning because it avoids continuous retraining costs, scales faster across diverse use cases, and surfaces current production data.
Enabling conversational data discovery with LLMs at Grab
Grab faced major bottlenecks in dataset discovery across its 200,000 data lake tables due to low documentation coverage and vanilla Elasticsearch limitations in its internal tool, Hubble. To resolve this, engineers first tuned Elasticsearch by boosting frequently queried P80 tables and certified schemas while deboosting deprecated datasets. They then deployed a GPT-4 pipeline that generated documentation from table schemas and sample data, raising P80 documentation coverage from 20% to roughly 90%. Finally, the team developed HubbleIQ, an LLM-powered assistant built via Glean Apps and integrated into Slack and Hubble search to answer semantic queries. These initiatives increased search click-through rates to 94% and lifted the proportion of users finding discovery easy to 73%.
Chimera Sandbox: A scalable experimentation and development platform for Notebook services
Grab's Chimera team built Chimera Sandbox to address resource accessibility challenges and enable rapid machine learning experimentation across technical and non-technical teams. The platform runs on multi-cluster Kubernetes infrastructure capable of scaling to thousands of nodes per cluster during peak loads. Users can provision customizable Docker environments with varying CPU and GPU configurations, collaborate in real time, and query data using Spark-on-Kubernetes integrations and helper magic cells. The workspace integrates with Grab's AI Gateway for LLM access alongside tools like MLflow, Optuna, RStudio, and LabelStudio on shared storage, backed by user-level cost attribution.
How we improved translation experience with cost efficiency
Grab faced translation quality and cost challenges in its booking chat system when tourism resumed after COVID restrictions eased. Device language settings proved unreliable for determining message language, prompting the team to use Lingua alongside booking context heuristics and third-party fallbacks for real-time detection. To replace expensive and inaccurate general translation APIs, Grab distilled a large open-source Hugging Face model into lightweight, language-specific models trained on ten million synthetic chat examples. A post-translation validation step verifies non-translatable entities such as numbers and emojis before falling back to external services if mutations occur. Caching layers were also introduced across translation paths to curb redundant on-the-fly execution.
LLM-powered data classification for data entities at scale
Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.
More problems
The post outlines research problems for the next phase of AI programming, centered on making coding systems predict actions, perform larger edits, understand codebases, and debug software. Proposed directions include low-latency next-action models for keystrokes, locations, files, terminal commands, and edits; background execution for “perfect edits” and pseudocode compilation; and multi-file changes supported by stronger tool use, reasoning, and agent trajectories. For context, retrieval alone is described as insufficient for codebase questions requiring multiple hops, while useful context may span documentation, source, history, UI pixels, logs, and messages. The post recommends combining retrieval, recurrence, and long-context attention, alongside specialized rerankers, multi-hop embedders, caching, and codebase-level retrieval. For reliability, it proposes calibrated bug detection, always-on AI linting, runtime-state tracking through cursor/debug, and improved debugger tool use, while noting trade-offs involving latency, false positives, infrastructure, and user experience.
Editing Files at 1000 Tokens per Second
Large code edits expose coding agents to laziness, inaccuracy, and latency, especially when frontier models must rewrite hundreds of lines. The post presents fast apply, a specialized full-file code-edit model that separates planning from applying: Cursor's frontier model plans the change, while the apply model rewrites the current file. It evaluates roughly 450 edits on files under 400 lines, using Claude-3 Opus as grader, and finds llama-3-70b-ft nearly matches claude-3-opus-diff while outperforming GPT-4-Turbo and GPT-4o. Rather than generate conventional diffs, it uses full-file rewrites and search/replace hunks, which avoid line-number errors and give models more output tokens for reasoning. A custom speculative decoding method, deployed with Fireworks, reaches approximately 1,000 tokens per second on a 70b model, about 13x faster than vanilla Llama-3-70b inference and 9x faster than the previous GPT-4 deployment.
Picture This: Open Source AI for Image Description
Nolan Darilek presents a self-hosted image-description service intended to help blind screen-reader users understand photos, diagrams, menus, and other visual material. Humans often provide inadequate alt text, and the post contrasts that history with increasingly capable LLM-generated descriptions from apps and browser add-ons. The implementation combines Ollama, running the Apache-licensed LLaVA 34B multimodal model, with a PocketBase API over SQLite and a small Python client; collection hooks send images and follow-up questions, retain chat history, and enforce per-user access rules. In testing, a cold request on an A100-40GB Fly Machine took about 45 seconds, while stopping idle GPU Machines reduced resource use at the cost of startup delay. The example produced useful scene descriptions but could not reliably identify leafless tree species, illustrating both the prototype’s value and its limitations.
NeurIPS 2023: Our Favorite Papers on LLMs, Statistical Learning, and More
Researchers reviewed prominent machine learning papers presented at NeurIPS 2023 covering large language models and statistical learning theory. One investigation showed that claimed emergent abilities in models such as GPT-3 often result from nonlinear evaluation metrics rather than fundamental shifts in model capability. To reduce the computational burden of model adaptation, QLoRA enables 65-billion-parameter model fine-tuning on a single 48-gigabyte GPU via 4-bit NormalFloat quantization and paged optimization. Direct Preference Optimization eliminates complex reward modeling by casting reinforcement learning from human feedback into a preference classification task. Additional work resolved statistical anomalies like double descent using effective parameter counts and introduced stochastic gradient approximations for Gaussian processes.
Cutting Through the Noise: Three Things We've Learned About Generative AI and Developer Productivity
eBay describes three tracks for generative AI developer productivity: adopting a commercial offering, fine-tuning an open-source LLM on internal data, and building a knowledge base. Because no single metric is reliable, the exercise combines surveys with Git, DORA, Flow, code-quality, and performance measures. In a 300-developer Copilot A/B test, the treatment group had 27% code acceptance, 17% faster pull-request creation-to-merge time, and 12% lower Lead Time for Change; Sonar quality was unchanged. eBayCoder, based on Code Llama 13B and trained on eBay code and documentation, is presented as useful for software upkeep and broader context. Its internal GPT uses retrieval-augmented generation to answer questions from distributed sources, but early results remain preliminary and nonsensical answers require feedback and RLHF.
How Yoko Li makes towns, tamagoes, and tools for local AI
Yoko Li’s local AI work spans playful agent experiments and practical tools for running language-model workflows on personal hardware. AI Town uses conversations among AI agents to produce changing relationships and emergent behavior, while AI Tamago feeds game state and user inputs to an unmodified LLaMA 2 7B model and uses Ollama’s JSON mode to generate pet actions. The Local AI Starter Kit packages document ingestion, vector storage, and language-model responses into a private, local “chat with a knowledge base” pipeline, avoiding setup across at least four API providers. The post also describes unreliable JSON generation, hallucinations, and grammar-guided workarounds, concluding that randomness is a liability for factual outputs but useful for game behavior and creative experimentation.
Scaling Large Language Models to zero with Ollama
Running GPU nodes for language-model inference can be costly because they often sit idle between requests, so this guide configures Ollama on a Fly.io Machine that scales to zero when idle. Ollama is described as a wrapper around llama.cpp, enabling self-hosted models with GPU acceleration, while about 70 lines of Go add authentication to protect the endpoint from unauthorized use. Deployment uses an A100-40GB GPU, a private Flycast address, automatic start and stop settings, and a 100GB persistent volume mounted at /root/.ollama so downloaded models survive stopped Machines. A stopped Machine no longer incurs GPU charges, but scaling to zero suspends it on the same host and may fail to wake during capacity issues. The setup supports CLI and API requests, WireGuard access, custom Modelfiles, and JSON-mode function-calling workflows.
Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling
Nextdoor implemented a generative AI framework to improve email notification subject lines for its New and Trending emails. Using off-the-shelf ChatGPT API outputs directly resulted in marketing-like phrases, hallucinations, and lower click-through rates compared to original user-generated post snippets. To resolve this, the team prompted the language model strictly to extract authentic post phrases without rewriting, eliminating hallucinations while preserving the original tone. Nextdoor then introduced a rejection sampling mechanism featuring a dedicated reward model that predicts user engagement and accepts generated subjects only when their score exceeds the user-written baseline. This combined system lifted sessions by 1%, boosted Weekly Active Users by 0.4%, and increased advertising revenue by 1%.
Sidekick’s Improved Streaming Experience
Sidekick addresses two streaming problems in LLM chatbots: Markdown fragments can render awkwardly before syntax is complete, and tool-driven responses can delay visible output across multiple LLM roundtrips. Its solution combines a buffering Markdown parser with an event emitter that multiplexes the main response and asynchronously resolved content into one stream. A Node.js Transform stream processes Unicode characters through a finite-state machine, buffering ambiguous sequences such as emphasis, list markers, and links until they resolve or are flushed as raw text. The parser also recognizes card links, triggering asynchronous card-resolution tasks while Server-Sent Events carry named streams to the UI, which demultiplexes them into the main response and placeholders. The design uses Markdown as a human-readable transport between the LLM, backend, and frontend, allowing content to appear as it is produced.
Supabase Launch Week 8 Hackathon
Supabase Launch Week 8 will run an asynchronous hackathon for new Open Source projects that use Supabase in some capacity. The event is listed as starting at 09:00 am PT on Friday 4 August 2023, with the main schedule ending on Sunday 13 August at 11:59 pm PT; the submission instructions separately give a deadline of 11:59 pm PT on Sunday 14 August 2023. Individuals or teams of up to five can build an app, tool, library, or other project in any language or framework, with all design and code created during the event. Judges will award winners and runners-up across five categories, including overall quality, AI use, technical impressiveness, visual appeal, and fun, while submissions must include source code and may earn extra points for a demo video.
Inference characteristics of Llama
Llama-2-70B is evaluated against GPT-3.5-turbo on serving cost and latency, with the comparison deliberately excluding GPT-4 and noting fine-tuning as a separate reason to choose Llama. The evaluation serves Llama on two 80-GB A100 GPUs in 16-bit precision, then uses transformer FLOP, memory-bandwidth, and KV-cache analysis to explain the difference between prompt processing and token generation. The measurements indicate that Llama-2 has worse completion-token pricing than GPT-3.5 on two A100s, while its prompt-token pricing is more than three times cheaper; eight A100s might improve pricing but introduce unacceptable latency. The resulting guidance favors GPT-3.5 for most workloads and open-source Llama models for prompt-heavy tasks such as classification or reranking, with batch processing and workloads lacking prompt tokens identified as additional cases where Llama may make sense.
Supabase Beta June 2023
Supabase’s June 2023 beta update announces product, tooling, authentication, billing, and community developments. Mozilla chose Supabase Vector and OpenAI for AI Help, an AI chatbot that supports search and discovery for MDN, while Supabase Auth added native Sign in with Apple and Google support for one-tap methods including web integrations and Chrome extensions. The Supabase CLI added Dashboard migration visibility with executed SQL, local serving for all Functions, and a usage flag for monitoring key Postgres statistics; Studio also gained usage, overage, plan-limit, subscription, billing, and compute-detail improvements. Other updates include Kakao login, a Realtime Inspector for testing endpoints with JWTs and Row Level Security policies, an object ID in Storage upload responses, OrioleDB parallel index builds, database partitioning guidance, community projects, and an upcoming Launch Week.
Go From 0 To 1 To Enterprise-Ready With MongoDB Atlas And LLMs
Generative AI applications require grounding in up-to-date operational data to avoid incorrect or uninformed answers, yet moving from prototype to enterprise-ready production is a significant challenge. MongoDB Atlas unifies operational, analytical, and vector search data services, enabling developers to provide long-term memory to large language models without costly fine-tuning. The post presents five resources for integrating MongoDB Atlas with LLMs, including tutorials for Atlas Vector Search with OpenAI, LlamaIndex, LangChain, MindsDB, and HuggingFace. These integrations support semantic search, chat applications with proprietary data, prediction generation, and automated reactions to data changes via serverless triggers. MongoDB Atlas offers a flexible document model, high performance, and scalability, validated by enterprises, to accelerate building production-ready generative AI applications.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.