Activation Steering: One Dial, Two Costs
Activation steering edits a model's internal activations to add or remove a concept at inference time. This explainer maps its one control knob, its two costs, and when it beats prompting.
Loading…
Topic
LLMs
656 posts about LLMs. Every summary links to the original.
- 656 posts
- 49 companies
- 5 explainers
See the idea
LLMs explainers
Antibodies as a Prediction Pipeline: Where AI Models Break Down
Antibody discovery is a three-stage prediction pipeline, and AI models that excel on familiar targets degrade sharply on unfamiliar ones. This explainer maps where that gap appears and what it costs.
Agent Memory: What Outlives the Run
Agent memory is the knowledge that survives a run, not the working state inside one. This explainer separates the two, then maps the write path, the read path, and the database limits that cap recall.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How Yoko Li makes towns, tamagoes, and tools for local AI
Yoko Li’s local AI work spans playful agent experiments and practical tools for running language-model workflows on personal hardware. AI Town uses conversations among AI agents to produce changing relationships and emergent behavior, while AI Tamago feeds game state and user inputs to an unmodified LLaMA 2 7B model and uses Ollama’s JSON mode to generate pet actions. The Local AI Starter Kit packages document ingestion, vector storage, and language-model responses into a private, local “chat with a knowledge base” pipeline, avoiding setup across at least four API providers. The post also describes unreliable JSON generation, hallucinations, and grammar-guided workarounds, concluding that randomness is a liability for factual outputs but useful for game behavior and creative experimentation.
Scaling Large Language Models to zero with Ollama
Running GPU nodes for language-model inference can be costly because they often sit idle between requests, so this guide configures Ollama on a Fly.io Machine that scales to zero when idle. Ollama is described as a wrapper around llama.cpp, enabling self-hosted models with GPU acceleration, while about 70 lines of Go add authentication to protect the endpoint from unauthorized use. Deployment uses an A100-40GB GPU, a private Flycast address, automatic start and stop settings, and a 100GB persistent volume mounted at /root/.ollama so downloaded models survive stopped Machines. A stopped Machine no longer incurs GPU charges, but scaling to zero suspends it on the same host and may fail to wake during capacity issues. The setup supports CLI and API requests, WireGuard access, custom Modelfiles, and JSON-mode function-calling workflows.
Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling
Nextdoor implemented a generative AI framework to improve email notification subject lines for its New and Trending emails. Using off-the-shelf ChatGPT API outputs directly resulted in marketing-like phrases, hallucinations, and lower click-through rates compared to original user-generated post snippets. To resolve this, the team prompted the language model strictly to extract authentic post phrases without rewriting, eliminating hallucinations while preserving the original tone. Nextdoor then introduced a rejection sampling mechanism featuring a dedicated reward model that predicts user engagement and accepts generated subjects only when their score exceeds the user-written baseline. This combined system lifted sessions by 1%, boosted Weekly Active Users by 0.4%, and increased advertising revenue by 1%.
Sidekick’s Improved Streaming Experience
Sidekick addresses two streaming problems in LLM chatbots: Markdown fragments can render awkwardly before syntax is complete, and tool-driven responses can delay visible output across multiple LLM roundtrips. Its solution combines a buffering Markdown parser with an event emitter that multiplexes the main response and asynchronously resolved content into one stream. A Node.js Transform stream processes Unicode characters through a finite-state machine, buffering ambiguous sequences such as emphasis, list markers, and links until they resolve or are flushed as raw text. The parser also recognizes card links, triggering asynchronous card-resolution tasks while Server-Sent Events carry named streams to the UI, which demultiplexes them into the main response and placeholders. The design uses Markdown as a human-readable transport between the LLM, backend, and frontend, allowing content to appear as it is produced.
Supabase Launch Week 8 Hackathon
Supabase Launch Week 8 will run an asynchronous hackathon for new Open Source projects that use Supabase in some capacity. The event is listed as starting at 09:00 am PT on Friday 4 August 2023, with the main schedule ending on Sunday 13 August at 11:59 pm PT; the submission instructions separately give a deadline of 11:59 pm PT on Sunday 14 August 2023. Individuals or teams of up to five can build an app, tool, library, or other project in any language or framework, with all design and code created during the event. Judges will award winners and runners-up across five categories, including overall quality, AI use, technical impressiveness, visual appeal, and fun, while submissions must include source code and may earn extra points for a demo video.
Inference characteristics of Llama
Llama-2-70B is evaluated against GPT-3.5-turbo on serving cost and latency, with the comparison deliberately excluding GPT-4 and noting fine-tuning as a separate reason to choose Llama. The evaluation serves Llama on two 80-GB A100 GPUs in 16-bit precision, then uses transformer FLOP, memory-bandwidth, and KV-cache analysis to explain the difference between prompt processing and token generation. The measurements indicate that Llama-2 has worse completion-token pricing than GPT-3.5 on two A100s, while its prompt-token pricing is more than three times cheaper; eight A100s might improve pricing but introduce unacceptable latency. The resulting guidance favors GPT-3.5 for most workloads and open-source Llama models for prompt-heavy tasks such as classification or reranking, with batch processing and workloads lacking prompt tokens identified as additional cases where Llama may make sense.
Supabase Beta June 2023
Supabase’s June 2023 beta update announces product, tooling, authentication, billing, and community developments. Mozilla chose Supabase Vector and OpenAI for AI Help, an AI chatbot that supports search and discovery for MDN, while Supabase Auth added native Sign in with Apple and Google support for one-tap methods including web integrations and Chrome extensions. The Supabase CLI added Dashboard migration visibility with executed SQL, local serving for all Functions, and a usage flag for monitoring key Postgres statistics; Studio also gained usage, overage, plan-limit, subscription, billing, and compute-detail improvements. Other updates include Kakao login, a Realtime Inspector for testing endpoints with JWTs and Row Level Security policies, an object ID in Storage upload responses, OrioleDB parallel index builds, database partitioning guidance, community projects, and an upcoming Launch Week.
Go From 0 To 1 To Enterprise-Ready With MongoDB Atlas And LLMs
Generative AI applications require grounding in up-to-date operational data to avoid incorrect or uninformed answers, yet moving from prototype to enterprise-ready production is a significant challenge. MongoDB Atlas unifies operational, analytical, and vector search data services, enabling developers to provide long-term memory to large language models without costly fine-tuning. The post presents five resources for integrating MongoDB Atlas with LLMs, including tutorials for Atlas Vector Search with OpenAI, LlamaIndex, LangChain, MindsDB, and HuggingFace. These integrations support semantic search, chat applications with proprietary data, prediction generation, and automated reactions to data changes via serverless triggers. MongoDB Atlas offers a flexible document model, high performance, and scalability, validated by enterprises, to accelerate building production-ready generative AI applications.
Prompt design
The post argues that “prompt design” better describes prompt construction than “prompt engineering,” comparing it with web design. It frames prompts as clear communication with dynamic inputs that must adapt to changing content and context-window limits, then identifies shared practices such as inspecting rendered output, using composable declarative components, and pursuing precise formatting. For LLM agents, the analogy extends to interactive websites whose prompts re-render after function calls, although text-only output and caching constraints differ. This perspective motivates Priompt, a React-like, JSX-based library used internally at Cursor, with a preview site that renders prompts using real request data and serialized props; the author considers JSX more ergonomic than string templates while noting that model capabilities and platform abstractions may weaken the analogy over time.
Supabase Beta May 2023
Supabase’s May 2023 update announces two major product areas: Supabase Vector, an open-source toolkit for storing, indexing, and querying Postgres vector embeddings at scale, and Supabase Vault, a Postgres extension and UI for encrypted secrets and data. Auth Helpers now support client and server use of the Next.js App Router and server-side authentication by default with PKCE. The release also adds Turnstile CAPTCHA support, password reauthentication nonces, Wrappers integrations for Firebase, Stripe, S3, and clickhouse, CLI deployment for all Edge Functions, custom and vanity domains, and generally available Image Transformation. Supabase says it reached 50,000 GitHub stars, while the team solicits Dashboard SQL Editor feedback and reports community projects, AI resources, and a first-ever Flutter Hackathon.
ChatGPT plugins now support Postgres & Supabase
Retrieval plugins let ChatGPT access relevant content from private datasets by chunking documents, generating embeddings with OpenAI’s text-embedding-ada-002, and querying a vector database. The article describes Postgres and Supabase datastore implementations using pgvector, with a shared documents schema and a match_page_sections function that filters by source, author, document, and date before ranking vectors with inner product distance. It then gives a local example: downloading the Postgres 15 documentation as a PDF, uploading it through the plugin’s /upsert-file endpoint, and configuring environment variables for Supabase or Postgres. The example produced 7,904 document records, and an optional IVFFlat index can trade exact nearest-neighbor search for approximate search; after registering the local plugin in ChatGPT, users can ask questions answered from the documentation.
Building ChatGPT Plugins with Supabase Edge Runtime
A TypeScript template for ChatGPT Plugins runs on Supabase Edge Runtime and provides a path for exposing a TODO API to ChatGPT. The implementation serves the required ai-plugin.json manifest from an Edge Function because static file hosting is not currently supported, and generates an OpenAPI JSON document from endpoint JSDoc annotations with swagger-jsdoc. An Oak router defines TODO routes, while shared CORS headers allow requests from https://chat.openai.com and credentials. Docker configuration runs Edge Runtime locally with a bind-mounted functions directory, enabling changes without rebuilding and exposing requests through localhost:8000. The guide also covers local installation and testing through the ChatGPT UI, then deployment to Fly.io; it notes authentication as a possible next step.
Launch Week 7 Hackathon Winners
Supabase Launch Week 7 announces the winners of its hackathon, which introduced an AI theme and recognized projects across five categories. Page Assist won Best Overall for an open-source Chrome extension that lets users chat with webpages through the ChatGPT API, while other winners included Rect, Supathreads, Replay.ai, Supa0SQL, and Generation: Hotdog. The projects apply AI to educational content, blog-to-thread conversion, browsing-history search with cosine similarity, natural-language chart creation, and hotdog-image generation using OpenAI and Stable Diffusion APIs; runner-ups included tools for video chat, drum patterns, tweet scheduling, and music guessing. Members of the Best Overall winning team receive a limited-edition Launch Week 7 Keyboard, while winners and runner-ups in other categories receive Launch Week 7 swag kits, alongside guides for getting started with Supabase resources.
Supabase Studio 2.0: help when you need it most
Supabase Studio 2.0 adds AI-assisted database development alongside a set of Postgres, GraphQL, and Table Editor improvements. The ⌘K menu can use AI to create tables, views, indexes, functions, and triggers, while also providing Studio navigation and Docs search; a future opt-in will send table schemas to OpenAI to help fine-tune queries. The release adds integrated GraphiQL, cascade-delete configuration, a Query Performance tool for investigating slow queries, foreign-key row selection, revised Postgres role management, editable database webhooks that can call Edge Functions, SQL definitions, API autodocs, and downloadable TypeScript types. It also improves responsiveness with many tables, expands JSON editing, and permits text and boolean cells to be null or empty, while the announcement says future work will focus on RLS policy assistance.
Supabase Beta March 2023
Supabase’s March 2023 update previews Launch Week 7, an accompanying Web3 AI hackathon, and a collection of community projects and resources. It announces a Next.js OpenAI documentation-search template hosted on Vercel, combining GPT-3 text completion, a Postgres database with pgvector, and Deno Edge Functions. The update also documents the Launch Week ticket system, including an AI-fueled design and an infinite ticket scroll built with Next.js, Framer Motion, and Supabase, with separate posts covering implementation and artwork generation. It additionally promotes the SupaClub April 1 announcement, a Launch Week kickoff Twitter Space, and projects such as Markprompt, which creates embeddings from Markdown, Markdoc, and MDX files in GitHub repositories using Next.js, Vercel, OpenAI, and Supabase.
The Supabase AI Hackathon
Supabase announced an async, AI-themed hackathon running from 6:00 am PT on 7 April 2023 until 11:59 pm PT on 16 April 2023. Participants may enter individually or in teams of up to five and must create a new Open Source project using Supabase in some capacity; projects can be apps, tools, libraries, or other forms, in any language or framework. Entries compete for overall, AI, fun or Easter egg, technical, and visual categories, with a winner and runner-up in each category receiving Supabase Launch Week swag, while the overall winner also receives mechanical keyboards. The Supabase team will judge creativity, functionality, visual quality, technical achievement, Supabase usage, and fun, and submissions require source-code links, with an optional demo video earning extra points.
Supabase Clippy: ChatGPT for Supabase Docs
Supabase introduced “Supabase Clippy,” a ChatGPT-like interface trained on its documentation, as a hidden MVP in the Supabase docs. It responds to questions by retrieving relevant documentation and streaming GPT-3's answer in realtime. The implementation parses docs into sections, creates OpenAI embeddings, stores them in Postgres with pgvector, and injects matching documents as context for generation. The project was motivated by the growth of Supabase's product suite and documentation, plus the need to support developers unfamiliar with Postgres; the current version uses only Guides and is presented as a one-week MVP intended to gather feedback. Planned directions include adding reference docs and community discussions, caching repeated queries, thumbs-up relevance feedback, and interactive guidance.
Accelerate Large Model Training using DeepSpeed
Training large models on hardware with limited GPU memory frequently causes out-of-memory errors when using standard Distributed Data Parallel. To solve this bottleneck, Hugging Face Accelerate integrates DeepSpeed ZeRO data parallelism to shard optimizer states, gradients, and model parameters across workers. For a 900-million-parameter DeBERTa model, ZeRO Stage 2 increased the maximum per-device batch size from eight to forty while achieving a 3.5-fold training speedup over DDP without degrading accuracy or F1 score. Advanced configurations allow sequence-to-sequence chatbot finetuning and full ZeRO Stage 3 CPU offloading for models like the 1.5-billion-parameter GPT-XL, enabling batch size 16 training where DDP fails. Accelerate enables these memory optimizations through simple configuration files and the accelerate launch command with minimal or zero code modifications.
Efficient Table Pre-training without Real Data: An Introduction to TAPEX
Conventional language model pre-training relies on massive natural text corpora and general-purpose objectives like masked language modeling, leaving a substantial gap when applied to structured table tasks. To bridge this gap efficiently, TAPEX (Table Pre-training via Execution) pre-trains a language model, such as BART, by learning a neural SQL executor over a synthetic corpus. The system pairs web tables with systematically sampled executable SQL queries and gathers target execution outputs via an off-the-shelf SQL engine. During downstream fine-tuning, the model consumes natural language questions and flattened tables to predict answers. TAPEX achieved new state-of-the-art results across four benchmark datasets, including WikiTableQuestions and TabFact, while achieving a pre-training speedup of nearly 50 times over TaBERT.
Machine Learning Experts - Sasha Luccioni
Research scientist Sasha Luccioni discusses her work on evaluating the environmental, societal, and ethical impacts of machine learning models. Her research includes measuring and reducing artificial intelligence emissions through initiatives such as CodeCarbon, regional grid selection, and the BigScience carbon footprint working group. BigScience aims to make large language models transparent by curating multilingual datasets, assessing manufacturing and compute costs, and analyzing participant workloads. Luccioni also emphasizes expanding model evaluation criteria beyond standard accuracy metrics to encompass efficiency, emissions, and subset-level fairness across demographics and data features. Overall, these efforts aim to democratize artificial intelligence and encourage deliberate trade-offs between model performance and computational resource demands.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.