---
title: "LLMs"
description: "673 posts about LLMs, summarised, each linking to the original."
---

# LLMs
> 673 posts about LLMs, summarised, each linking to the original.

## Articles

### [Shared Selective Persistent Memory for Agentic LLM Systems](https://yomu.fyi/post/shared-selective-persistent-memory-for-agentic-llm-systems.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Sep 16, 2026
- Type: Benchmark

Agentic LLM systems that generate code through multi-turn tool use lose configuration choices, domain constraints, schemas, and tool-use patterns when each session starts from zero. The proposed shared selective persistent memory retains task specifications, data schemas, tool configurations, and output constraints while discarding session-specific reasoning traces, and allows workspaces to be transferred across users with role-based access control. In a deployed collaborative workspace platform, agents create and maintain git-versioned dashboards, reports, and data-driven documents from CSV, SQL, REST API, and MCP sources; draft isolation supports risk-free changes and restoration without re-invoking the model. Results across three enterprise scenarios show 96% task completion, compared with 79% without memory and 71% with full history, while zero-token refresh cut recurring-update task time 14× and summary-driven generation reduced token cost 97× versus raw data injection.


### [Glyph: A Multi-Strategy Agentic System for Column Description and Sensitivity-Ontology Tagging of Enterprise Data Catalogs](https://yomu.fyi/post/glyph-a-multi-strategy-agentic-system-for-column-description-and-sensi.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Sep 16, 2026
- Type: Explainer

Glyph addresses enterprise data lakes where tables accumulate faster than human stewards can document or classify them, leaving column descriptions and governance labels incomplete. It models column description generation and sensitivity-ontology tagging as cooperating LLM agents orchestrated as stateful graphs. The Descriptor uses active Retrieval-Augmented Generation to retrieve pipeline source code from an enterprise GitHub, while the Tagger runs description, line-of-business regex, and metadata strategies in parallel before combining ranked outputs with Reciprocal Rank Fusion. A fine-tuned six-layer MiniLM metadata encoder raised same-tag retrieval on an in-distribution held-out split from NDCG@10 0.55 to 0.92 and MAP@100 0.19 to 0.90, relative to the stock base encoder. Glyph also evaluates end-to-end multilabel tagging with a recall-weighted F2 objective across three groups and an ablation, emphasizing value-free, code-grounded operation, per-tag provenance, and graceful degradation for auditable production cataloging.


### [Beyond the Model: Engineering AI Infra with Scientific Judgement](https://yomu.fyi/post/beyond-the-model-engineering-ai-infra-with-scientific-judgement.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: AirbnbEng
- Published: Sep 15, 2026
- Type: Explainer

Airbnb describes Insight Miner, an agent harness designed to make unstructured data investigations scalable, reproducible, and audit-ready. Its problem was a bespoke process spanning data discovery, expert review, representative datasets, and manually curated methods that could not keep pace with new languages, geographies, and LLM-based products. The shared package combines extraction, embedding, and clustering with prompt tuning, hard-example mining, contrastive labeling, and a path from unsupervised exploration to classification, while an agent handles research questions, execution, and methodological guidance. Airbnb says investigations that once took months can now take days, allowing mechanical work to scale while data scientists spend more time inspecting ambiguous evidence, testing hypotheses, and improving techniques. The tool also lets nontechnical experts run analyses across use cases including survey coding, model evaluation, and fraud patterns.


### [Your Agent Aced the Task. Will It Do It Again?](https://yomu.fyi/post/your-agent-aced-the-task-will-it-do-it-again.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Evelyn Duesterwald, Lilian Ngweta, Vatche Isahagian, Jayaram Radhakrishnan, Vinod Muthusamy, Gaodan Fang, Ashwath Vaithinathan Aravindan, Punleuk Oum, G Thomas, Merve Unuvar, Ayhan Sebin, Michał Ulewicz
- Published: Sep 15, 2026
- Type: Explainer

Agent reliability can be obscured by average benchmark scores: Mean@k reports average success, while Pass^k measures tasks succeeding on every repeated run. On AppWorld, a ReAct agent using GPT-4.1 achieved Mean@5 of 77.4% but Pass^5 of 53.0%, leaving a 24.4-point consistency gap, with temperature set to 0.0. The Consistency Analyzer diagnoses flip-prone decision points by resampling each step in one recorded trajectory with k=5 completions, requiring no ground truth, logits, or end-to-end replay. ALTK-Evolve turns flagged instability into reusable consistency guidelines that are retrieved at inference time. In the reported evaluation, these guidelines raised aggregate Pass^5 to 69.0% from 53.0% and Mean@5 to 81.0% from 77.4%, cutting the gap to 12.0 percentage points; similar-task Pass^5 also improved by 13.0pp.


### [Understanding W8A8 INT8 LLM quantization: Accuracy and performance results](https://yomu.fyi/post/understanding-w8a8-int8-llm-quantization-accuracy-and-performance-resu.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Sana Fayyaz
- Published: Sep 14, 2026
- Type: Benchmark

The post evaluates whether W8A8 INT8 quantization can reduce the size of a Llama 3.1 8B Instruct model without sacrificing accuracy or inference performance. Using SmoothQuant to address activation outliers and GPTQ with error compensation for BF16 weights, the model was compressed from 14.9 GB to 8.0 GB after calibration on WikiText-2. Identical zero-shot lm-eval-harness tests across MMLU, ARC Easy, HellaSwag, and IFEval found MMLU, ARC Easy, and HellaSwag deltas smaller than their standard errors, while the IFEval change was described as likely within a similar noise range; the post concludes no measurable degradation. vLLM and GuideLLM tests on one NVIDIA L40S reported 29% more concurrent requests, 24% lower synchronous TTFT, and 44% faster token generation at maximum load, while ITL degraded more quickly under high concurrency because of dynamic activation quantization overhead.


### [Product updates: Sandbox Sidecars, new models, a refreshed dashboard, and more](https://yomu.fyi/post/product-updates-sandbox-sidecars-new-models-a-refreshed-dashboard-and.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Sep 14, 2026
- Type: Announcement

Modal added day-zero support for Kimi K3, Qwen 3.8, GLM 5.3, and GLM 5.3 Flash on Auto Endpoints, with custom DFlash speculators boosting throughput to 460 tokens per second. New environment-level budgets give Team and Enterprise workspaces per-environment compute limits, while the pricing multiplier for broad region selection dropped from 1.5x to 1.15x. The dashboard got a refreshed navigation, color system, and beta light mode, and Private Environments now support a default No Access role. Sandbox Sidecars are in public alpha, running additional containers via low-latency internal networking, and VM Sandboxes are in public beta with a real Linux kernel for Docker and eBPF. Sandbox directory snapshots and a new filesystem API are now GA, and regional Proxies allow selecting a region for static outbound IPs, while SDK updates improved log APIs and autoscaler reporting.


### [AI Functions in ClickHouse: Upgrade your SQL to the AI age](https://yomu.fyi/post/ai-functions-in-clickhouse-upgrade-your-sql-to-the-ai-age.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Sep 11, 2026
- Type: Explainer

ClickHouse’s beta AI Functions let SQL queries call an LLM or embedding provider directly, keeping classification, extraction, generation, translation, redaction, filtering, embedding, and similarity work alongside stored data. The feature addresses workflows that export rows to separate LLM, orchestration, or vector-database systems and then reload results, with the stated goal of running the full RAG cycle in ClickHouse. Functions make remote HTTP calls per input, or batch embedding inputs, and return native ClickHouse values; in open-source deployments, named collections store provider credentials and endpoint settings. Examples use Hacker News data for topic classification and natural-language filtering, while the guidance emphasizes cheap predicates, quotas, LIMIT tests, prompt-injection defenses, non-determinism, cost, latency, and provider access to data in cleartext after TLS termination.


### [Together AI expands fine-tuning service with more models, live metrics, and finer controls](https://yomu.fyi/post/together-ai-expands-fine-tuning-service-with-more-models-live-metrics.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Sep 11, 2026
- Type: Announcement

Together AI is expanding its Fine-Tuning service to cover newer open-weight models and more of the workflow from dataset inspection through model selection. The update adds live training and evaluation metrics through the API, CLI, dashboard, and Python SDK, including curves that can be compared across in-flight jobs and underlying series that can be exported. Training controls now include Expert LoRA for mixture-of-experts layers, early stopping based on validation loss, and arbitrary effective batch sizes through gradient accumulation. The post reports that Expert LoRA recalled up to 89% of 200 invented facts versus 15% for attention-only adapters, while listed LoRA training prices fall by 30% to 70% for selected models, including the gpt-oss series.


### [DiscoSign: Discourse-Aware Text to Sign Language Gloss Translation](https://yomu.fyi/post/discosign-discourse-aware-text-to-sign-language-gloss-translation.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Sep 11, 2026
- Type: Problem & solution

DiscoSign addresses a limitation in sign language processing: systems have traditionally translated at the sentence level, overlooking discourse phenomena important to sign language comprehension. It presents a modular Large Language Model (LLM)-based framework for text-to-sign language gloss translation that handles spatial coreference resolution, Question-Answer Clauses (QACs), and concept-gloss consistency, preserving entity locations and stable English-to-ASL sign mappings across discourse. Because traditional translation metrics do not measure discourse-level quality, the authors introduce metrics for each coherence dimension targeted by the framework. Experiments on sentence-level and discourse-level datasets report improved spatial consistency and entity tracking over sentence-only translation, while maintaining competitive single-sentence gloss quality; the work describes this as the first systematic framework with a corresponding evaluation methodology.


### [Why don’t machine learning research agents overfit?](https://yomu.fyi/post/why-don-8217-t-machine-learning-research-agents-overfit.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Martin Bertran Lopez, Aaron Roth
- Published: Sep 10, 2026
- Type: Explainer

Repeatedly optimizing against long-lived machine-learning benchmarks appears to invite overfitting, yet the post notes that gains on heavily reused benchmarks have largely transferred to newly created test sets. It describes a paper that resets LLM-based research agents, gives an explorer repeated access to validation feedback, and passes the resulting strategy through a token bottleneck to a fresh reproducer. Across eight datasets, strategies from hundreds of experiments were reproducible from prompts as short as 16 to 32 tokens; at shorter budgets, performance diverged, showing that the compressed message carried data-dependent choices rather than only prior knowledge. Deliberate overfitting occurred in 38 of 102 runs, with validation accuracy more than 10% above held-out accuracy, but those gains disappeared after compression; possible pretraining side channels remain unresolved.


### [Prompting Claude's long context window](https://yomu.fyi/post/prompting-claude-s-long-context-window.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: Sep 9, 2026
- Type: Benchmark

An evaluation of Claude’s 100,000-token context window tests prompting techniques for recalling specific information from long documents. Researchers built randomized collages from sections of a July 13 government document, using Claude-generated multiple-choice questions and filtering out questions it could not answer from the source passage alone. They compared a baseline with unrelated examples, dynamically selected contextual examples, and two or five examples, each with or without a scratchpad that instructed Claude Instant 1.2 to extract relevant quotes across roughly 70K and 95K token contexts. Scratchpads and contextual examples generally improved performance, with more examples performing better, although recall could decline when the answer passage was at the end. For Claude 2, the best prompting strategy raised performance from 0.939 to 0.961, a reported 36% reduction in errors; reproducible code was released in the Anthropic Cookbook.


### [Claude 2](https://yomu.fyi/post/claude-2.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: Sep 9, 2026
- Type: Announcement

Claude 2 is introduced as a new model with improved performance, longer responses, expanded coding, math, and reasoning capabilities, and access through both an API and the claude.ai beta website. It supports prompts of up to 100K tokens and can produce documents of a few thousand tokens, enabling work across hundreds of pages of technical documentation or a book. Reported results include 76.5% on the Bar exam multiple-choice section versus 73.0% for Claude 1.3, 71.2% on Codex HumanEval versus 56.0%, and 88.0% on GSM8k versus 85.2%. Anthropic says internal red-team evaluation found Claude 2 2x better at giving harmless responses than Claude 1.3, while acknowledging that jailbreaks remain possible and inappropriate outputs can occur. The API is offered to businesses at the same price as Claude 1.3, and the chat experience is available in the US and UK, with broader availability planned.


### [Rust AI in Practice: Building LLM Applications With Rig](https://yomu.fyi/post/rust-ai-in-practice-building-llm-applications-with-rig.md)
- Company: [Jetbrains](https://yomu.fyi/company/jetbrains.md)
- Author: Irina Mihajlovic
- Published: Sep 9, 2026
- Type: Explainer

Rig is an open-source Rust library that provides a common interface for LLM providers including OpenAI, Anthropic, and Gemini, avoiding application rewrites when switching services. It structures applications around a provider client, completion model, agent, preamble, and registered tools, with async APIs handling provider network requests. In the Rat Code demo, built with Ratatui, separate tools read and write files and run shell commands, while Rig’s streaming API updates the terminal as responses arrive. The library also supports RAG through vector store abstractions, local inference via Ollama, llama.cpp, or Candle, and cassette-based integration tests that replay recorded HTTP interactions in CI; live model quality requires separate scheduled testing.


### [The Open Source AI Stack](https://yomu.fyi/post/the-open-source-ai-stack.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Sep 9, 2026
- Type: Explainer

The post presents an open model AI stack for developers moving from closed-source models, focusing on ownership, control, and economics. It organizes the stack into independent layers: model, inference, gateways and routers, harness, and tools such as Skills and MCP. Models generate responses from prompts; large models generally handle ambiguity, multi-step reasoning, and long context better, while small models can be faster and cheaper for tightly specified work. The separation lets developers choose providers and models independently and experiment with newly released models without rebuilding their workflow, while the post uses Kimi K3 and GLM 5.3 Flash to illustrate differences in active parameters, capability, speed, and cost.


### [Safety for Whom? Refusing the Right Subset of a Topic, Not the Whole Topic](https://yomu.fyi/post/safety-for-whom-refusing-the-right-subset-of-a-topic-not-the-whole-top.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Antonio Tiene, Alejo Lopez Avila, Iker García-Ferrero
- Published: Sep 8, 2026
- Type: Explainer

Topic-level safety systems can conflate a dangerous subject with dangerous intent, causing models to refuse legitimate requests alongside harmful ones. The paper studies deployment-specific boundaries using political persuasion as a testbed: refuse manipulation or targeted persuasion while answering factual political questions. Its training pipeline repairs self-generation coverage through escalating retries, adds verified surface-dangerous benign examples, and evaluates held-out harmful-benign prompt pairs rather than relying on aggregate refusal rates. On Qwen3-8B, escalated coverage raises in-distribution political refusal from 9.47% to 84.75%, while the strongest configuration reduces unsafe responses across three broader benchmarks from 26.26% to 0.14%; however, XSTest over-refusal rises to 74.00%. Adding benign boundary data reduces over-refusal on comply-worthy held-out prompts from 32.94% to 4.16%, with harmful-side refusal declining from 91.88% to 87.72%.


### [Choosing a low-latency infrastructure layer for conversational AI](https://yomu.fyi/post/choosing-a-low-latency-infrastructure-layer-for-conversational-ai.md)
- Company: [Twilio](https://yomu.fyi/company/twilio.md)
- Author: Luke Morgan
- Published: Sep 8, 2026
- Type: Problem & solution

Conversational AI call-center latency can feel like dead air, and the delay may come less from LLM inference than from the infrastructure moving audio among telephony, media servers, ASR, the agent, and TTS. The described approach uses a unified voice path: ConversationRelay provides a WebSocket API to stream live call audio, handle speech-to-text and text-to-speech in one connected pipeline, connect with a backend agent, and support barge-in. It is model-agnostic, allowing teams to retain their LLM, knowledge bases, AI tools, and third-party voice providers, while Conversation Orchestrator, Conversation Memory, and Agent Connect address context and integration. Twilio’s internal benchmarks report median latency below 0.5 second and below 0.725 second at the 95th percentile, though relevance and context also matter alongside speed.


### [How Agentforce Achieves 100% Deterministic Rendering for AI Agent UX](https://yomu.fyi/post/how-agentforce-achieves-100-deterministic-rendering-for-ai-agent-ux.md)
- Company: [Salesforce](https://yomu.fyi/company/salesforce.md)
- Author: Scott Nyberg
- Published: Sep 3, 2026
- Type: Explainer

Agentforce faced a production problem: LLM-driven rendering could recognize a required form or regulated disclosure yet choose a different output because its reasoning was stochastic. The Agentforce Connections layer addresses this by separating probabilistic reasoning and orchestration from deterministic presentation: builders associate an action with a connection-specific response format through the render: directive, and the platform applies that format instead of asking the LLM to select a component. Multi-action, long-running, and supervised flows required the system to identify the configured action and render only after approval and execution. The same abstraction supports Lightning Web Components, third-party interfaces, and headless structured JSON outputs. Per-action, per-session event logs record the action, format, connection, and conversation point, providing auditable evidence that required rendering occurred.


### [Speeding up LLM inference with P-EAGLE in vLLM Speculators](https://yomu.fyi/post/speeding-up-llm-inference-with-p-eagle-in-vllm-speculators.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Helen Zhao, Megan Flynn, Dipika Sikka
- Published: Sep 3, 2026
- Type: Explainer

P-EAGLE (Parallel EAGLE), implemented in Speculators v0.6.0, extends EAGLE-3 with parallel drafting so a draft model predicts multiple future tokens in one forward pass rather than sequentially. Speculative decoding pairs this small draft model with a verifier model, which checks the proposed sequence in parallel and accepts only its correct prefix while preserving the verifier’s output distribution. The post adds Conditional Drop-token (COD) sampling, learnable mask embeddings and hidden states, and a custom flex-attention mask to support multi-depth prediction within the EAGLE-3 framework. The post presents an end-to-end workflow covering data preparation, vLLM hidden-state extraction, training and deployment, including a Qwen3-8B example that takes about 50 minutes on four H100s. A published Qwen3-a8B draft model reports an average length of 3.044 across the listed datasets.


### [Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps](https://yomu.fyi/post/fine-tuning-a-350m-model-for-better-structured-outputs-in-100-grpo-ste.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Leonie Monigatti, ben burtenshaw, Sergio Paniego
- Published: Sep 3, 2026
- Type: Tutorial

This guide presents a low-cost recipe for improving structured-output compliance in Liquid AI’s LFM2.5-350M, a task where parseable, schema-adherent responses determine whether an LLM can feed downstream systems. It fine-tunes the model with Group Relative Policy Optimization (GRPO) through TRL, using about 500 samples, 100 steps, three weighted rewards for format, field count, and JSON Schema validation, and a LoRA adapter training roughly 6M parameters. Prompt augmentation covers fenced-code-block instructions and top-level-array tasks, while evaluation uses IFStruct with llama.cpp on an OpenAI-compatible local server. On 2,000 IFStruct samples, the BF16 base model passes 22.6%, rising to 29.7% after tuning. JSON improves from 18.0% to 31.9%, bare-list compliance from 16.6% to 29.7%, while YAML changes little.


### [How we make AI coding more cost efficient without sacrificing task quality](https://yomu.fyi/post/how-we-make-ai-coding-more-cost-efficient-without-sacrificing-task-qua.md)
- Company: [Github](https://yomu.fyi/company/github.md)
- Author: Erik Kristensen
- Published: Sep 2, 2026
- Type: Explainer

GitHub Copilot’s AI coding efficiency work focuses on completed task cost rather than tokens in individual tool calls, because shorter responses can trigger recovery work. Using agentic coding benchmarks and controlled online experiments, the team evaluated selective output compression, removal of unused line-number prefixes, prompt shortening, and direct delivery of completed background results. The shipped compressor preserves source-like and arbitrary output, reorganizes search results without dropping content, and compresses repetitive install, build, test, and progress output while retaining a recovery path to originals. Removing line-number prefixes cut offline model-inference cost roughly 5% and online daily cost about 3%; direct background-result delivery reduced AI Credit usage about 2.3%, while prompt changes required workload-specific validation.


[Newer posts](https://yomu.fyi/topic/llm/page/4.md) · [Older posts](https://yomu.fyi/topic/llm/page/6.md)