---
title: "LLMs"
description: "673 posts about LLMs, summarised, each linking to the original."
---

# LLMs
> 673 posts about LLMs, summarised, each linking to the original.

## Articles

### [How Visa went from multi-day reporting to conversational analytics agents with ClickHouse Cloud and LibreChat](https://yomu.fyi/post/how-visa-went-from-multi-day-reporting-to-conversational-analytics-age.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 24, 2026
- Type: Problem & solution

Visa used ClickHouse Cloud to modernize reporting on Authorize.net payments data, replacing multi-day analyst-led turnaround with natural-language queries for internal teams. The solution pairs LibreChat, an open-source conversational interface, with the ClickHouse MCP server and runs on ClickHouse Cloud on AWS, allowing agents to explore and query a roughly 40 PB data footprint containing millions of rows. ClickHouse materialized views pre-compute and incrementally update results, reducing scanned data and bringing example analyses down to 100–200 milliseconds, with reported speedups of roughly 30–70 times. The platform also uses encrypted transfers, Amazon S3, VPC PrivateLink, change data capture, and federated access controls, while Visa reports identifying millions of dollars in revenue risk and reclaiming 8–10 hours per user each week.


### [Achieve state-of-the-art inference latencies with speculative decoding](https://yomu.fyi/post/achieve-state-of-the-art-inference-latencies-with-speculative-decoding.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 24, 2026
- Type: Problem & solution

Modal launched Auto Endpoints, combining speculative decoding with Blackwell GPUs, SGLang engine, and Modal Servers to achieve low inference latency for production deployments. The decode phase, where each token requires a full forward pass loading model weights, is the primary latency source, making speculative decoding the most impactful optimization. Speculative decoding runs a separate DFlash speculator model in parallel with the target model using KV projections and a custom Triton kernel, achieving near-linear speedup proportional to acceptance length. In the Decagon voice case study, Modal cut 100ms from a 290ms baseline by mid-training a custom speculator on synthetic data, beating proprietary providers by 60ms. Modal plans to automate speculator training with continuous improvement on production data, moving toward autospec for Auto Endpoints.


### [Announcing the Managed ClickStack MCP Server](https://yomu.fyi/post/announcing-the-managed-clickstack-mcp-server.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 24, 2026
- Type: Announcement

Managed ClickStack in ClickHouse Cloud now includes the ClickStack MCP server, extending its observability investigation capabilities to external agents and AI workflows. Unlike the generic ClickHouse MCP server’s SQL-oriented interface, it offers semantic tools for log patterns, outlier correlation, slow traces, and repeatable investigations across logs, metrics, and traces. For event patterns, the server samples matching events, processes them with Drain3 in the MCP layer, and extrapolates and ranks counts, reducing output cardinality and making query overhead fixed rather than linear in the full dataset. Internal benchmarks report 25% fewer tool calls, 2.5x greater consistency, and an almost 20% increase in evaluation scores versus the standard ClickHouse MCP. Managed deployments also support full-fidelity telemetry, dedicated compute for agent workloads, and orchestration such as persisted dashboards, while native SQL remains available for unusual or deeper questions.


### [Maximize AI Factory Energy Efficiency Through Full-Stack Inference and Training Optimizations](https://yomu.fyi/post/maximize-ai-factory-energy-efficiency-through-full-stack-inference-and.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Sachin Idgunji
- Published: Jun 23, 2026
- Type: Explainer

Power costs comprise up to 40% of AI factory operating expenses, making performance per watt critical within fixed site power limits. Operators maximize throughput by combining architectural enhancements, narrow-precision formats like NVFP4, and mixture-of-experts model designs that activate fewer parameters per token. During distributed LLM training in Megatron-LM, tuning individual GPU speeds along critical execution paths eliminates idle power waste without extending total training time. Additionally, the NVIDIA DSX platform orchestrates dynamic power allocation, 45°C direct-to-chip liquid cooling, and grid integration across racks and facilities. Together, these full-stack optimizations enable performance-optimized AI factories to deliver up to 2.6 times more tokens per second per megawatt.


### [Build Trust in Agentic AI: From POC to Production](https://yomu.fyi/post/build-trust-in-agentic-ai-from-poc-to-production.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Ken Wiebke
- Published: Jun 23, 2026
- Type: Explainer

Transitioning autonomous agentic AI systems from sandbox environments to enterprise production exposes a trust gap driven by hallucination, reasoning errors, and financial liability. To mitigate these risks, organizations can implement a proactive four-step framework covering foundations, verification, governance, and business outcomes. The architecture grounds agents using retrieval-augmented generation, short-term memory, long-term procedural memory, and observability traces consolidated on a unified data layer. Active verification computes an Agent Confidence Score alongside a deterministic Business Risk Score to produce an Agent Decision Score governing autonomy levels. Aggregated execution telemetry feeds business observability dashboards that monitor unit economics and enable analytical agents to recommend operational interventions.


### [Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding](https://yomu.fyi/post/boost-inference-performance-up-to-15x-on-nvidia-blackwell-using-dflash.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Amr Elmeleegy
- Published: Jun 23, 2026
- Type: Benchmark

Sequential token generation in autoregressive LLMs limits GPU utilization and serving throughput, while conventional speculative decoding remains constrained by autoregressive draft models. DFlash addresses this bottleneck by employing a lightweight block-diffusion drafter that predicts an entire block of masked candidate tokens in a single forward pass. The architecture incorporates target hidden-state conditioning and key-value injection into draft model projections while relying on the target model for parallel verification. Evaluated on NVIDIA Blackwell GPUs across TensorRT-LLM, vLLM, and SGLang, DFlash increases gpt-oss-120b throughput by more than 15x at high interactivity targets and delivers up to 5.8x speedups on Gemma 4 31B compared to autoregressive decoding.


### [Your AI shipped a backend that boots. That is the whole problem.](https://yomu.fyi/post/your-ai-shipped-a-backend-that-boots-that-is-the-whole-problem.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Devlin Duldulao
- Published: Jun 23, 2026
- Type: Problem & solution

Backend security vulnerabilities often emerge from permissive default settings in web frameworks, a risk amplified when AI agents generate code focused solely on passing tests. Common failure points include unmetered request body parsing, unvalidated schemas, prototype pollution, and wildcard cross-origin resource sharing policies. To eliminate these foot-guns, the TypeScript framework DaloyJS enforces secure-by-default behavior across its entire routing and middleware model. It provides built-in request timeouts, streamed body size caps, prototype-stripping JSON revivers, and unified Zod schemas that simultaneously validate payloads and generate OpenAPI contracts. While secure defaults lower the risk of common infrastructure exploits, developers still remain responsible for application-level authorization and business logic correctness.


### [The 2026 Developer Survey is now open (for human developers only)!](https://yomu.fyi/post/the-2026-developer-survey-is-now-open-for-human-developers-only.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Ryan Donovan
- Published: Jun 23, 2026
- Type: Announcement

The 2026 Developer Survey is open to gather data on software builders' tools, working lives, and emerging practices. Marking fifteen years since its initial 2011 release, the annual survey focuses heavily on the role of artificial intelligence and autonomous agents across the software development lifecycle. Past survey results indicated that while developer adoption of AI and agents doubled, user trust declined alongside escalating concerns regarding generated code quality. In addition to modern AI trends and financial return considerations, the questionnaire maintains traditional inquiries regarding day-to-day technologies and developer workflows. The survey is hosted on the Qualtrics platform, which requires users to pause third-party ad blockers and adheres to regional access restrictions across multiple countries.


### [Build an AI Scientist for Life Science Discovery with NVIDIA BioNeMo Agent Toolkit](https://yomu.fyi/post/build-an-ai-scientist-for-life-science-discovery-with-nvidia-bionemo-a.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Kyle Tretina
- Published: Jun 23, 2026
- Type: Tutorial

General-purpose AI agents often struggle with biomolecular discovery because scientific workflows require precise model selection, strict input formatting, and artifact interpretation. The NVIDIA BioNeMo Agent Toolkit bridges this gap by providing BioNeMo Skills and Model Context Protocol wrappers on top of accelerated NVIDIA NIM microservices. These skills instruct agents on tool capabilities, required inputs, expected file formats like CIF and SMILES, and specific failure modes. Developers can route requests through hosted endpoints for testing or deploy local NIM microservices on GPU nodes to lower warm per-call latency during iterative candidate generation loops. Internal benchmarks using Codex CLI demonstrate that equipping agents with BioNeMo Skills improves task completion rates from 57.1% to 100% while doubling token efficiency across passing assertions.


### [End-to-End RAG Workflow: How Retrieval Augmented Generation Works](https://yomu.fyi/post/end-to-end-rag-workflow-how-retrieval-augmented-generation-works.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 23, 2026
- Type: Tutorial

Retrieval Augmented Generation (RAG) connects a large language model to external knowledge at inference time, addressing outdated training data and access to proprietary or real-time information. The workflow ingests and normalizes sources, splits documents into chunks, embeds them in a vector store, retrieves context, assembles a prompt, and generates an answer. Semantic search can be combined with BM25 keyword search through reciprocal rank fusion, while reranking can improve precision; the same embedding model must be used during ingestion and querying. The guide presents evaluation and deployment considerations, including separate measurement of retrieval precision and generation faithfulness, versioning, monitoring, and containerized components. It identifies poor retrieval as the most common failure mode and explains that RAG reduces, but does not eliminate, hallucinations.


### [How Telcos Build Autonomous Networks with Agentic AI](https://yomu.fyi/post/how-telcos-build-autonomous-networks-with-agentic-ai.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Amogh Dendukuri
- Published: Jun 23, 2026
- Type: Explainer

Telecom operators currently operate mostly within Levels 2 and 3 of the TM Forum autonomous networks taxonomy, relying on predefined scripts. Progressing to Levels 4 and 5 demands autonomous systems capable of intent comprehension, real-time telemetry sensing, and cross-domain action coordination. To address this, an autonomy platform architecture combines domain-specific reasoning models, agent harnesses, and isolated sandboxed runtimes. Workflows separate responsibilities among on-demand agents, long-running execution loops, and deep-research multi-agent systems that evaluate ranked remediation paths or design wireless algorithms. Initial experiments in simulated SR-MPLS routing and wireless physical or MAC layer design show autonomous discovery can deliver measurable gains over standard operational and engineering baselines.


### [Introducing Modal Auto Endpoints: Optimized inference you actually own](https://yomu.fyi/post/introducing-modal-auto-endpoints-optimized-inference-you-actually-own.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 23, 2026
- Type: Announcement

Modal Auto Endpoints is a self-serve service for production-grade LLM inference that gives teams ownership of the entire serving stack, not just an API. Unlike proprietary providers, it exposes the inference code, engine flags, and GPU selection, and provides detailed metrics like speculative decoding acceptance length and per-replica token latency quantiles in a dashboard. Built on Modal’s infrastructure, it uses autoscaling to handle traffic spikes by spinning up additional replicas automatically, global GPU capacity without capacity management, and Modal Servers for ultra-low-latency routing with only 5ms overhead. For each supported model, it offers a pre-configured deployment using speculative decoding with the DFlash block-diffusion drafter from Z Lab, and teams can benchmark latency and throughput tradeoffs with a click after deployment. The platform is designed with a declarative interface based on workloads and SLOs, with future automation levels including autoinference, autospec, autodistill, and autoresearch for continuous improvement of inference performance.


### [Shipping huggingface\_hub every week with AI, open tools, and a human in the loop](https://yomu.fyi/post/shipping-huggingface-hub-every-week-with-ai-open-tools-and-a-human-in.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Lucain Pouget, Célina Hanouti
- Published: Jun 23, 2026
- Type: Problem & solution

huggingface\_hub, the Python client underlying libraries such as transformers, datasets, diffusers, and sentence-transformers, moved from releases every four to six weeks to a weekly cadence. A single manually triggered GitHub Actions workflow now computes versions, manages branches, tags and publishes both huggingface\_hub and the hf CLI, creates downstream RC test branches, and handles post-release tasks. An open-weights model drafts structured release notes and Slack announcements using PR metadata and documentation diffs, while OpenCode, HF Inference Providers, PyPI Trusted Publishing, and pinned, checksum-verified tooling provide the stack. Deterministic manifest validation checks missing or extra PR references, then a human reviews and publishes the drafts. The reported result is weekly shipping at about $0.25 per full release, with more consistent notes, earlier downstream breakage detection, and shorter contributor feedback loops.


### [What's new in ClickStack - May 2026](https://yomu.fyi/post/what-s-new-in-clickstack-may-2026.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 23, 2026
- Type: Announcement

ClickStack’s May and early-June release focused on making observability dashboards more useful during investigations and expanding access to related workflows. Dashboard table actions can open searches against selected sources with templated WHERE clauses or link to other dashboards, while scoped filters prevent source-specific conditions from affecting unrelated tiles. Number visualizations gained fixed or threshold-based colors and automatic text scaling, and the service map added latency percentiles, throughput metrics, server-side filtering, and focus mode. The release also includes a Browser RUM dashboard template, experimental PromQL support through the ClickHouse TimeSeries Engine, expanded MCP tools, AI Notebooks beta for Managed ClickStack, and ClickStack Cloud in private preview.


### [Measuring What Matters with Jules](https://yomu.fyi/post/measuring-what-matters-with-jules.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Nghi Bui, Georgios Evangelopoulos, Zack Elliott
- Published: Jun 22, 2026
- Type: Benchmark

Proactive coding agents need evaluation beyond task completion because existing public benchmarks such as SWE-Bench do not measure goal-oriented insight policies: deciding what matters, supporting evidence, and whether to interrupt or stay silent. To create preliminary ground truth, researchers clustered 705 bugs and 1,178 CLs from internal Google codebases using temporal proximity and semantic similarity, reverted each codebase to its pre-fix state, and gave the agent up to three exploration rounds. An LLM scored predicted insights from 1 to 5 against clustered bug targets, while success used average top score and Hit@K. With one exploration round, the agent averaged 4.5 out of 5; increasing the budget from two to three rounds raised Hit@5 from 33% to 57%, although the authors describe the findings as preliminary and are expanding the evaluation to public GitHub data and richer context streams.


### [We got local models to triage the OpenClaw repo for FREE!\*](https://yomu.fyi/post/we-got-local-models-to-triage-the-openclaw-repo-for-free.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Onur Solmaz, ben burtenshaw, shaun smith
- Published: Jun 22, 2026
- Type: Problem & solution

OpenClaw receives hundreds of issues and pull requests daily, creating a triage problem for maintainers who need near-real-time filtering without consuming a paid closed-model quota. The system uses Gemma and Qwen local models inside a Pi-based agent harness: each item is normalized, enriched with GitHub context, classified into finite topic labels, and required to return structured output. For pull requests, the agent can inspect the repository through reposhell, a read-only command-restricted shell, while localpager, SQLite, a worker queue, and deterministic notification policy handle orchestration and Discord delivery. Tests described for gemma-4-26b-a4b and qwen3.6-35b-a3b report hundreds of tokens per second and good zero-shot classification accuracy without fine-tuning, while an LLM audit loop monitors errors; the authors present the pattern as agentic classification for high-throughput triage.


### [Dispatches from O'Reilly: From capabilities to responsibilities](https://yomu.fyi/post/dispatches-from-o-reilly-from-capabilities-to-responsibilities.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Artur Huk
- Published: Jun 19, 2026
- Type: Problem & solution

High-stakes AI agents capable of mutating external state often face governance failures when relying on system prompts or manual Human-in-the-Loop approval queues that quickly degrade into alert fatigue. The Responsibility-Oriented Agent architecture addresses this operational bottleneck by shifting system design from open-ended capability framing to deterministic, contract-enforced responsibilities. Under this model, underlying orchestration frameworks like LangChain operate in User Space with their side-effecting tools removed, isolating the agent to epistemic reasoning. The agent expresses its intended action exclusively by emitting a structured policy proposal to a privileged Kernel Space runtime. The runtime deterministically evaluates the proposal against versioned YAML contracts registered in an agent registry, ensuring that only genuine policy exceptions are escalated to human supervisors.


### [Speculation Is All You Need](https://yomu.fyi/post/speculation-is-all-you-need.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 19, 2026
- Type: Explainer

Speculative decoding is presented as the most important optimization for LLM inference, delivering speedups of 2x-3x rather than small percentage gains from kernel tuning. The technique accelerates the decode phase by using a lightweight draft model to generate candidate tokens that the target model processes in parallel, with lossless resampling. The authors released DFlash speculators for multiple Qwen models, achieving an additional 5-20% speedup over existing baselines, and worked with SGLang to integrate them. On a B200 node, the Qwen 3.5 122B-A10B model with speculation reaches over 1000 tok/s at concurrency 1, compared to 250 tok/s without speculation, demonstrating a 4x speedup. Training these speculators is simplified because the data-generating process is the target model itself, making data collection trivial, and the post envisions a flywheel where custom speculators and distillation enable continuous improvement in latency, cost, and quality for self-hosted inference.


### [Why everyone is talking about real-time analytics (a note from “the yellow company”)](https://yomu.fyi/post/why-everyone-is-talking-about-real-time-analytics-a-note-from-the-yell.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 18, 2026
- Type: Explainer

Real-time analytics is presented as a foundational workload for customer-facing analytics, observability, fraud detection, operational applications, and AI agents rather than a niche use case. The discussion argues that performance claims require open, transparent, reproducible benchmarks that disclose methodology, datasets, queries, results, and operating conditions, including continuous ingestion, concurrency, memory pressure, and fresh data. A real-time database should support queryable data within milliseconds to a couple of seconds, incremental transformations, bounded tail latency as writes increase, configurable read-time versus write-time work, unified support for text, vector, JSON, and structured analytics, and efficiency as workload scales. The article connects renewed attention to agentic workloads, whose continuous, high-volume querying makes low latency, concurrency, and fresh context increasingly necessary, and calls for shared standards beyond a single latency number.


### [MosaicLeaks: Can your research agent keep a secret?](https://yomu.fyi/post/mosaicleaks-can-your-research-agent-keep-a-secret.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Alexander Gurung, Rafael Pardinas
- Published: Jun 18, 2026
- Type: Benchmark

MosaicLeaks examines privacy leakage in deep-research agents that combine private enterprise documents with external web retrieval. Queries that appear harmless individually can reveal sensitive facts when an observer combines them over multiple hops, producing intent, answer, or full-information leakage. The benchmark contains 1,001 multi-hop chains interleaving local and web questions, and compares baseline behavior with task-only reinforcement learning and Privacy-Aware Deep Research (PA-DR). Task-only training raises strict chain success from 48.7% to 59.3% but increases answer/full-information leakage from 34.0% to 51.7%, whereas PA-DR reaches 58.7% success with 9.9% leakage by rewarding stage-specific decisions and penalizing query-level privacy risk. The evaluation is controlled, using synthetic enterprise documents, a fixed web corpus, three company contexts, and one multi-hop question-answering harness.


[Newer posts](https://yomu.fyi/topic/llm/page/17.md) · [Older posts](https://yomu.fyi/topic/llm/page/19.md)