---
title: "LLMs"
description: "673 posts about LLMs, summarised, each linking to the original."
---

# LLMs
> 673 posts about LLMs, summarised, each linking to the original.

## Articles

### [Agentic Testing: Where Agents Fit in the E2E Testing Stack](https://yomu.fyi/post/agentic-testing-where-agents-fit-in-the-e2e-testing-stack.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Sergii Gorbachov
- Published: Jun 11, 2026
- Type: Benchmark

Traditional end-to-end tests validate rigid user journeys, whereas agentic tests verify whether broad goals can be achieved by adapting actions dynamically. To evaluate agentic testing tradeoffs, researchers executed over 200 runs across Playwright Model Context Protocol (MCP), Playwright CLI, and agent-generated Playwright tests using Claude models. Playwright MCP demonstrated high reliability with failure rates of 0% on simple thread replies and approximately 12% on complex search discovery flows. Playwright CLI and generated code struggled more on complex workflows, exhibiting failure rates of approximately 20% and 48% respectively. Although generated tests were faster with average runtimes of roughly three minutes, agentic testing provides a distinct exploratory layer atop deterministic CI test suites.


### [Making FlashAttention-4 faster for inference](https://yomu.fyi/post/making-flashattention-4-faster-for-inference.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Charles Frye
- Published: Jun 11, 2026
- Type: Explainer

FlashAttention-4, originally designed for pre-training, required modifications to efficiently handle LLM inference workloads dominated by the memory bandwidth-limited decode phase. Modal’s contributions include adjusting parallelism strategies from query parallelism to key/value parallelism, implementing split KV to parallelize across KV tiles, and adding a heuristic for optimal splits. They added support for FP8 inputs, arbitrary KV page sizes via cp.async loads, and reduced wasted work for small query sizes by using a single query tile. The kernel also gained extended GQA packing for irregular head ratios, using cp.async for query tiles to bypass TMA alignment restrictions. These changes, measured in throughput improvements up to 3x for single-token decode and up to 2.4x for small page sizes, demonstrate the leverage of flexible parallelism and tile-based programming for inference optimization.


### [Catch visual regressions with Snapshots, now in beta](https://yomu.fyi/post/catch-visual-regressions-with-snapshots-now-in-beta.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Max Topolsky
- Published: Jun 11, 2026
- Type: Announcement

Sentry has released the beta version of Sentry Snapshots to catch visual regressions on pull requests across any platform with a frontend. The tool captures screenshots of an application across various viewports, themes, and languages, compares them against a baseline, and blocks pull requests when diffs are detected. Teams can attach custom context metadata, including test file paths, to help developers and AI agents locate test sources and generate additional test cases. By leveraging the sentry-mcp integration, REST API, and sentry-cli, workflows can retrieve base images, diff masks, and run local diff verification before pushing commits. Originally created at Emerge Tools for mobile applications, this iteration expands automated visual testing capabilities to broader frontend codebases.


### [Metric Semantic Layer: How Lyft Governs and Scales Key Data Definitions](https://yomu.fyi/post/metric-semantic-layer-how-lyft-governs-and-scales-key-data-definitions.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Iraklikhorguani
- Published: Jun 10, 2026
- Type: Problem & solution

As Lyft scaled, different teams developed conflicting definitions for key business metrics due to the lack of centralized version control and shared standards. To resolve this, Lyft implemented an internal Metric Semantic Layer as a Python package that stores authoritative metric definitions in YAML files with Jinja SQL templates. The system restricts onboarding to Golden Metrics used across multiple applications, requiring team-based approval from both Business and Operational Owners for any definition changes. Standardized definitions are exposed through Python APIs, integrated into the Amundsen data catalog and self-service user interfaces, and surfaced to AI tools via a Model Context Protocol.


### [Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant](https://yomu.fyi/post/encoding-your-domain-expert-the-context-layer-behind-spotify-s-data-as.md)
- Company: [Spotify](https://yomu.fyi/company/spotify.md)
- Author: Spotify Engineering
- Published: Jun 10, 2026
- Type: Problem & solution

Spotify developed an AI data assistant, Vedder, to scale access to over 70,000 internal datasets without overwhelming domain experts. Because raw warehouse schemas omit critical semantic nuances and overwhelm LLM context windows, engineers implemented a domain-owned context layer organized into clusters. Each cluster contains profiled datasets, expert-curated business documentation, and vetted question-and-SQL pairs that guide a ReAct-based agent. Automated query history extraction proved insufficient on its own, as domain curators accepted only 12.5% of inferred query pairs due to noise. Deployed across Slack, IDEs, and a dedicated web interface, the system maintains reliability through continuous cluster health monitoring that flags schema drift and degraded examples.


### [DiffusionGemma: The Developer Guide](https://yomu.fyi/post/diffusiongemma-the-developer-guide.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Ian Ballantyne, Omar Sanseviero
- Published: Jun 10, 2026
- Type: Tutorial

DiffusionGemma is an experimental text-generation model and developer guide covering serving, customization, and non-autoregressive generation. Built on the Gemma 4 backbone, its 26B Mixture of Experts architecture activates 3.8B parameters during inference, refines 256-token canvases in parallel, and shifts the bottleneck from memory bandwidth to compute. Block autoregressive diffusion commits denoised 256-token blocks to the KV cache, combining bidirectional denoising with causal prefill for longer sequences; vLLM integration provides an OpenAI-compatible local server. A Sudoku fine-tuning example using Hackable Diffusion reports improvement from roughly 0% base-model success to 80%, while SFT reduces the demonstrated solve from 48 to 12 steps through adaptive stopping. The guide also identifies bidirectional attention and re-noising as mechanisms for global constraint handling and self-correction.


### [Quick: An internal hosting platform for the AI era](https://yomu.fyi/post/quick-an-internal-hosting-platform-for-the-ai-era.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 10, 2026
- Type: Explainer

Shopify built Quick to make internally created prototypes, dashboards, tools, and games easier to share than deploy, giving employees a secure URL for a folder of HTML and assets. Launched internally in July 2025, it places sites behind Identity-Aware Proxy, serves bucket content through NGINX and gcsfuse, and deploys with a wrapper around gcloud rsync. A service in front of CloudSQL provides database and realtime APIs, while server-held credentials support calls to Shopify’s AI proxy; file uploads, BigQuery access, WebSockets, and employee identity are also available. Quick now hosts more than 50,000 sites, has been used by over half of Shopify employees, and runs on one $200-per-month VM, with constraints, rate limiting, and a migration from Node to Go supporting its operation.


### [ClickHouse Agents: Claude-powered agentic analytics, now in public beta](https://yomu.fyi/post/clickhouse-agents-claude-powered-agentic-analytics-now-in-public-beta.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 9, 2026
- Type: Announcement

ClickHouse announced the public beta of ClickHouse Agents, a fully managed agentic analytics service in ClickHouse Cloud powered by Claude. It provides a no-code builder for agents grounded in live ClickHouse data, plus a managed chat experience in the Cloud console, so users need no SQL or separate hosting setup. Built on LibreChat, the service supports natural-language questions, SQL and visualizations, sandboxed Bash, Python, and JavaScript execution, shareable artifacts, skills, memories, MCP-compatible connections, and multi-agent workflows. Claude Sonnet and Haiku are pre-configured, while enterprise controls include SSO, encrypted message storage, and governed data access. ClickHouse says its internal deployment handles roughly 70% of data warehouse queries and processes more than 45 million tokens daily, confirming its view that real-time analytics is a natural context for high-volume agent workloads.


### [Real-world grounding in agentic AI](https://yomu.fyi/post/real-world-grounding-in-agentic-ai.md)
- Company: amazon.science
- Author: Rose Yu
- Published: Jun 8, 2026
- Type: Problem & solution

Foundation models deployed as physical AI agents face critical risks when natural language hallucinations translate into physical law violations and operational hazards. To ensure physical consistency and reliability, researchers from the University of California, San Diego and Amazon Fulfillment Technology introduced four grounding pillars. Physics-guided deep learning embeds symmetries and differential equations into pretraining to obey conservation laws while reducing dataset requirements. The UQ4CT framework applies mixture-of-experts architectures to produce calibrated uncertainty estimates, cutting expected calibration error by over 25% and halting operations when safety thresholds are breached. Furthermore, the Adapting-While-Learning framework and formal verifiers like Hilbert integrate numerical simulators and interactive proof systems to enforce mathematical and scientific correctness.


### [Bridging intent and execution in agentic systems](https://yomu.fyi/post/bridging-intent-and-execution-in-agentic-systems.md)
- Company: amazon.science
- Author: Gaurav Gupta, Vatshank Chaturvedi
- Published: Jun 8, 2026
- Type: Problem & solution

AI agent performance increasingly depends on the harness mediating interactions between language models and tools rather than raw model reasoning. To address mismatches between model intent and harness execution, the authors developed Simple Strands Agent, an open-source single-agent harness. The architecture implements robust tool-interface practices, such as condensing middle log outputs from bash commands, enforcing line-aware string replacements with ambiguity checks, and returning diff files after edits for verification. Evaluated across benchmarks like SWE-Bench-Verified, SWE-Bench-Pro, and Terminal-Bench 2, these model-harness alignments yielded consistent performance gains across frontier models without task-specific tuning. The findings show that model families require tailored interface codesign alongside stable evaluation infrastructure to achieve optimal benchmark reliability.


### [Works on my machine: how we use AI to reproduce reported bugs](https://yomu.fyi/post/works-on-my-machine-how-we-use-ai-to-reproduce-reported-bugs.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Neel Shah
- Published: Jun 8, 2026
- Type: Problem & solution

Sentry maintains 159 open-source SDK packages across multiple languages, leading to high triage volume and significant boilerplate setup overhead when reproducing reported bugs. To streamline this workflow, the team developed a Claude skill that parses GitHub issue URLs to extract language, framework, and SDK versions. The skill generates a reproduction branch, sets up minimal boilerplate using ecosystem package managers like uv or npm, attempts execution, and opens a pull request with explicit run instructions. By allowing the model to exit early when encountering overly complex environments, the tool reliably prepares reproducible playgrounds without stumbling on edge cases. This workflow saved several hours in testing scenarios like Django HTTPS proxy bugs while avoiding excessive bot noise through scoped human interaction.


### [What is AI Search?](https://yomu.fyi/post/what-is-ai-search.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 4, 2026
- Type: Explainer

AI search interprets natural-language questions using semantic understanding and large language models, then returns synthesized answers grounded in cited sources. Unlike traditional search, which matches query words to indexed pages, it tracks context and intent for conversational follow-ups. Its pipeline includes query understanding, embeddings, vector search, retrieval, LLM synthesis, and citations; retrieval-augmented generation (RAG) connects generated responses to source material. The article distinguishes consumer tools that search the open web from enterprise systems that retrieve private, governed company data while respecting access permissions. It concludes that AI search can improve direct information access but should not be treated as definitive, and that sensitive business use requires purpose-built platforms with governance, security, and retrieval quality.


### [Nemotron 3.5 Content Safety: Customizable Multimodal Safety for Global Enterprise AI](https://yomu.fyi/post/nemotron-3-5-content-safety-customizable-multimodal-safety-for-global.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Varun Singh, Isabel Hulseman, Anuj Doshi, Shyamala Prayaga
- Published: Jun 4, 2026
- Type: Announcement

Nemotron 3.5 Content Safety is NVIDIA’s 4B-parameter model for evaluating user prompts, optional images, and optional assistant responses together, with multilingual coverage and enterprise-specific policy enforcement. It extends Nemotron 3 by accepting natural-language custom policies, producing optional auditable reasoning traces, and supporting binary verdicts, category-labelled verdicts, or THINK mode; disabling THINK preserves the default low-latency path. Built on Google Gemma 3 4B IT with a LoRA adapter and a 128K context window, it uses the Aegis 2.0 taxonomy of 13 core categories and 10 subcategories, and targets deployment on GPUs with 8GB or more of VRAM. The accompanying multimodal, multilingual dataset includes concise reasoning traces, while reported results average 96.5% on Multilingual Aegis, 88.8% on RTP-LX, and about 85% across the evaluated benchmark set.


### [Playing a different game](https://yomu.fyi/post/playing-a-different-game.md)
- Company: [Intercom](https://yomu.fyi/company/intercom.md)
- Author: Eoghan McCabe
- Published: Jun 4, 2026
- Type: Announcement

Fin Voice 2 is presented as a customer-service voice agent designed to address the latency, complexity, and reliability demands of phone support. Built on Apex Flash, described as a model trained on millions of customer experience interactions and tuned for low-latency channels, it separates real-time speech processing from the answer-generation layer rather than routing speech through a general text workflow. The product is claimed to deliver faster, more natural conversations, higher resolution rates, and higher customer satisfaction, though the post does not provide measurements. It can reportedly handle identity verification, refunds, appointments, emotional-state changes, confirmations, external-system actions, proactive follow-up calls, and contextual handoffs to human agents. Fin Voice 2 also adds configuration and management features, including call-quality and behavior insights, topic analysis, one-click recommendations, and self-service controls.


### [Designing the hf CLI as an agent-optimized way to work with the Hub](https://yomu.fyi/post/designing-the-hf-cli-as-an-agent-optimized-way-to-work-with-the-hub.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Célina Hanouti, Lucain Pouget
- Published: Jun 4, 2026
- Type: Benchmark

The post presents a redesign of hf, the Hugging Face Hub’s official CLI, for both human users and coding agents that download, upload, manage repositories, run Jobs, and administer other Hub resources. The CLI detects agents through environment variables, switches to compact, complete TSV output, and supports explicit human, agent, JSON, and quiet formats; agent mode was introduced in hf v1.9.0. Next-command hints, stderr separation, non-interactive failure messages, retry-safe operations, confirmations, and dry-run previews support multi-step automation without removing human-friendly terminal behavior. Benchmarks report equal-or-better success with roughly 1.3× to 1.8× fewer tokens than curl or the SDK on comparable tasks, and up to 6× fewer on complex workflows; the generated hf-cli skill reduced tool calls by about 30% without reducing token use or improving reliability.


### [Click-a-thon 2026: 24 hours, Bengaluru, and the fastest analytical database on the planet](https://yomu.fyi/post/click-a-thon-2026-24-hours-bengaluru-and-the-fastest-analytical-databa.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 4, 2026
- Type: Announcement

ClickHouse announces Click-a-thon 2026, its first in-person hackathon in India, scheduled for Bengaluru on August 1–2, 2026. The event invites working tech professionals to form teams of two to four for a 24-hour build with four tracks—Real-Time Analytics, Observability, Data Warehousing, and Agentic AI and Analytics—and a ₹10,00,000 prize pool. Its stated purpose is to bring developers together to build practical systems under tight constraints and experiment with ClickHouse alongside modern open-source tools. The post points to ClickStack for observability, Langfuse for LLM tracing and evaluation, and LibreChat for model and agent interfaces, while describing combinations such as streaming ingestion, materialized views, projections, analytical queries, and tool-using agents over data. Applications close June 25, with shortlists planned for the first week of July; the team captain applies on behalf of the team.


### [Agentic Supplier Management with MongoDB Atlas, Voyage AI, and Multi-Modal Search](https://yomu.fyi/post/agentic-supplier-management-with-mongodb-atlas-voyage-ai-and-multi-mod.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Ronan Conlon
- Published: Jun 3, 2026
- Type: Problem & solution

Retailers face major operational bottlenecks when managing supply chain disruptions through legacy enterprise resource planning systems and siloed documents. Traditional tabular structures struggle to index multi-modal data like images and PDF contracts, delaying critical supplier discovery during regional crises or tariff changes. To overcome these constraints, organizations decouple supplier management into a dedicated operational data layer using MongoDB Atlas and Voyage AI. This architecture stores polymorphic supplier profiles and high-dimensional vector embeddings within a unified collection, enabling natural language semantic search for alternate partners. Real-time updates propagate across systems using MongoDB Change Streams, allowing businesses to rapidly assess supply chain impacts and identify alternative vendors in minutes.


### [Ground truth is a process, not a dataset](https://yomu.fyi/post/ground-truth-is-a-process-not-a-dataset.md)
- Company: amazon.science
- Author: Venkatesh Saligrama
- Published: Jun 3, 2026
- Type: Problem & solution

Evaluating search-augmented AI research reports requires verifying complex, multi-source claims that depend heavily on surrounding context. Unassisted PhD-level domain specialists tasked with establishing static ground-truth labels achieved only 60.8% accuracy on a hidden test set during controlled testing. To resolve benchmark errors on cognitively demanding tasks, Amazon AGI introduced the audit-then-score evaluation protocol alongside DeepFact-Bench and the DeepFact-Eval verification system. Under this protocol, model disagreements serve as challenges requiring concrete counterevidence and written rationales, enabling human auditors to evaluate competing claims and update benchmark ground truth prior to scoring. Operating under this iterative workflow, expert test accuracy rose to 90.9% across four rounds, and DeepFact-Eval reached 83.4% accuracy with GPT-4.1.


### [Coding Is No Longer the Constraint: Scaling Developer Experience to Teams and Agents at Spotify](https://yomu.fyi/post/coding-is-no-longer-the-constraint-scaling-developer-experience-to-tea.md)
- Company: [Spotify](https://yomu.fyi/company/spotify.md)
- Author: Spotify Engineering
- Published: Jun 3, 2026
- Type: Problem & solution

Spotify experienced rapid codebase growth alongside widespread adoption of AI coding tools, resulting in a 76% increase in pull request frequency. To manage fleet-wide codebase maintenance and complex refactoring, the company combined its Fleetshift automation platform with Honk, a background coding agent powered by Claude. Honk runs inside Kubernetes pods with access to CI build environments and integrates with Backstage via Model Context Protocol tools to inspect component metadata. Standardized service architectures and active lint guardrails enable the agent to maintain consistency and auto-correct invalid design patterns during execution. By automating routine migrations, Spotify reduced fleet-wide Java backend updates from multi-week cross-team efforts down to three days handled by a single engineer.


### [Direct Preference Optimization Beyond Chatbots](https://yomu.fyi/post/direct-preference-optimization-beyond-chatbots.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Erick Lachmann, Gabriel Pimenta de Freitas Cardoso, Francisco de Almeida Rocha Alves
- Published: Jun 3, 2026
- Type: Explainer

DharmaOCR applies Direct Preference Optimization (DPO) beyond chat alignment to reduce text degeneration in structured OCR, where outputs are either acceptable transcriptions or failure loops. In a benchmark of Brazilian Portuguese structured-document OCR, vanilla degeneration rates among open-source model families ranged from below 1% to above 33%, while supervised fine-tuning (SFT) did not consistently reach production-acceptable levels. After SFT, the pipeline generated multiple candidates for 23,726 documents, used an automated LLM judge to score them, and retained the model’s degenerate outputs as rejected examples paired with chosen outputs. DPO reduced degeneration in every tested family, by 59.4% on average and up to 87.6%, without reducing extraction quality according to the benchmark; applicability is presented as dependent on failures being categorically identifiable, reliably scoreable, and sufficiently numerous.


[Newer posts](https://yomu.fyi/topic/llm/page/19.md) · [Older posts](https://yomu.fyi/topic/llm/page/21.md)