---
title: "LLMs"
description: "673 posts about LLMs, summarised, each linking to the original."
---

# LLMs
> 673 posts about LLMs, summarised, each linking to the original.

## Articles

### [Architecting offline-first generative AI applications for edge deployments using AWS services](https://yomu.fyi/post/architecting-offline-first-generative-ai-applications-for-edge-deploym.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Ahmed Ewis
- Published: Jul 22, 2026
- Type: Problem & solution

Industrial edge environments often face significant unplanned downtime yet lack reliable cloud connectivity to run large-scale artificial intelligence models. To address this limitation, an offline-first reference architecture shifts generative inference to edge devices while leveraging cloud services for preparation and lifecycle management. The workflow uses Amazon Bedrock with Amazon Nova Pro to create labeled dataset pairs, Amazon SageMaker AI Pipelines to fine-tune a compact model, and AWS IoT Greengrass to orchestrate edge deployments. Local execution relies on Ollama, Strands Agents, and a CPU-based ChromaDB retrieval pipeline that consumes zero GPU VRAM. In evaluations across three foundation model judges, the fine-tuned gpt-oss-20b model combined with retrieval-augmented generation consistently outperformed the base model configuration across accuracy, completeness, and relevance metrics.


### [Claude Opus 4.6](https://yomu.fyi/post/claude-opus-4-6.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: Jul 22, 2026
- Type: Announcement

Claude Opus 4.6 improves on Opus 4.5 with stronger coding, planning, debugging, code review, long-running agentic work, and performance in large codebases. It is the first Opus-class model with a beta 1M-token context window, while API features include adaptive thinking, four effort levels, context compaction, and up to 128k output tokens. In reported evaluations, it achieves the highest score on Terminal-Bench 2.0, leads frontier models on Humanity’s Last Exam and BrowseComp, and exceeds GPT-5.2 by around 144 Elo points and Opus 4.5 by 190 points on GDPval-AA. The release also adds Claude Code agent teams, upgraded Excel capabilities, a PowerPoint research preview, and availability on claude.ai, the API, and major cloud platforms, with base pricing of $5/$25 per million tokens.


### [Introducing Sonnet 4.6](https://yomu.fyi/post/introducing-sonnet-4-6.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: Jul 21, 2026
- Type: Announcement

Claude Sonnet 4.6 is introduced as an upgrade across coding, computer use, long-context reasoning, agent planning, knowledge work, and design. It adds a 1M token context window in beta, becomes the default for Free and Pro users in claude.ai and Claude Cowork, and keeps Sonnet 4.5’s starting price of $3/$15 per million tokens. Early testing reports that users preferred it over Sonnet 4.5 in Claude Code roughly 70% of the time and over Opus 4.5 59% of the time. The release also reports gains on OSWorld computer use, prompt-injection resistance, OfficeQA document comprehension, and long-horizon planning, while adding adaptive and extended thinking, beta context compaction, and generally available code execution, memory, programmatic tool calling, tool search, and tool-use examples.


### [Setting a World Record for MoE Pre-Training on NVIDIA GB300 NVL72](https://yomu.fyi/post/setting-a-world-record-for-moe-pre-training-on-nvidia-gb300-nvl72.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Kirthi Devleker
- Published: Jul 21, 2026
- Type: Benchmark

Frontier AI pre-training increasingly relies on mixture of experts architectures, which lower compute per token while introducing severe all-to-all communication bottlenecks. To address these scaling limits, the NVIDIA GB300 NVL72 rack-scale system co-designs Blackwell Ultra GPUs, fifth-generation NVLink scale-up fabrics, and 800 Gbps SuperNIC scale-out networking. Testing on DeepSeek-V3 671B across 256 GPUs achieved a pre-training benchmark of 1,648 TFLOPs per GPU using Megatron Core. In addition to hardware co-design, ongoing software optimizations across Megatron Core, TorchTitan, and JAX contributed major throughput gains over time. Scaling from 256 to 1,024 GPUs sustained over 97% per-GPU performance efficiency across all evaluated frameworks by overlapping scale-out gradient traffic behind compute windows.


### [Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI](https://yomu.fyi/post/inside-nvidia-rubin-gpu-architecture-powering-the-era-of-agentic-ai.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Eduardo Alvarez
- Published: Jul 21, 2026
- Type: Explainer

Agentic artificial intelligence workloads require sustained multistep inference, demanding low per-step latency, high decode throughput, and efficient long-context processing. NVIDIA designed the Rubin GPU to deliver up to ten times more agentic throughput per unit of energy than Blackwell. The hardware combines 336 billion transistors across two dies unified by NV-HBI, up to 288 GB of HBM4 memory providing 22 TB/s peak bandwidth, and the third-generation Transformer Engine reaching up to 50 petaflops of NVFP4 compute. Performance optimizations include inline descriptor updates for the Tensor Memory Accelerator, doubled K-dimension instruction throughput, 3-bit lookup-table weights, and structured 2:4 sparse attention compression. At the rack level, Vera Rubin NVL72 leverages Intelligent Power Smoothing and DSX MaxLPS to enable up to 40% more GPUs within a fixed power budget.


### [NVIDIA Vera CPU: Olympus Cores Built for Maximum Single-Thread Performance in Agentic AI](https://yomu.fyi/post/nvidia-vera-cpu-olympus-cores-built-for-maximum-single-thread-performa.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Praveen Menon
- Published: Jul 21, 2026
- Type: Explainer

Agentic AI workloads shift critical execution paths onto CPUs through sandboxed environments that execute code, invoke tools, and manage runtime states. The NVIDIA Vera CPU addresses these compute requirements via Olympus cores built to maximize single-thread performance and instructions per cycle under full socket load. To handle irregular control flows and pointer-heavy structures, Olympus combines a ten-wide decode engine, neural branch prediction supporting two taken branches per cycle, dependency-breaking acceleration, and dedicated graph prefetchers. Across scale-up systems, dual-socket Vera configurations leverage second-generation NVLink-C2C interconnects to maintain a clean two-NUMA-node topology alongside PCIe 6.4, CXL 3.1, and Arm CCA confidential computing. These architectural choices provide predictable latency and memory bandwidth across highly concurrent reinforcement learning and agent loops.


### [Scaling document classification to 100k+ labels](https://yomu.fyi/post/scaling-document-classification-to-100k-labels.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jane Zhang, Arnav Singhvi
- Published: Jul 20, 2026
- Type: Benchmark

Large-taxonomy classification must map freeform text to 35,000–100,000 labels while meeting quality, cost, and throughput requirements. The benchmark compares vector search, a two-step vector-search-plus-AI-Classify workflow, and direct frontier-model calls across transaction, company, and biomedical datasets, using hybrid semantic and BM25 retrieval with Reciprocal Rank Fusion for the search stages. The combined workflow performed best, reaching 0.81 average accuracy versus 0.76 for Gemini 3.5 Flash, while costing roughly one-hundredth as much per document; its strongest average setting shortlisted the top 20 labels. Vector search alone was nearly 100 times cheaper than the workflow but scored more than 20 points lower, while full-taxonomy frontier calls could exceed context limits. Taxonomy updates can be handled by embedding and ingesting changed labels rather than retraining or redeploying classifiers.


### [Online Learning for Cost-Efficient LLM Routing](https://yomu.fyi/post/online-learning-for-cost-efficient-llm-routing.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Kedar Thakkar
- Published: Jul 20, 2026
- Type: Problem & solution

Ramp describes an internal LLM gateway that processes trillions of tokens daily and uses dynamic failure-aware routing to reduce AI spend. It combines an EWMA of genuine provider-side failure rates with Thompson Sampling over a Normal-Inverse-Gamma posterior for the mean and variance of log-latency. For each request, the router estimates the probability of outright failure or missing the caller's deadline, incorporates model cost, and orders options while preserving caller-defined model preferences and fallbacks. In a reranker with a six-second deadline, the strategy reacted to Gemini 3.1 Flash Lite rate limiting by switching to GPT 5.4 Nano Flex, whose latency matched Standard at half the price. Larger experiments reported more than 25% cost savings without degradation, while a streaming-equivalent deployment for Ramp Inspect produced 30% savings.


### [NVIDIA NVLink: The Scale-Up Network for AI Factories](https://yomu.fyi/post/nvidia-nvlink-the-scale-up-network-for-ai-factories.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Jul 20, 2026
- Type: Announcement

Scale-up networking has emerged as a critical architectural component for AI factories supporting trillion-parameter models, mixture-of-experts architectures, and disaggregated serving. NVIDIA presents its sixth-generation NVLink scale-up fabric and NVLink 6 Switch to address latency-sensitive all-to-all GPU communication and offload collective operations. In simulation benchmarks on models like DeepSeek-R1, Qwen 235B, and a simulated 2T parameter model, NVLink achieved up to 2.3X higher decode throughput than off-the-shelf Ethernet in a 72-accelerator domain. The architecture integrates NVLink-C2C to supply 1.8 TB/s of coherent CPU-GPU bandwidth in the Vera Rubin NVL72 platform, as well as NVLink Fusion for integrating semi-custom silicon XPUs. Through co-designed hardware, SHARP in-network compute, and NCCL libraries, the platform optimizes token throughput, factory uptime, and deployment economics.


### [How our universal content processing platform Riviera evolved for AI and beyond](https://yomu.fyi/post/how-our-universal-content-processing-platform-riviera-evolved-for-ai-a.md)
- Company: [Dropbox](https://yomu.fyi/company/dropbox.md)
- Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash
- Published: Jul 20, 2026

Dropbox developed Riviera, a centralized content processing platform that evolved from an internal file preview generator into a shared infrastructure supporting over 300 file formats. By decomposing complex conversion tasks into modular, reusable transformations, the system now powers search indexing, media transcoding, and AI context extraction across Dropbox products.


### [Coding Agent Horror Stories: The Agent That Deleted Production](https://yomu.fyi/post/coding-agent-horror-stories-the-agent-that-deleted-production.md)
- Company: [Docker](https://yomu.fyi/company/docker.md)
- Author: Jennifer Kohl
- Published: Jul 20, 2026
- Type: Announcement

Recent industry updates focus on AI agent security, container maintenance, and developer workflows within Docker ecosystems. The OpenAI and Hugging Face incident involved 17,600 attacker actions, demonstrating that AI agent security requires automated system controls to constrain, observe, and govern agents rather than relying on human review. In software maintenance, Docker Extended Lifecycle Support (ELS) addresses escalating supply-chain risks and MinIO end of life by maintaining patches, building software from source, and enforcing local developer policies. For continuous integration, Docker Sandboxes enable isolated AI agents to run Testcontainers tests, implement code fixes, and submit draft pull requests within GitHub Actions. Additionally, Docker Verified Publisher applications have transitioned to a self-serve process accessible directly through Docker Hub.


### [In-House LLM Serving at Netflix](https://yomu.fyi/post/in-house-llm-serving-at-netflix.md)
- Company: [Netflix](https://yomu.fyi/company/netflix.md)
- Author: Netflix Technology Blog
- Published: Jul 17, 2026
- Type: Problem & solution

Netflix established an in-house serving platform to run large language model inference directly inside existing production environments alongside traditional machine learning models. Built upon NVIDIA Triton Inference Server and vLLM, the unified architecture handles member-scale routing, candidate generation, feature fetching, and model execution over gRPC and OpenAI-compatible HTTP endpoints. The team shifted their primary engine from TensorRT-LLM to vLLM to support custom architectures, simpler debugging, and non-trivial constraint logic. Addressing production obstacles required patching Triton's frontend for guided decoding, pinning dependent library versions, rewriting logits processors in C++ for vLLM V1, and handling state machine resets during engine preemptions. The resulting platform unifies deployment flows while preserving operational stability across evolving model schemas.


### [The cost of saying yes has changed](https://yomu.fyi/post/the-cost-of-saying-yes-has-changed.md)
- Company: [Github](https://yomu.fyi/company/github.md)
- Author: Dalia Abuadas
- Published: Jul 17, 2026
- Type: Explainer

Generating initial code patches has become significantly cheaper with AI agents, shifting the primary expense of small feature requests from writing code to debating scope in meetings. Teams can use agent-generated patches as diagnostic probes rather than final deliverables, turning abstract scope debates into concrete artifacts that reveal true system touchpoints and risks. However, low generation costs do not translate to low ownership costs, as changes touching areas like authorization, compliance, or public contracts still demand significant human review. Constrained attempts allow engineers to price uncertainty quickly and shift scope discipline from pre-implementation speculation to evidence-based code review.


### [DE Dost: Teaching Machines to Be a Friend on the Last Mile](https://yomu.fyi/post/de-dost-teaching-machines-to-be-a-friend-on-the-last-mile.md)
- Company: [Swiggy](https://yomu.fyi/company/swiggy.md)
- Author: Aishwary Yadav
- Published: Jul 17, 2026
- Type: Explainer

DE Dost is a GenAI onboarding companion embedded in Swiggy’s delivery-executive app, designed for new DEs who receive information but lack confidence-building, immediate guidance. It supports questions in nine languages across SOPs, earnings, payouts, app navigation, shifts, and support, adapting detail and tone to conversational context. Before the first LLM call, a graph state assembles user context, preprocessing resolves references, and embedding-based semantic routing separates general knowledge-base questions from personal diagnostics. Retrieval uses authored QnA pairs plus hybrid semantic and lexical scoring to handle paraphrases and terms such as “MG” and “COD,” while guardrails constrain follow-ups to available knowledge. Evaluation spans about 800 cases per language across fleet and payout combinations, with macro accuracy trending at 90%+ and a target P50 latency of 4.5 seconds after reducing an initial 15-second pilot.


### [The skills gap behind agentic AI — and how Databricks is closing it with a new context engineer certification and agent trainings](https://yomu.fyi/post/the-skills-gap-behind-agentic-ai-and-how-databricks-is-closing-it-with.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Rachel Canetta, Trang Le
- Published: Jul 16, 2026
- Type: Announcement

Databricks introduced the Databricks Certified Context Engineer Associate beta exam to validate skills for building reliable, production-grade AI agent systems as organizations scale agentic AI. Context engineering is presented as the practice of curating, maintaining, and filtering tokens, memory banks, and tool parameters so an LLM can solve a task; the certification is intended to benchmark that technical skill set. Databricks also added AI Agent Fundamentals, Building Retrieval Agents on Databricks, and Agent Evaluation, covering agent reasoning, retrieval-augmented architectures, and systematic performance testing and improvement. Its AI-first certification-prep guide is available on every certification page, works with free tiers of major LLMs, includes guardrails and disclaimers about hallucinations, and shows candidates how to use Free Edition for hands-on practice; registration for the Context Engineer Associate is open, with the first exam scheduled for July 29, 2026.


### [Integrating Context-Aware Video AI Agents Into Enterprise Workflows](https://yomu.fyi/post/integrating-context-aware-video-ai-agents-into-enterprise-workflows.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Jul 16, 2026
- Type: Problem & solution

Video analytics AI agents often struggle to integrate into enterprise workflows because video platforms, organizational knowledge bases, and operational systems are isolated. To resolve this, NVIDIA NemoClaw orchestrates the Metropolis Video Search and Summarization blueprint alongside the Retrieval-Augmented Generation blueprint into an automated pipeline. Upfront human-in-the-loop interactions collect analysis parameters, objects of interest, and relevant retrieval queries before video ingestion begins. The system combines hierarchical video summaries with contextual enterprise reference documents to generate structured reports with timestamps and citations. Finally, NemoClaw executes programmatic downstream tasks, such as creating prioritized Jira tickets or dispatching industrial maintenance work orders directly into management tools.


### [Scaling Agentic AI Factories Through Extreme Co-Design with NVIDIA BlueField](https://yomu.fyi/post/scaling-agentic-ai-factories-through-extreme-co-design-with-nvidia-blu.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Jul 16, 2026
- Type: Explainer

Agentic artificial intelligence workflows transform data center infrastructure by turning context management, security enforcement, and inter-service communication into active components of the inference data path. To prevent host central processing units and graphics processing units from stalling on these tasks, NVIDIA introduces BlueField-4 data processing units, Vera BlueField-4 STX storage processors, and the DOCA software platform. The BlueField-4 processor offloads Layer 3 routing, encryption, and telemetry using a sixty-four-core Grace processor with up to eight hundred gigabits per second connectivity. Simultaneously, the Vera BlueField-4 STX powers the NVIDIA CMX context memory platform alongside DOCA Memos to store, protect, and share key-value cache across flash storage tiers. This co-designed infrastructure isolates tenant traffic and accelerates context reuse without degrading host-side agent execution.


### [AI Agents Explained: How to Build with Them Safely](https://yomu.fyi/post/ai-agents-explained-how-to-build-with-them-safely.md)
- Company: [Docker](https://yomu.fyi/company/docker.md)
- Author: Jin Kim
- Published: Jul 16, 2026
- Type: Explainer

AI agents combine reasoning models, external tools, memory, orchestration, and execution environments to autonomously pursue goals across iterative perceive-reason-act loops. While simple chatbots only generate text responses to direct prompts, autonomous agents execute multi-step operations such as querying databases, modifying codebases, and executing shell commands without requiring continuous manual approvals. However, this unchecked operational autonomy significantly widens the blast radius when models encounter logic errors or vague instructions, risking destructive filesystem operations and credential leaks. Safe deployment requires shifting focus from model-level guardrails to infrastructure-level containment strategies, such as scoping tool access and isolating execution inside disposable microVM sandboxes. Controlling networking boundaries, file access, and credential exposure ensures teams achieve automated execution velocity without exposing developer host environments to uncontained systemic risk.


### [Your agent should understand what you see](https://yomu.fyi/post/your-agent-should-understand-what-you-see.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Mihir Mavalankar
- Published: Jul 16, 2026
- Type: Problem & solution

Sentry's Seer Agent initially parsed user screens using DOM-scraped character-by-character ASCII grids, leading to severe token consumption, model context rot, and an inability to support interactive actions. To resolve these bottlenecks, Sentry replaced visual text scraping with a composable semantic context system where React UI components directly declare their state, metadata, and interactive affordances. A root React context provider maintains a registry of component nodes organized hierarchically through higher-order wrappers, serializing the resulting structure to JSON and backend-generated Markdown within the system prompt. Across twenty-four live pages, this approach reduced dashboard context tokens from roughly 5,500 to 1,300 while maintaining identical satisfaction rates and tool-call volumes across more than 14,000 evaluated conversations.


### [Security incident disclosure — July 2026](https://yomu.fyi/post/security-incident-disclosure-july-2026.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: system
- Published: Jul 16, 2026
- Type: Incident / postmortem

An intrusion into part of production infrastructure was driven end to end by an autonomous AI agent system, accessing a limited set of internal datasets and several credentials used by services. The attack began through two code-execution paths in dataset processing, then escalated from a worker to node-level access and moved laterally across internal clusters while running thousands of actions in short-lived sandboxes. AI-assisted anomaly detection surfaced the compromise, and LLM-driven analysis of more than 17,000 attacker events reconstructed the timeline, identified indicators, mapped credentials, and separated impact from decoys. Hosted frontier models initially blocked forensic requests containing attack artifacts, so the investigation used zai-org/GLM-5.2 on internal infrastructure. The initial vulnerabilities were closed, affected infrastructure rebuilt, credentials rotated, and cluster controls and alerting strengthened; assessment of possible partner or customer impact remains ongoing.


[Newer posts](https://yomu.fyi/topic/llm/page/14.md) · [Older posts](https://yomu.fyi/topic/llm/page/16.md)