---
title: "LLMs"
description: "673 posts about LLMs, summarised, each linking to the original."
---

# LLMs
> 673 posts about LLMs, summarised, each linking to the original.

## Articles

### [34 Amazon Research Awards Build on Trainium recipients announced](https://yomu.fyi/post/34-amazon-research-awards-build-on-trainium-recipients-announced.md)
- Company: amazon.science
- Author: Amazon Research Awards team
- Published: Aug 5, 2026
- Type: Announcement

Amazon announced 34 recipients from 30 universities for the Fall 2025 Build on Trainium: Responsible AI call for proposals. Part of a 110 million dollar credit initiative, the program funds academic research across new model architectures, machine learning libraries, optimizations, and distributed systems running on AWS Trainium hardware. The funded projects center on five key areas: AI safety and alignment, multilingual language models, representation engineering, sustainability with small language models, and deep learning for synthetic data generation. Awardees receive AWS Promotional Credits, access to more than 700 Amazon public datasets, assigned Amazon research contacts, and dedicated technical training resources. Highlighted efforts include parallelization strategies for trillion-parameter mixture-of-experts models scaling across 1,024 Trainium chips and inference optimization frameworks designed to enhance token efficiency.


### [Explorers, exploiters, and the myth of the 100x engineer](https://yomu.fyi/post/explorers-exploiters-and-the-myth-of-the-100x-engineer.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Eira May
- Published: Aug 5, 2026
- Type: Explainer

Engineering leadership often responds to rapid AI productivity gains by attempting to isolate and replicate the traits of individual outlier engineers. Vivek Raghunathan, SVP of engineering at Snowflake, frames this dynamic using reinforcement learning's explore-versus-exploit split, where roughly five percent of an organization fearlessly tests new AI tools while ninety-five percent prioritize execution on paved paths. Instead of treating these categories as fixed personality types or relying on external hiring, organizations succeed by viewing them as a continuum. Effective leadership establishes structured learning time, AI communities of practice, and direct mentorship to capture explorer discoveries and elevate the median engineer. Measuring progress requires tracking organizational movement along this capability scale rather than celebrating isolated high-performing individuals.


### [Sidekick's continual learning loop](https://yomu.fyi/post/sidekick-s-continual-learning-loop.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 5, 2026
- Type: Explainer

Frontier models can accelerate an AI product launch, but their serving cost and latency can become impractical as usage grows, while their weights do not learn from production failures. Shopify’s proposed flywheel captures low-quality conversations, defines quality with a rubric and calibrated judges, improves the surrounding harness through autoresearch, and then converts repaired trajectories into supervised fine-tuning and GRPO training signal. A panel of frontier reasoning models critiques failures, an arbiter produces repair instructions, and unresolved cases go to expert annotators; gist compression then shortens the agent’s static prompt without measured judge-quality loss. In production, the GraphQL agent serves up to 2,000 requests per minute and reportedly achieves a 96% serving-cost reduction versus an estimated frontier-model cost, with lower latency and higher throughput. The approach is presented as a continual-learning system that turns production experience into model-weight updates.


### [Turn one giant AI-generated pull request to a reviewable stack](https://yomu.fyi/post/turn-one-giant-ai-generated-pull-request-to-a-reviewable-stack.md)
- Company: [Github](https://yomu.fyi/company/github.md)
- Author: Julia Muiruri
- Published: Aug 4, 2026
- Type: Problem & solution

Coding agents often consolidate full-stack features into monolithic pull requests spanning thousands of lines, resulting in delayed reviews and lower feedback quality. To resolve this bottleneck, engineers can decompose comprehensive features into ordered, single-concern layers using GitHub stacked pull requests and the gh-stack CLI extension. Distinct specialized agents handle individual branches—ranging from foundational data access modules up to client interfaces—while CI checks continuously evaluate each layer against the stack base. When reviewers request upstream modifications, running gh stack sync executes a cascading rebase across all dependent branches, preserving commit signatures and keeping the entire pull request chain mergeable.


### [Unity AI Gateway is Generally Available](https://yomu.fyi/post/unity-ai-gateway-is-generally-available.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: David Nasi, Stefania Leone, Kevin Stumpf, Kasey Uhlenhuth, Kelly Albano
- Published: Aug 4, 2026
- Type: Announcement

Unity AI Gateway is generally available as a centralized way for enterprises to manage AI costs, security, governance, and model choice across agents and other AI assets. It positions Unity AI Gateway alongside Unity Catalog: Unity Catalog supplies identity, permissions, lineage, and auditing, while the gateway applies runtime guardrails and contextual policies across AI interactions. The gateway centralizes usage data in Unity Catalog, offering dashboards, Genie-powered analysis, granular cost attribution, budgets, hard spend caps, and a single-query API for frontier models from providers including Anthropic, OpenAI, Gemini, Kimi, and GLM. Smart Routing is in beta and dynamically selects models using quality, cost, performance, availability, and budget; the announcement says customers already use the gateway at scale, including more than one quadrillion tokens passing through it in the past year.


### [The Software Supply Chain Is Under Siege. Devs Are Still the First Line of Defense](https://yomu.fyi/post/the-software-supply-chain-is-under-siege-devs-are-still-the-first-line.md)
- Company: [Docker](https://yomu.fyi/company/docker.md)
- Author: Jin Kim
- Published: Aug 4, 2026
- Type: Explainer

A research report by Omdia examines escalating software supply chain security risks driven by growing dependencies on third-party code, open-source software, and artificial intelligence. Over three-fourths of surveyed organizations experienced a software supply chain incident in the preceding year, with known vulnerabilities in third-party software representing the most common exploit vector. Surveyed respondents ranked AI technology as their top supply chain concern, alongside broader issues involving vulnerability remediation and code composition identification. While shifting security left is a high priority for 98% of organizations, 45% still lack confidence in the robustness of their supply chain defenses. Although secure container libraries and Software Bills of Materials provide measurable mitigation benefits, many organizations only generate inventories on a case-by-case basis.


### [Generate Trajectories, Reasoning Traces, and Auto-Labels with NVIDIA Alpamayo 2 Super](https://yomu.fyi/post/generate-trajectories-reasoning-traces-and-auto-labels-with-nvidia-alp.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Aug 4, 2026
- Type: Announcement

Autonomous vehicle development typically relies on disjoint models for trajectory generation, intent prediction, scene understanding, and data labeling. NVIDIA Alpamayo 2 Super addresses this fragmentation with an open 34-billion-parameter vision-language-action model that couples a 32-billion-parameter Cosmos 3 Super Reasoner with a 2-billion-parameter diffusion-based Action Expert. Processing surround-view video across up to seven cameras, the model simultaneously produces future vehicle trajectories, Chain-of-Causation reasoning traces, high-level meta-actions, and grounded scene question answers. In benchmark evaluations, the model achieved a 6.4-second minADE\_6 of 0.911 meters on the Physical AI AV Dataset and ranked first on LingoQA with a score of 79.2. Released under the OpenMDW-1.1 permissive license, the architecture serves as an offline policy teacher, evaluation critic, or auto-labeling engine that can be distilled for NVIDIA DRIVE AGX Thor deployment.


### [Kimi K3: the complete developer guide](https://yomu.fyi/post/kimi-k3-the-complete-developer-guide.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Aug 1, 2026
- Type: Tutorial

Kimi K3 is presented as Moonshot AI’s 2.8-trillion-parameter open-weights model for long-horizon coding, end-to-end knowledge work, and deep reasoning, served through Together AI’s OpenAI-compatible API. Its architecture combines Kimi Delta Attention (KDA), Attention Residuals, and Stable LatentMoE, which activates 16 of 896 experts per token—roughly 2%—and supports a 1M-token context. The implementation guide shows Python calls for configurable reasoning effort, reasoning-free instant mode, streaming, multi-image vision, strict JSON-schema output, and function tools, while noting automatic prefix caching. It also covers production-oriented details such as generous max\_tokens for reasoning-heavy structured output, a 100 MB request-body limit for vision, tool-call history handling, and deployment from US infrastructure.


### [Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference](https://yomu.fyi/post/co-designing-ai-model-attention-for-fast-interactive-long-context-infe.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Jul 31, 2026
- Type: Explainer

As context lengths increase in long-context and agentic workloads, attention increasingly dominates GPU inference runtime. Dense attention performance depends on architectural parameters like group size, head dimension, and sequence length, alongside GPU parallelization strategies. Prefill remains compute-bound and scales quadratically with input sequence length, showing minimal runtime sensitivity to group size variations. Conversely, decode generation is memory-bound and scales linearly with KV cache length, where doubling group size roughly halves decode execution time. To maximize throughput and interactivity on NVIDIA hardware, model architects should push group sizes higher, select head dimensions of 128 or 256, reduce effective KV state, and keep tensor parallelism less than or equal to the number of KV heads.


### [Introducing Supabase Evals](https://yomu.fyi/post/introducing-supabase-evals.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: matt\_rossman
- Published: Jul 31, 2026
- Type: Announcement

Supabase has open sourced supabase/evals, a benchmark and framework for testing how AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real tasks such as schema creation, Edge Function debugging, and RLS policy repair, scoring behavior with deterministic checks and LLM-as-a-judge. The framework executes scenarios in hosted-like and local containerized environments through the actual MCP server and CLI, with separate breadth-focused benchmark and depth-focused regression suites, plus a retry after failure. Initial findings show many agents pass broad tasks without skills, while skills improve some scores and documentation use; agents still often choose hand-written migrations, miss newer libraries, and use documentation inconsistently across harnesses.


### [Autoscaling endpoints for LLM inference](https://yomu.fyi/post/autoscaling-endpoints-for-llm-inference.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 31, 2026
- Type: Explainer

Dedicated LLM inference deployments face a trade-off between paying for idle GPUs and suffering sharp p95 latency when replicas hit batching limits. The autoscaling loop converts observed load into desired replicas with ceil(N × observed/target), then applies scale-up and scale-down windows, replica bounds, and GPU placement. It presents inflight\_requests as a leading, engine-agnostic default, while TTFT and end-to-end latency target responsiveness but trail demand, and GPU or token utilization prioritize efficiency with less latency headroom. Cold starts can take several minutes because replicas require GPU placement, weight downloads, VRAM loading, and warm-up, so policies should scale early; streaming is required for TTFT, decoding speed, and throughput-per-replica metrics, whereas end-to-end latency supports both streaming and non-streaming workloads.


### [Four Ways to Deploy More Secure AI Agents](https://yomu.fyi/post/four-ways-to-deploy-more-secure-ai-agents.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Jul 30, 2026
- Type: Problem & solution

Connecting large language models to live tools and enterprise systems introduces severe security vulnerabilities across multiple deployment harnesses. Over six months of evaluations, the NVIDIA AI Red Team identified recurring failures in AI agents, including missing access controls, arbitrary code execution via shell or file-write tools, unrestricted network egress, and exposed plaintext credentials. Mitigations operating in the same control plane as the model, such as system prompts or LLM-as-a-judge review agents, consistently fail against adversarial manipulation. Securing agentic deployments instead requires deterministic, architectural controls enforced outside the model control plane. Effective defenses include authenticating users, isolating command execution inside hardened sandboxes, blocking network egress by default, and delivering ephemeral secrets via external managers.


### [GenRec: Towards LLM-Native Recommendation at Netflix](https://yomu.fyi/post/genrec-towards-llm-native-recommendation-at-netflix.md)
- Company: [Netflix](https://yomu.fyi/company/netflix.md)
- Author: Netflix Technology Blog
- Published: Jul 30, 2026
- Type: Problem & solution

Netflix developed GenRec, an LLM-backed recommendation ranker built by adapting an internal foundation model for large-scale personalization. Traditional recommendation stacks rely on thousands of hand-crafted features and specialized architectures that are costly to maintain across diverse content types. To replace manual feature pipelines, GenRec verbalizes user histories, metadata, and contexts into natural-language prompts and trains with multi-objective losses, including catalog-aware ranking and reward-weighted alignment. At inference time, the model executes in prefill-only mode on vLLM without decoding text. In large-scale online A/B testing against a mature production ranker, GenRec achieved statistically significant improvements in short-term and long-term metrics while using fewer labeled examples.


### [How Salesforce Built an Agentic Engineering Enablement Strategy for Thousands of Software Engineers](https://yomu.fyi/post/how-salesforce-built-an-agentic-engineering-enablement-strategy-for-th.md)
- Company: [Salesforce](https://yomu.fyi/company/salesforce.md)
- Author: Scott Nyberg
- Published: Jul 30, 2026
- Type: Problem & solution

Salesforce addressed the challenge of scaling agentic engineering across thousands of software developers by focusing on organizational learning rather than tooling. While individual engineers initially developed disparate, uncoordinated workflows, the Technology, People, Innovation, and Learning (TPIL) team unified these approaches into the Proficiency Level (PL) Framework. The framework maps developer growth across four mindset-driven stages: AI-Assisted, AI-Validating, AI-Orchestrating, and AI-Native. To facilitate progress across these stages, TPIL introduced global AI camps, weekly working sessions, and manager coaching guides. Rather than tracking course completions or telemetry scores, Salesforce measured transformation through behavioral indicators, such as shifts toward validating agent output, designing multi-agent workflows, and asking higher-level questions regarding human judgment.


### [The Future of Agentic AI Depends on Openness and Trust. That’s Why Docker Is Joining Nvidia’s Open Secure AI Alliance.](https://yomu.fyi/post/the-future-of-agentic-ai-depends-on-openness-and-trust-that-s-why-dock.md)
- Company: [Docker](https://yomu.fyi/company/docker.md)
- Author: Jin Kim
- Published: Jul 30, 2026
- Type: Announcement

Modern discussions surrounding agentic AI increasingly center on whether autonomous systems can be trusted and securely placed at the core of enterprise workflows. While intelligence is generated by underlying models, operational trust depends on the surrounding execution runtime, identity frameworks, governance models, and security boundaries. To address these challenges across an open ecosystem, Docker has joined NVIDIA's Open Secure AI Alliance. This initiative aims to ensure developers can seamlessly route tasks between open-weight and frontier models without rewriting their application architectures or sacrificing control. By extending security and governance into the agent execution environment, the alliance seeks to deliver predictable control as AI agents become fundamental components of software stacks.


### [Agentic media buying cannot scale without the right foundation. See how buyers and sellers get there on Databricks.](https://yomu.fyi/post/agentic-media-buying-cannot-scale-without-the-right-foundation-see-how.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Joe Hu, Mandy Baker, Luke Barnes
- Published: Jul 30, 2026
- Type: Explainer

Media buying remains slowed by fragmented coordination across emails, spreadsheets, PDFs, and phone calls, leaving teams to research inventory, compare pricing, negotiate, and issue orders manually. The post presents an agentic workflow in which buyer and seller agents discover inventory, negotiate prices, and book deals through IAB Tech Lab’s AAMP standards, including AdCOM, OpenDirect, OpenRTB deals, and registry-based discovery and trust. Its Databricks implementation runs self-contained applications on Databricks Apps, with CrewAI agent crews using Databricks Foundation Model APIs and Claude models, Lakebase providing transactional Postgres state, and MLflow tracing capturing decisions and tool calls. The example processes a $200,000 Q3 Brand Launch across CTV and Linear TV with a $38 CPM ceiling, and is packaged as a Databricks Automation Bundle accelerator deployable with a single command.


### [Your trusted knowledge layer: Introducing Stack Internal's new platform experience](https://yomu.fyi/post/your-trusted-knowledge-layer-introducing-stack-internal-s-new-platform.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Phoebe Sajor, Caroline Thomas
- Published: Jul 30, 2026
- Type: Announcement

Organizations face growing challenges managing fragmented institutional data as AI agents and human workers operate across disparate tools. When agents consume stale or conflicting information, they produce confident errors that require costly human verification. Stack Internal has introduced an AI-native knowledge platform that ingests raw data across sources—including Google Docs, Slack, and Stack Internal communities—and evaluates it for trust, freshness, and authority. The updated platform adds a chat interface featuring response-level confidence labels and provenance cards, along with API and MCP server integrations that enforce identity-aware permission boundaries for AI agents. Engineering leaders can also track internal knowledge usage and high-traffic dependencies across versions via new Swagger API v3 endpoints.


### [Meet Stripe's Knowledge AI Platform](https://yomu.fyi/post/meet-stripe-s-knowledge-ai-platform.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Anna Mason
- Published: Jul 30, 2026
- Type: Explainer

Stripe built Kai, a knowledge AI platform for non-engineering work that existing no-code and coding agents could not support with Stripe’s required data security and workflow specificity. It serves tasks including account research, incident triage, revenue modeling, compliance reviews, and data-warehouse queries, while adapting to different tools, data sources, outputs, and definitions of completion. Kai is organized into surface-agnostic APIs, AgentStudio for domain teams to build and govern agents, and shared execution environments with per-session Kubernetes sandboxes, multi-tenant virtual filesystems, access controls, and long-session state management. The platform connects to more than 1,000 skills and tools and supports web, Slack, embedded, and custom application experiences. Within two weeks of its April launch, most of Stripe was using Kai; it now has 83% weekly active users, and reported effects include 25,000 annual hours shifted from administrative work and higher sales activity and deal creation among Account Executives.


### [How to Self-Host a Validated AI Coding Assistant with NVIDIA NeMo Guardrails](https://yomu.fyi/post/how-to-self-host-a-validated-ai-coding-assistant-with-nvidia-nemo-guar.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Jul 29, 2026
- Type: Tutorial

Deploying AI coding assistants in regulated or sovereign environments presents risks around data privacy, hallucinated dependencies, and auditability. This tutorial outlines a self-hosted architecture deploying StarCoder2-7B as an NVIDIA NIM endpoint served entirely from local GPUs. NVIDIA NeMo Guardrails sits as an OpenAI-compatible proxy between developer IDEs and the model to intercept and reject requests for restricted or human-only code paths. Downstream CI gates catch package hallucinations and verify changes, while commit trailers and a Prometheus and Grafana metrics loop monitor defect escape rates. Because the model remains distinct from the external policy and validation layers, teams can incrementally adopt components or upgrade models without rearchitecting the pipeline.


### [A new benchmark for evaluating patient-facing health AI agents](https://yomu.fyi/post/a-new-benchmark-for-evaluating-patient-facing-health-ai-agents.md)
- Company: amazon.science
- Author: Korosh Vatanparvar, Ashutosh Joshi
- Published: Jul 29, 2026
- Type: Benchmark

Healthcare AI benchmarks often rely on static medical knowledge questions or clinician-facing tool evaluations, failing to test multiturn patient interactions and clinical workflow execution. To address this gap, PatientAgentBench introduces an automated evaluation pipeline using synthetic patient charts, dual-agent conversations, and an LLM-as-a-jury panel. The evaluation framework applies over one hundred clinician-vetted criteria across six dimensions, including clinical safety, triage quality, and workflow accuracy. Testing frontier models revealed a severity paradox where agents scored higher on obvious emergencies than on routine administrative requests harboring hidden risks. Common safety failures included omitting crisis hotline resources and fabricating clinical information, showing that raw foundation model capability alone cannot guarantee safe patient-facing execution.


[Newer posts](https://yomu.fyi/topic/llm/page/12.md) · [Older posts](https://yomu.fyi/topic/llm/page/14.md)