---
title: "LLMs"
description: "673 posts about LLMs, summarised, each linking to the original."
---

# LLMs
> 673 posts about LLMs, summarised, each linking to the original.

## Articles

### [2,000 robots walk into a shop: Simulated A/B testing (2026) - Shopify](https://yomu.fyi/post/2-000-robots-walk-into-a-shop-simulated-a-b-testing-2026-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 27, 2026
- Type: Benchmark

Shopify’s SimGym addresses the difficulty of evaluating storefront changes when merchants lack enough traffic for conventional A/B tests, by simulating shopping sessions with LLM-guided robots. Each robot runs in a cloud Chromium browser with a persona, budget, and intent, while a counterpart tests the alternate theme; the system uses real page representations rather than a mocked DOM. To support bursty, sequential workloads with 89K–127K accumulated tokens per session, Shopify self-hosted gpt-oss-120b on 48 NVIDIA B200 GPUs, combining MXFP4 expert quantization, FP8 KV-cache, FlashInfer kernels, async scheduling, and guided decoding. Production measurements report 80K to 88K tokens/second per B200 from async scheduling, a 57% reduction in HTTP/gRPC overhead, and MIG experiments yielding about 20% lower latency and 12% higher daily throughput without quality regression; speculative decoding is validated but queued.


### [Using LLMs to amplify human labeling and improve Dash search relevance](https://yomu.fyi/post/using-llms-to-amplify-human-labeling-and-improve-dash-search-relevance.md)
- Company: [Dropbox](https://yomu.fyi/company/dropbox.md)
- Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra,Eric Wang,Dmitriy Meyerzon,Dmitriy Meyerzon
- Published: Feb 26, 2026

Dropbox Dash uses large language models (LLMs) to amplify human labeling efforts for training its search relevance and ranking models. By validating and optimizing LLM evaluators against a small set of human-labeled internal data, Dropbox creates massive, high-quality training datasets offline for production rankers like XGBoost without incurring high latency or latency costs at query time.


### [Mixture of Experts (MoEs) in Transformers](https://yomu.fyi/post/mixture-of-experts-moes-in-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Aritra Roy Gosthipaty, Pedro Cuenca, merve, Ilyas Moutawwakil, Arthur Zucker, Sergio Paniego, Pablo Montalvo
- Published: Feb 26, 2026
- Type: Explainer

Mixture of Experts (MoEs) in Transformers let a Transformer retain its backbone while replacing selected feed-forward layers with routed expert sub-networks, allowing total model capacity to exceed the active parameters used per token. The post explains why sparse models can reduce inference cost and memory pressure, then focuses on making them first-class in Transformers. Its weight-loading refactor converts independently serialized expert tensors into packed runtime layouts through composable WeightConverter operations and lazy materialization. The Experts Backend supplies eager, batched\_mm, and grouped\_mm execution strategies, while expert parallelism shards experts across devices and aggregates partial outputs. Training support uses grouped GEMM and custom Triton plus LoRA kernels, with reported improvements of approximately 12× speed, over 35% VRAM reduction, and roughly 6× longer context.


### [Innovating with MongoDB | Customer Successes, February 2026](https://yomu.fyi/post/innovating-with-mongodb-customer-successes-february-2026.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Katie Palmer
- Published: Feb 25, 2026
- Type: Problem & solution

Modern artificial intelligence startups frequently face operational bottlenecks when deploying autonomous agents and dynamic interfaces on rigid relational schemas or restricted database engines. To address these limitations, platforms including Modelence, Thesys, Emergent Labs, and Heidi integrated MongoDB Atlas as their operational data foundation. Emergent Labs eliminated recursive schema migration failures previously encountered with PostgreSQL by adopting document structures that match native JSON generation, supporting millions of deployed applications. Meanwhile, Heidi migrated from Amazon DocumentDB to utilize MongoDB Vector Search for retrieval-augmented generation pipelines under a single unified API without requiring external search databases. These architectural transitions streamlined continuous agent iterations, eliminated operational scaling downtime, and accelerated deployment cycles across diverse production environments.


### [Detecting and preventing distillation attacks](https://yomu.fyi/post/detecting-and-preventing-distillation-attacks.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: Feb 23, 2026
- Type: Explainer

Anthropic reports identifying industrial-scale distillation campaigns by DeepSeek, Moonshot, and MiniMax that generated more than 16 million Claude exchanges through approximately 24,000 fraudulent accounts. The campaigns used proxy services and coordinated account networks to extract agentic reasoning, tool use, coding, computer-use, and other capabilities, including chain-of-thought data and reinforcement-learning tasks. Anthropic attributed the operations with high confidence using IP correlation, request metadata, infrastructure indicators, and partner corroboration, while describing hydra cluster architectures that replace banned accounts and obscure patterns. The response combines classifiers, behavioral fingerprinting, coordinated-activity detection, intelligence sharing, stronger account verification, and product, API, and model-level countermeasures. The report argues illicitly distilled models may strip safeguards and undermine export controls, and says coordinated action across AI labs, cloud providers, and policymakers is required.


### [GGML and llama.cpp join HF to ensure the long-term progress of Local AI](https://yomu.fyi/post/ggml-and-llama-cpp-join-hf-to-ensure-the-long-term-progress-of-local-a.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Georgi Gerganov, Xuan-Son Nguyen, Aleksander Grygier, Lysandre, Victor Mustar, Julien Chaumond
- Published: Feb 20, 2026
- Type: Announcement

Hugging Face announces that GGML, the creators of llama.cpp, are joining the company to support the ggml and llama.cpp community as local AI advances. Georgi Gerganov and his team will continue dedicating 100% of their time to maintaining llama.cpp, retaining autonomy and leadership over its technical direction and community, while the project remains fully open-source and community driven. The technical focus is tighter integration between llama.cpp’s local-inference stack and Transformers’ model definitions, aiming to make shipping new models from Transformers’ “source of truth” nearly single-click, alongside better packaging and user experience for ggml-based software. The longer-term goal is an efficient inference stack that runs on devices and helps make open-source superintelligence accessible, with llama.cpp becoming ubiquitous and easier for casual users to deploy and access.


### [Train AI models with Unsloth and Hugging Face Jobs for FREE](https://yomu.fyi/post/train-ai-models-with-unsloth-and-hugging-face-jobs-for-free.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: ben burtenshaw, Daniel (Unsloth), Michael Han, Maxime Labonne, Daniel van Strien, shaun smith
- Published: Feb 20, 2026
- Type: Tutorial

The post presents a workflow for fine-tuning the LiquidAI/LFM2.5-1.2B-Instruct small language model with Unsloth on Hugging Face Jobs, including optional use of Claude Code, Codex, or another coding agent. It explains that Unsloth is described as providing about 2x faster training and 60% lower VRAM use than standard methods, while the model runs under 1GB of memory and is optimized for on-device deployment. The tutorial covers claiming free Hugging Face Jobs credits through the Unsloth Jobs Explorers organization, installing the hf CLI or a model-training skill, and submitting a job with a dataset, GPU flavor, timeout, and output repository. An agent can generate a UV training script using 4-bit loading and LoRA, submit the managed cloud-GPU job, report Trackio monitoring details, and push the trained model to the Hugging Face Hub.


### [Scaling Localization with AI at Lyft](https://yomu.fyi/post/scaling-localization-with-ai-at-lyft.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Stefan Zier
- Published: Feb 19, 2026
- Type: Problem & solution

Lyft re-architected its batch localization pipeline to combine artificial intelligence with professional linguist oversight. The previous workflow relied exclusively on human translation, which resulted in multi-day turnarounds and linear cost scaling that hindered rapid market expansions. Under the new architecture, a dual-path system generates three candidate translations using a fast generation model, evaluates them against defined rubrics using a reasoning-focused model, and immediately releases approved drafts. Linguists review the drafts asynchronously in the Smartling translation management system to establish the final system of record. This pipeline reduced delivery latency from days to minutes while achieving a state where 95% of machine-generated translations require no significant changes during human review.


### [Minions: Stripe’s one-shot, end-to-end coding agents—Part 2](https://yomu.fyi/post/minions-stripe-s-one-shot-end-to-end-coding-agents-part-2.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Alistair Gray
- Published: Feb 19, 2026
- Type: Explainer

Stripe’s minions are an unattended agentic coding flow whose pull requests are human-reviewed but contain no human-written code; more than 1,300 merge weekly. The system runs agents in AWS EC2-based devboxes that are prewarmed to become usable within about 10 seconds, providing parallel, predictable, and isolated environments. Its custom harness uses code-defined blueprints that combine deterministic nodes, such as linting and pushing changes, with agent loops for tasks such as implementation and fixing CI failures. Minions receive automated feedback through local linting, one full CI iteration, and a possible second repair attempt, while devbox isolation and security controls limit destructive actions; Stripe presents the approach as an extension of long-running investments in human developer productivity.


### [IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST](https://yomu.fyi/post/ibm-and-uc-berkeley-diagnose-why-enterprise-agents-fail-using-it-bench.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Ayhan Sebin, Rohan Arora, Saurabh Jha
- Published: Feb 18, 2026
- Type: Benchmark

IBM Research and UC Berkeley analyze how agentic LLM systems fail in enterprise IT automation, including incident triage, logs and metrics queries, and Kubernetes actions across long-horizon tool loops. To move beyond benchmark success rates, the analysis applies MAST (Multi-Agent System Failure Taxonomy) to execution traces, converting failures into structured signatures across system design, inter-agent misalignment, and task verification. It covers 310 ITBench SRE traces from Gemini-3-Flash, Kimi-K2, and GPT-OSS-120B, whose mean recalls were 75.5%, 28.6%, and 12.4%, respectively. Gemini averaged 2.6 failure modes per failed trace, compared with 4.7 for Kimi and 5.3 for GPT-OSS-120B, while FM-3.3, Incorrect Verification, was identified as the strongest predictor across models; the analysis recommends external verification, explicit termination controls, and clarification branches.


### [CX Score: How we built a metric support leaders can defend](https://yomu.fyi/post/cx-score-how-we-built-a-metric-support-leaders-can-defend.md)
- Company: [Intercom](https://yomu.fyi/company/intercom.md)
- Author: Mark O'Donovan
- Published: Feb 13, 2026
- Type: Explainer

CX Score was built to give support leaders a defensible view of customer experience across conversations, addressing whether an automated metric can be trusted. The team grounded its criteria in how experienced support professionals assess strong, mixed, and poor interactions, then created a dataset of thousands of conversations reviewed by experts, with two reviewers where possible and disagreement resolution. It operates as a production system using specialized prompt engineering, support-focused model configuration, neutral defaults for ambiguous evidence, and multi-stage evaluation across LLM models. Precision, recall, F1, regression, and bias checks set a required F1 above 0.8, while field testing with real customers examined varied tickets, languages, agent types, and conversation lengths. The stated conclusion is that CX Score is designed to be inspected, explained, and defended rather than treated as a black-box output.


### [OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments](https://yomu.fyi/post/openenv-in-practice-evaluating-tool-using-agents-in-real-world-environ.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Christian Washington, Ankit Jasuja, Santosh Sah, Lewis Tunstall, ben burtenshaw
- Published: Feb 12, 2026
- Type: Explainer

OpenEnv is an open-source framework from Meta and Hugging Face for evaluating AI agents against real systems rather than simulations, with stateful environments that connect to tools and APIs. Turing's production-grade Calendar Gym applies this model to calendar operations, combining access control lists, partial visibility, temporal reasoning, and multi-step workflows in isolated sessions. Agents often handled individual actions but became less reliable on longer or ambiguous tasks: success was close to 90% with explicit calendar identifiers and roughly 40% when requests used natural-language descriptions. More than half of failures involved malformed arguments or incorrect action ordering despite selecting the correct tool, while structured validation, permission, and datetime errors provided concrete recovery signals.


### [Minions: Stripe’s one-shot, end-to-end coding agents](https://yomu.fyi/post/minions-stripe-s-one-shot-end-to-end-coding-agents.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Alistair Gray
- Published: Feb 9, 2026
- Type: Explainer

Stripe’s Minions are homegrown, fully unattended coding agents designed to complete tasks in a single run, beginning with an invocation such as a Slack thread and ending with a pull request ready for human review. More than 1,000 pull requests merged weekly at Stripe are minion-produced and contain no human-written code, although engineers review them and can provide follow-up instructions. Each run uses an isolated, pre-warmed devbox and a customized fork of Block’s coding agent goose, combining agent loops with deterministic operations for version control, linting, and testing. Minions use curated MCP tools, including Stripe’s Toolshed with more than 400 tools, to retrieve internal context before and during work. Local checks and selective CI provide feedback, with no more than two CI rounds, balancing correction against token, compute, and time costs.


### [Introducing SyGra Studio](https://yomu.fyi/post/introducing-sygra-studio.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Surajit Dasgupta, Bidyapati Pradhan, Amit Kumar Saha, Vipul Mittal, Sriram Puttagunta
- Published: Feb 5, 2026
- Type: Announcement

SyGra 2.0.0 introduces Studio, an interactive environment for building, previewing, executing, and debugging synthetic data generation flows on a visual canvas. Users can configure connectors and models, preview source rows, reference automatically created state variables in prompts, define structured outputs, and inspect the generated YAML or JSON graph configuration. Studio streams node-level execution status, token usage, latency, cost, logs, and guardrail outcomes, while storing run history in .executions/runs/\*.json and autosaving drafts. It also runs existing workflows, including a Glaive Code Assistant example with conditional revision loops, tunable dataset and batching inputs, and observable intermediate critiques; generated outputs can support training, evaluation, or annotation workflows.


### [The Future of the Global Open-Source AI Ecosystem: From DeepSeek to AI+](https://yomu.fyi/post/the-future-of-the-global-open-source-ai-ecosystem-from-deepseek-to-ai.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Adina Yakefu, Irene Solaiman
- Published: Feb 3, 2026
- Type: Explainer

This third and final article in a series examines how Chinese AI organizations are shaping an open-source ecosystem after DeepSeek R1's January 2025 release. It argues that open sharing of models, papers, deployment infrastructure, and techniques has become the dominant near-term approach, with organizations ranging from Alibaba, Tencent, ByteDance, and Baidu to startups and application companies contributing at different layers. Qwen illustrates the ecosystem strategy: by mid-2025, more than 113,000 Hugging Face models used Qwen as a base, while organizations also connected models with chips, cloud platforms, toolchains, evaluation, and deployment systems. The article places this shift on infrastructure built since 2017, including the East Data, West Compute plan, and says R1 catalyzed wider use of existing compute, energy, and data capacity. It concludes that open-source models are becoming reusable, composable components in industrial workflows and localized AI systems as China moves from DeepSeek toward AI+.


### [We Got Claude to Build CUDA Kernels and teach open models!](https://yomu.fyi/post/we-got-claude-to-build-cuda-kernels-and-teach-open-models.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: ben burtenshaw, shaun smith, merve, Pedro Cuenca
- Published: Jan 28, 2026
- Type: Problem & solution

Domain-specific coding tasks like writing optimized CUDA kernels often require capable, expensive frontier models, leading to high operational costs. The upskill tool addresses this challenge by capturing specialized workflows from teacher models such as Claude Opus 4.5 into portable skill files that can be consumed by smaller or locally hosted models. The framework automatically generates input-output test cases from execution traces to benchmark student model accuracy and token efficiency with and without the skill. In an evaluation targeting HuggingFace kernel-builder development, a local GLM-4.7 Flash model improved its pass rate from 40% to 85% when supplied with the generated skill. Measuring token usage across different models remains essential, as skills can lower token consumption for some architectures while introducing unnecessary overhead for others.


### [Architectural Choices in China's Open-Source AI Ecosystem: Building Beyond DeepSeek](https://yomu.fyi/post/architectural-choices-in-china-s-open-source-ai-ecosystem-building-bey.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Adina Yakefu, Irene Solaiman
- Published: Jan 27, 2026
- Type: Explainer

Chinese open-source artificial intelligence development has shifted focus toward architectural choices, ecosystem toolchains, and domestic hardware integration following the January 2025 DeepSeek release. Leading groups have adopted Mixture-of-Experts architectures like Kimi K2 and Qwen3 to balance capability, dynamic compute allocation, and operating costs. Development expanded beyond text into multimodal domains including speech, video, and 3D systems while prioritizing smaller 0.5B to 30B distilled models under permissive Apache 2.0 licensing. Furthermore, model releases increasingly provide day-zero alignment with domestic inference and training infrastructure, including Huawei Ascend, Cambricon, and Baidu Kunlun chips. Production serving engines such as Moonshot AI's Mooncake and Baidu's FastDeploy 2.0 reflect a transition from standalone models to comprehensive system-level design.


### [Alyah ⭐️: Toward Robust Evaluation of Emirati Dialect Capabilities in Arabic LLMs](https://yomu.fyi/post/alyah-toward-robust-evaluation-of-emirati-dialect-capabilities-in-arab.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Omar saif alkaabi, Ahmed Alzubaidi, Hamza Alobeidli, Shaikha Alsuwaidi, Mohammed Alyafeai, Leen AlQadi, Basma Boussaha, Hakim Hacid
- Published: Jan 27, 2026
- Type: Benchmark

Standard Arabic language model evaluations predominantly assess Modern Standard Arabic while overlooking regional spoken dialects that feature distinct syntax, vocabulary, and cultural grounding. To address this evaluation gap, researchers created Alyah, a benchmark comprising 1,173 manually curated multiple-choice questions collected from native Emirati speakers across seven cultural and linguistic categories. Distractor answer choices were synthetically generated using language models and manually reviewed for semantic plausibility. Evaluating base and instruction-tuned Arabic-native, adapted, and multilingual models revealed that instruction-tuned systems consistently outperformed base variants, with falcon-h1-arabic-7b-instruct achieving the top instruction accuracy at 82.18%. However, questions involving implicit meanings, rare expressions, and specific dialect phenomena remained difficult across all tested model families and sizes.


### [Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective](https://yomu.fyi/post/unlocking-agentic-rl-training-for-gpt-oss-a-practical-retrospective.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Jason Zhu, Hejian Sang, Arup De, Rohit Jain, Yanning Chen
- Published: Jan 27, 2026
- Type: Problem & solution

Engineers evaluated the GPT-OSS model for agentic reinforcement learning across multi-step tasks such as ReTool and GSM8K using the verl framework. Initial training runs suffered from exploding gradient norms, increasing KL divergence, and stagnating rewards. Investigation revealed that non-deterministic expert routing in the Mixture of Experts architecture caused log-probability mismatches between forward passes, which falsely triggered Proximal Policy Optimization clipping. The team resolved this by substituting detached current log-probabilities for old log-probabilities during on-policy updates. To address further training instabilities and memory bottlenecks, the implementation integrated an attention-sink-aware backward pass into FlashAttention v3 and applied sequence parallelism using all-to-all communication around attention layers.


### [AI in Investment Management: 2026 Outlook (Part II)](https://yomu.fyi/post/ai-in-investment-management-2026-outlook-part-ii.md)
- Company: [Two Sigma](https://yomu.fyi/company/two-sigma.md)
- Author: Emily Majewski
- Published: Jan 21, 2026
- Type: Explainer

Two Sigma leadership outlines the evolving role of artificial intelligence across quantitative investing workflows heading into 2026. The firm is embedding frontier large language models into internal systems, incident management, and feature generation pipelines to accelerate research tasks that previously took months into days. Technical focus across the broader field is shifting from raw parameter scaling toward efficiency optimizations, multimodal unified representations, and mechanistic interpretability circuits. In forecasting pipelines, rapid automated hypothesis generation introduces severe risks of overfitting and compromised backtesting, particularly when pre-trained models already contain historical regime knowledge prior to their cutoff dates. Consequently, engineering success requires strong institutional research discipline, production monitoring, and skepticism alongside the adoption of automated agentic tooling.


[Newer posts](https://yomu.fyi/topic/llm/page/25.md) · [Older posts](https://yomu.fyi/topic/llm/page/27.md)