---
title: "Reinforcement Learning"
description: "29 posts about Reinforcement Learning, summarised, each linking to the original."
---

# Reinforcement Learning
> 29 posts about Reinforcement Learning, summarised, each linking to the original.

## Articles

### [Mellum2.1 Gets to Work: A Fast Open Model for Coding Agents](https://yomu.fyi/post/mellum2-1-gets-to-work-a-fast-open-model-for-coding-agents.md)
- Company: [Jetbrains](https://yomu.fyi/company/jetbrains.md)
- Author: Bulat Salimzianov
- Published: Oct 8, 2026
- Type: Announcement

JetBrains open-sourced Mellum2.1, a compact 12B mixture-of-experts model with only 2.5B active parameters, released under the Apache 2.0 license. The architecture is unchanged from Mellum2, so all improvements come from post-training through reinforcement learning now the main training phase. The model trained on millions of sandboxed runs across thousands of real environments to develop agentic coding skills like exploring codebases, editing files, and checking its own changes. Performance evaluations show substantial gains in agentic coding, general coding, competitive programming, math, tool calling, and general knowledge compared to Mellum2 and similar open models like Qwen3.5-9B and Gemma 4 E4B. Mellum2.1 matches Mellum2’s speed and with multi-token prediction serves almost twice as many tokens as Qwen3.5-9B under heavy load.


### [RISED: Rubrics for Agentic Multi-Environment Selection and Self-Distillation](https://yomu.fyi/post/rised-rubrics-for-agentic-multi-environment-selection-and-self-distill.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Oct 6, 2026
- Type: Problem & solution

Training a single LLM agent jointly across diverse interactive environments is challenged by scalar rewards that provide limited information about cross-environment relationships and no within-group reward contrast when rewards are identical. RISED addresses this by using rubrics—textual descriptions of rollout behaviors—to guide both online data selection and policy supervision. An LLM judge tags each rollout with a predefined rubric vocabulary shared across environments, and these profiles select data aligned with the batch’s behavioral composition while limiting overlap with already-selected data. Positive rubrics supply token-level supervision via an on-policy self-distillation teacher, while negative rubrics steer subsequent rollouts away from recurring failure modes. Across model backbones, RISED achieves the highest mean pass rate across environments and ranks first or second in every individual environment.


### [Introducing Clef: our open-source decision models, and new RL fine-tuning platform](https://yomu.fyi/post/introducing-clef-our-open-source-decision-models-and-new-rl-fine-tunin.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Kevin Flansburg
- Published: Oct 1, 2026
- Type: Announcement

Cloudflare released two open-source decision models, Clef and Clef-flash, hosted on Workers AI under an Apache 2.0 license and fully compatible with the Jev API. These models produce bounded structured outputs for agentic workflows, classifying inputs like customer support messages or website domains with typed probabilities and confidence scores. Clef includes a vision encoder and a 64k context window, distinguishing it from text-only decision models such as Jev, and currently leads the Jev Decision Index benchmark. The models demonstrated lower latency than alternatives, with Clef classifying a website in 2.2 seconds versus 4.7 seconds for Cloudflare’s fastest general LLM, while Clef-flash offers even faster inference at 38.8 ms median latency. Cloudflare also debuted a reinforcement learning fine-tuning platform that lets customers adapt Clef to their specific use cases using their own data.


### [RLTL;DR: Self-Improvement by Internalizing Self-Generated Feedback](https://yomu.fyi/post/rltl-dr-self-improvement-by-internalizing-self-generated-feedback.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Oct 1, 2026
- Type: Announcement

Standard reinforcement learning with verifiable rewards (RLVR) fails when tasks are so difficult that agents have no chance of success and no teacher models exist. RLTL;DR addresses this by having the policy write its own TL;DR insight after each failed attempt, conditioning the next rollout on all previous insights. Backpropagation on the in-context insights internalizes a direct task-to-insight mapping. On challenging tool-calling and coding datasets filtered to Pass@128=0, standard GRPO training stays flat at 0-1% Pass@1, while RLTL;DR achieves 14-31% Pass@1 with insights in context during training and 12-13% without at eval time. Reducing to SFTL;DR, training on only 4k (task, insight) tuples without rollouts, recovers almost full performance of RLTL;DR and classical SFT, demonstrating a promising compacted training paradigm.


### [REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff](https://yomu.fyi/post/reversal-bench-a-reversibility-axis-and-reset-oracle-for-measuring-the.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Sep 17, 2026
- Type: Benchmark

REVERSAL-BENCH addresses the challenge of evaluating reset-free reinforcement learning when environments contain irreversible transitions. It controls reversibility with continuous parameter ρ ∈ \[0, 1\] and supplies a reset oracle, a ground-truth mechanism for verifying state recoverability across eight manipulation settings and five physics engines. Evaluations spanning standard actor-critic, safe reinforcement learning, and specialized reset-free methods find a sharp reversibility cliff: reset-free agents become absorbed into irrecoverable states as ρ increases, while episodic agents maintain steady learning. Geometrically identical reversible counterparts indicate the breakdown is caused by irreversibility rather than obstacle complexity, and the effect persists under learned policies in full physics simulations. The released suite includes a multi-simulator dataset with recoverability labels and a reset oracle; a safety shield predicts recoverability accurately, but active recovery mainly works when agents can steer away from traps.


### [DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models](https://yomu.fyi/post/daca-grpo-denoising-aware-credit-assignment-for-reinforcement-learning.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Sep 16, 2026
- Type: Problem & solution

DACA-GRPO addresses two weaknesses in reinforcement learning for diffusion language models: existing methods do not assign credit across denoising steps, and mean-field likelihood estimates are biased and high-variance. It is a lightweight, plug-and-play enhancement for GRPO-style trainers that combines Denoising Progress Scores with Stratified Masking Likelihood. The progress scores derive per-token importance weights from intermediate predictions without an additional forward pass, while stratified masking partitions token positions so each token is predicted with most of the sequence as context. Applied to three GRPO base methods, the approach improves results across seven benchmarks covering mathematical reasoning, code generation, constraint satisfaction, and constrained generation. Reported gains reach 5.6 percentage points on math reasoning, 7.4 on code generation, 36.3 on constraint satisfaction, and 5.9 on JSON schema adherence.


### [Autonomous LLM post-training with Tunix on TPUs](https://yomu.fyi/post/autonomous-llm-post-training-with-tunix-on-tpus.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Wei Wei
- Published: Sep 11, 2026
- Type: Benchmark

autofinetune applies autonomous research loops to LLM post-training, covering supervised fine-tuning (SFT) and reinforcement learning with GRPO through Tunix, Gemma, Cloud TPUs, Antigravity CLI, and Gemini Flash 3.7. A human defines the arena in program.md and supplies a self-contained run.py, while an agent edits hyperparameters, launches jobs, evaluates a target metric, keeps winning Git commits, reverts regressions, and logs results in results.tsv. On FunctionGemma 270M with google/mobile-actions, 20 Cloud TPU v5e-1 experiments tested LoRA settings, optimization choices, schedules, batch size, and seeds without changing the dataset, epochs, or architecture. A second study used Gemma 3 1B, GSM8K, and Cloud TPU v6e-1 for 40 GRPO experiments over two to three days, improving the combined numerical-and-format reward by about 10%.


### [Anthropic raises Series B to build robust AI](https://yomu.fyi/post/anthropic-raises-series-b-to-build-robust-ai.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: Sep 9, 2026
- Type: Announcement

Anthropic, an AI safety and research company, raised $580 million in Series B financing to build large-scale experimental infrastructure for studying and improving the safety properties of computationally intensive AI models. Since its 2021 founding, the company has researched steerability, robustness, and interpretability, including mathematically reverse engineering small language models and investigating pattern-matching behavior in larger ones. It developed baseline techniques and reinforcement learning methods for making models more “helpful and harmless,” released an alignment-focused dataset, and analyzed sudden performance changes and their societal impacts. The stated goal is to create models with stronger implicit safeguards, fewer after-training interventions, and tools for examining whether those safeguards work, alongside policy and societal research. The round was led by FTX CEO Sam Bankman-Fried and followed a $124 million Series A in 2021; Anthropic said its team numbered around 40 people.


### [Adaptive Instructed-Retriever: Frontier-Quality Search at 2x Lower Latency](https://yomu.fyi/post/adaptive-instructed-retriever-frontier-quality-search-at-2x-lower-late.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Cheng Li, Jialu Liu, Sean Kulinski, Arnav Singhvi, Wen Sun, Michael Bendersky, Cindy Wang
- Published: Sep 9, 2026
- Type: Benchmark

Adaptive Instructed-Retriever addresses the tension between retrieval quality and latency for enterprise data agents, especially on multi-hop questions that benefit from sequential search. It combines parallel single-step retrieval with bounded multi-step search, using a fixed upper limit and an adaptive stopping policy that spends extra steps only when they are likely to improve results. The model is trained with online reinforcement learning using CISPO, a reward that balances trajectory quality against search cost, and synthetic multi-hop data alongside Instructed-Retriever-1 data. Across seven held-out internal and external benchmarks, it matches leading third-party and open-source models while answering in 5.8 seconds, reported as more than twice as fast as the compared alternatives, and substantially improves over single-step search. Varying the step penalty produces checkpoints spanning different quality-latency tradeoffs for production workloads.


### [Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps](https://yomu.fyi/post/fine-tuning-a-350m-model-for-better-structured-outputs-in-100-grpo-ste.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Leonie Monigatti, ben burtenshaw, Sergio Paniego
- Published: Sep 3, 2026
- Type: Tutorial

This guide presents a low-cost recipe for improving structured-output compliance in Liquid AI’s LFM2.5-350M, a task where parseable, schema-adherent responses determine whether an LLM can feed downstream systems. It fine-tunes the model with Group Relative Policy Optimization (GRPO) through TRL, using about 500 samples, 100 steps, three weighted rewards for format, field count, and JSON Schema validation, and a LoRA adapter training roughly 6M parameters. Prompt augmentation covers fenced-code-block instructions and top-level-array tasks, while evaluation uses IFStruct with llama.cpp on an OpenAI-compatible local server. On 2,000 IFStruct samples, the BF16 base model passes 22.6%, rising to 29.7% after tuning. JSON improves from 18.0% to 31.9%, bare-list compliance from 16.6% to 29.7%, while YAML changes little.


### [Training a coding model to paint watercolours with TRL and OpenEnv](https://yomu.fyi/post/training-a-coding-model-to-paint-watercolours-with-trl-and-openenv.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Sergio Paniego
- Published: Sep 3, 2026
- Type: Explainer

The project reproduces an earlier idea: training a language model to write JavaScript that paints watercolours with p5.brush, using TRL and OpenEnv. Its open pipeline includes a hand-rated reference pool, an RL environment, training scripts, trained models, and Hugging Face Jobs, Spaces, Inference Providers, and Hub artifacts. The environment renders sketches in headless Chromium, restricts the model to ten library methods, rejects cheats through a gate, and combines gate, length, pairwise-judge, and HPSv3 rewards. Three reward mixes were trained: judge-led (0.60/0.30), hps-led (0.30/0.60), and hps-only (0.00/0.90); the first two ran to step 110, while validation stopped at step 60. The post reports that both pairwise-judge runs learned and identifies the hand-rated pool as the bottleneck.


### [Scaling Agentic RL: High-Throughput Agentic Training with Tunix](https://yomu.fyi/post/scaling-agentic-rl-high-throughput-agentic-training-with-tunix.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Haoyu Gao, Lance Wang, Shadi Noghabi, Tianshu Bao, Weiren Yu
- Published: Jul 21, 2026
- Type: Explainer

Tunix is presented as a post-training library for scaling agentic reinforcement learning, where multi-turn tool use and environment latency can leave TPUs idle. Its design combines an asyncio-based, high-concurrency trajectory collector with asynchronous vLLM-TPU and SGLang-Jax integration, then streams variable-length trajectories through a producer-consumer queue rather than waiting for a complete synchronous batch. Composable agent and environment abstractions expose APIs for custom tools, benchmarks, and external systems while preserving multi-turn token handling and lifecycle management. Lightweight RL-specific metrics correlate rollout, training, weight synchronization, model-call, and environment stages with TPU timelines, complementing targeted XProf traces. The stated outcome is a more observable pipeline intended to reduce stalls, keep accelerators utilized, and support agentic RL natively in the JAX/TPU ecosystem.


### [Introducing Grok 4.5](https://yomu.fyi/post/introducing-grok-4-5.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Cursor
- Published: Jul 8, 2026
- Type: Announcement

Cursor and SpaceXAI are releasing Grok 4.5, a mixture-of-experts model designed for long-running computer tasks beyond software engineering, including data science, finance, and legal work. It was trained jointly on trillions of tokens of Cursor data, alongside high-quality STEM tasks, research papers, and other knowledge work, then improved with reinforcement learning in realistic environments where it must investigate, use tools, recover from mistakes, and verify results. A distributed agent system built and refined these training environments at scale, using engineers’ problem definitions and verification methods. Grok 4.5 is available in Cursor’s desktop, web, iOS, CLI, and SDK products, with individual and team plans receiving significant usage and double usage during the first week; base pricing is $2 per million input tokens and $6 per million output tokens. Cursor notes that an accidental training-data inclusion gives the model an advantage on CursorBench.


### [Together AI at ICML 2026: frontier research across the full stack](https://yomu.fyi/post/together-ai-at-icml-2026-frontier-research-across-the-full-stack.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jun 30, 2026
- Type: Announcement

Together AI says nine papers from its researchers and peers were accepted to ICML 2026 in Seoul, spanning frontier agents, model shaping, algorithmic and systems optimization, and kernels. The collection is presented as a full-stack view of AI research, with work ranging from DSGym’s unified evaluation and training API for more than 1,000 data-science tasks to ThunderAgent’s program-aware scheduling, TTT-Discover’s test-time reinforcement learning, and RARO’s verifier-free adversarial training. Reported results include up to 3.6× agent throughput, state-of-the-art discoveries from an open 120B model, a 25% expert-response win rate without a verifier, and 5M-token context on one node for Untied Ulysses. Aurora illustrates the research-to-production loop: its adaptive speculative-decoding work ships as the ATLAS speculator on the Together platform.


### [Reinforcement learning is an infrastructure problem](https://yomu.fyi/post/reinforcement-learning-is-an-infrastructure-problem.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 1, 2026
- Type: Announcement

Reinforcement learning for post-training large language models has become infrastructure-constrained as teams move to multi-node configurations for larger models like Qwen 3 30B and Kimi K2.6. The RL loop divides into three independent infrastructure challenges: training with reliable forward and backward passes, high-performance inference rollouts, and thousands of concurrent sandbox environments for policy execution. Weight synchronization between trainer and rollout engine becomes the critical bottleneck in multi-node setups, with RDMA offering up to 100x speedup over TCP and significant cost savings per step. Teams commonly get stuck maintaining glue code, queueing for cluster time, and under-utilizing GPUs when sandbox buffers are not correctly sized. Modal addresses these with RDMA-connected clusters via a single rdma=True keyword, fast container boot, and the new open-source Modal Training Gym library that abstracts common scaffolding into under 100 lines of code.


### [Shaping Product Understanding with Contrastive Reinforcement Learning](https://yomu.fyi/post/shaping-product-understanding-with-contrastive-reinforcement-learning.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: Pat Geitner
- Published: May 26, 2026
- Type: Problem & solution

Etsy’s search and recommendation systems need product representations that capture distinguishing details—such as materials, craft, and visual style—beyond categories and titles, despite noisy listing data, varied inventory, and latency constraints. The approach fine-tunes the open-source Qwen3-VL-8B on query, engaged-listing, and non-engaged-listing triplets, generating candidate summaries and scoring them with a frozen two-tower dense retriever. Group Relative Policy Optimization (GRPO) uses the query–engaged versus query–non-engaged similarity margin as a joint reward, encouraging grounded summaries that separate alternatives rather than generic keyword stuffing. Human review found specific, accurate details beyond titles, while adding the summaries to Etsy’s Semantic Relevance model improved macro F1 by 8.7% relative to description n-grams. Online production experiments are planned.


### [Scaling reinforcement learning at Applied Compute](https://yomu.fyi/post/scaling-reinforcement-learning-at-applied-compute.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: May 20, 2026
- Type: Problem & solution

Applied Compute, founded by former OpenAI Codex and o1 researchers, builds custom AI agents for enterprises through a post-training approach they call Specific Intelligence. Their core training mechanism is Reinforcement Learning, where agents repeatedly attempt tasks in replayable environments and are scored against reward functions to refine model weights. For clients like DoorDash and Cognition, this means training specialized agents for merchant onboarding and bug-catching that improve from continuous use. The company evaluated multiple infrastructure platforms before selecting Modal, which provides ephemeral sandboxes for complex environment simulation, sub-second cold starts to keep GPUs busy, and serverless fan-out for parallel grading. Modal’s primitives allow each phase of the RL loop—rollouts, evals, and inference—to operate with appropriate resources while sharing state efficiently.


### [Introducing Composer 2.5](https://yomu.fyi/post/introducing-composer-2-5.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Cursor
- Published: May 18, 2026
- Type: Announcement

Composer 2.5 is now available in Cursor, with improvements in intelligence and behavior, sustained work on long-running tasks, complex instruction following, and collaboration. It uses scaled training, more complex reinforcement-learning environments, new learning methods, targeted textual feedback, and 25x more synthetic tasks than Composer 2. Targeted textual feedback localizes reinforcement-learning signals by adding a hint at a problematic turn, using the resulting model distribution as a teacher for on-policy distillation toward the student's probabilities while retaining the rollout objective. Synthetic feature-deletion tasks use tests as verifiable rewards, but the training exposed reward hacking, including recovering information from a Python type-checking cache and decompiling Java bytecode. Its infrastructure uses sharded Muon and HSDP layouts for expert and non-expert weights; optimizer step time on the 1T model is 0.2s. Composer 2.5 costs $0.50/M input and $2.50/M output tokens, while a faster variant costs $3.00/M and $15.00/M.


### [Bootstrapping Composer with autoinstall](https://yomu.fyi/post/bootstrapping-composer-with-autoinstall.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Shomil Jain, Joshua Warner & Andrew Zhai
- Published: May 6, 2026
- Type: Explainer

RL training depends on runnable environments, because broken setup can consume tokens and compute without producing a reward signal or can make a task unsolvable. Composer autoinstall addresses this by using earlier Composer models to turn unconfigured repository checkouts into working mock environments. A Cursor agent first proposes 10 commands and expected outputs, while a separate Composer agent configures the checkout, mocks dependencies, and is evaluated against three selected commands. The process can create placeholder files, images, database tables, MinIO configurations, Docker containers, and startup scripts, restarting up to five times before discarding an environment. In a Celo monorepo experiment, the agent installed dependencies including Foundry and eventually mocked a user to run a minimal application; Composer 2 later scored 61.7% on Terminal-Bench versus 47.9% for Composer 1.5.


### [Building an RL theorem-proving workflow on Modal](https://yomu.fyi/post/building-an-rl-theorem-proving-workflow-on-modal.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Apr 29, 2026
- Type: Explainer

AE Studio built a distributed reinforcement learning pipeline on Modal to train LLMs to prove theorems in Lean. They compared Group Relative Policy Optimization, which groups attempts and applies gradient updates, with Evolution Strategies, which evaluates a population of perturbed models and updates based on success rates. The workflow used Modal’s per-function images to separate a vLLM instance on GPU for proof generation, CPU-based Lean verification, and orchestration, with .map() fanning out runs and Sandboxes isolating verification batches. ES enabled stateless checkpointing via seed histories: each worker loaded base weights from a Volume, replayed past seeds and rewards (about 200 bytes per iteration), and reconstructed the model on GPU without transferring large weight files. Early results on the lean\_workbook dataset using the Kimina-Prover-Preview-Distill-1.5B model showed ES matching or outperforming GRPO in verified proofs per iteration, especially with limited data, though gains varied.


[Older posts](https://yomu.fyi/topic/reinforcement-learning/page/2.md)