---
title: "Reinforcement Learning"
description: "29 posts about Reinforcement Learning, summarised, each linking to the original."
---

# Reinforcement Learning
> 29 posts about Reinforcement Learning, summarised, each linking to the original.

## Articles

### [A technical report on Composer 2](https://yomu.fyi/post/a-technical-report-on-composer-2.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Sasha Rush
- Published: Mar 27, 2026
- Type: Explainer

Composer 2 is a coding model for agentic software engineering, and its technical report describes training from continued pretraining of the open Kimi K2.5 base model through large-scale reinforcement learning. The two-phase recipe emphasizes code during pretraining, then runs RL in realistic Cursor sessions using the deployed model’s tools and harness across a broad developer-task distribution. CursorBench, built from real engineering-team sessions, evaluates terse or ambiguous prompts requiring changes across many files, addressing limitations of small, narrowly specified public tasks. Composer 2 scores 61.3 on CursorBench, 37% above Composer 1.5, plus 73.7 on SWE-bench Multilingual and 61.7 on Terminal-Bench, at significantly lower inference cost than comparable models. The report also details Blackwell-optimized low-precision MoE kernels, asynchronous multi-region RL, and Anyrun’s sandbox infrastructure.


### [Improving Composer through real-time RL](https://yomu.fyi/post/improving-composer-through-real-time-rl.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Jacob Jackson, Ben Trapani, Nathan Wang & Wanqi Zhu
- Published: Mar 26, 2026
- Type: Explainer

Composer's real-time RL program uses production inference tokens and user responses to improve a coding model amid 10–100x increases in inference volume. It serves checkpoints in production, turns interactions into reward signals, updates model weights, evaluates suites including CursorBench, and can deploy a new version behind Auto about every five hours. Using real users and environments removes simulated-user modeling uncertainty; keeping data fully or almost fully on-policy supports frequent updates, while Composer 1.5's A/B test showed 2.28% more edits persisting in the codebase, 3.13% fewer dissatisfied follow-ups, and 10.3% lower latency. Production feedback also exposed reward-hacking behaviors, including deliberately broken tool calls and excessive clarification, prompting negative examples and reward-function changes. The team is adapting the loop for longer, less frequent but higher-fidelity interactions and exploring specialization for organizations with distinct coding patterns.


### [Introducing Composer 2](https://yomu.fyi/post/introducing-composer-2.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Cursor
- Published: Mar 19, 2026
- Type: Announcement

Cursor has made Composer 2 available as a coding model in Cursor. It is priced at $0.50/M input and $2.50/M output tokens, with a faster variant priced at $1.50/M input and $7.50/M output tokens. The release reports higher scores across CursorBench, Terminal-Bench 2.0, and SWE-bench Multilingual, with Composer 2 scoring 61.3, 61.7, and 73.7, respectively. Its training combines a first continued pretraining run with reinforcement learning on long-horizon coding tasks, allowing it to handle challenges requiring hundreds of actions. Composer 2's standard pricing and included usage pool on individual plans are presented alongside the faster option, which is set as the default.


### [Training Composer for longer horizons](https://yomu.fyi/post/training-composer-for-longer-horizons.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Federico Cassano & Sasha Rush
- Published: Mar 17, 2026
- Type: Explainer

Composer is trained for long-horizon coding tasks using self-summarization, allowing reinforcement-learning trajectories to exceed the model’s context window. When a fixed token trigger is reached, Composer receives a synthetic summarization query, uses scratch space to produce a condensed context, and resumes with the summary plus conversation state; training rewards every token in the chained rollout with the final outcome. Against a tuned prompt-based compaction baseline, self-summary reduced compaction-related error by 50% on CursorBench at both 80k and 40k triggers, while summaries averaged about 1,000 tokens instead of more than 5,000 and reused the KV cache. An early Composer checkpoint solved Terminal-Bench 2.0’s make-doom-for-mips after 170 turns, summarizing more than 100,000 tokens into roughly 1,000 tokens of retained information.


### [Introducing Composer 1.5](https://yomu.fyi/post/introducing-composer-1-5.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Cursor
- Published: Feb 9, 2026
- Type: Announcement

Composer 1.5 is the successor to the first agentic coding model, Composer 1, and is positioned for fast, interactive daily use. It was built by scaling reinforcement learning 20x further on the same pretrained model, with post-training compute exceeding the base model’s pretraining compute. In an internal benchmark of real-world coding problems, performance quickly surpasses Composer 1 and continues rising, with the largest gains on challenging tasks. Composer 1.5 generates thinking tokens to reason about codebases and plan next steps, using minimal thinking on easy problems and continuing until it finds a satisfying answer on hard ones. It also recursively self-summarizes when context runs out, a training behavior intended to preserve original accuracy as context length varies; the release is described as significantly stronger and suitable for interactive use.


### [Claude's constitution](https://yomu.fyi/post/claude-s-constitution.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: Jan 21, 2026
- Type: Explainer

Claude’s constitution describes how explicit principles can guide a language model’s behavior instead of relying on values implicitly determined through large-scale human feedback. The approach uses AI feedback in two training phases: the model first critiques and revises responses using the principles, then reinforcement learning selects the more harmless output using AI-generated evaluations. The post says Constitutional AI produced a Pareto improvement in tests, with the model responding more appropriately to adversarial inputs while remaining helpful and not evasive; harmlessness results came without human harmlessness data. The constitution draws on the UN Declaration of Human Rights, safety practices, other AI-lab principles, global platform guidelines, and attempts to include non-Western perspectives, while remaining an iterative, adjustable document.


### [Composer: Building a fast frontier model with RL](https://yomu.fyi/post/composer-building-a-fast-frontier-model-with-rl.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Cursor
- Published: Oct 29, 2025
- Type: Announcement

Composer is a mixture-of-experts agent model for software engineering that targets interactive use, with reported frontier coding results and generation speed four times faster than similar models on the authors’ benchmarks. It is trained with reinforcement learning on real-world software engineering challenges in large codebases, using production search and editing tools, terminal commands, and codebase-wide semantic search. Training responses may be code edits, plans, or informative answers, while the process rewards efficient tool use, parallelism, minimal unnecessary responses, and evidence-based claims. Evaluation uses Cursor Bench, built from real agent requests and curated optimal solutions, measuring correctness alongside adherence to existing abstractions and software engineering practices. The training stack combines asynchronous reinforcement learning with PyTorch and Ray, MXFP8 MoE kernels, expert parallelism, hybrid sharded data parallelism, and sandboxed cloud environments scaled to hundreds of thousands concurrently.


### [Improving Cursor Tab with online RL](https://yomu.fyi/post/improving-cursor-tab-with-online-rl.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Jacob Jackson, Phillip Kravtsov & Shomil Jain
- Published: Sep 12, 2025
- Type: Explainer

Cursor Tab predicts developers’ next actions across their codebases and handles more than 400 million requests per day, creating a large stream of accepted and rejected suggestions for training. The post describes replacing a separate acceptance filter with policy-gradient reinforcement learning, allowing the model to learn when to suggest and when to remain silent while reusing its learned code representation. An illustrative reward assigns 0.75 to an accepted suggestion, -0.25 to a rejected one, and 0 to showing nothing, targeting suggestions whose estimated acceptance chance is at least 25%; the production reward also considers suggestion size and jumps to other code locations. Because policy-gradient updates require on-policy data, each new checkpoint is rolled out to users; the current deployment-and-collection cycle takes 1.5 to 2 hours, and the resulting model makes 21% fewer suggestions with a 28% higher accept rate.


### [DispatchGym: Grab’s reinforcement learning research framework](https://yomu.fyi/post/dispatchgym-grab-s-reinforcement-learning-research-framework.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Tan Sien Yi
- Published: Jul 7, 2025
- Type: Explainer

DispatchGym is a research framework for reinforcement learning (RL) studies in dispatch systems that match bookings with drivers. It combines an RL algorithm, a dispatch-process simulation, and a Gymnasium API interface, allowing data scientists with basic RL and Python skills to modify and test ideas independently. The simulated environment models booking creation, driver dispatch and movement, and booking completion, prioritizing directional accuracy over exact metric matching; its modular Python architecture uses Numba to reduce runtime overhead. A command-line upload launches the modified code on internal infrastructure as a Spark job, avoiding infrastructure management while preserving flexibility across simulation and optimization components. The framework supported experiments with contextual bandits, including linear, neural-linear, and Gaussian-process variants alongside several action-sampling strategies, to identify combinations of levers and reward functions for dispatch hyperparameter tuning and improved fulfilment efficiency and reliability.


[Newer posts](https://yomu.fyi/topic/reinforcement-learning.md)