Improving Cursor Tab with online RL

Summary

Cursor Tab predicts developers’ next actions across their codebases and handles more than 400 million requests per day, creating a large stream of accepted and rejected suggestions for training. The post describes replacing a separate acceptance filter with policy-gradient reinforcement learning, allowing the model to learn when to suggest and when to remain silent while reusing its learned code representation. An illustrative reward assigns 0.75 to an accepted suggestion, -0.25 to a rejected one, and 0 to showing nothing, targeting suggestions whose estimated acceptance chance is at least 25%; the production reward also considers suggestion size and jumps to other code locations. Because policy-gradient updates require on-policy data, each new checkpoint is rolled out to users; the current deployment-and-collection cycle takes 1.5 to 2 hours, and the resulting model makes 21% fewer suggestions with a 28% higher accept rate.

Takeaways

1. 1

   The illustrative reward of 0.75 for acceptance and -0.25 for rejection makes showing a suggestion worthwhile when its estimated acceptance chance exceeds 25%.

2. 2

   Policy-gradient training requires samples from the policy currently being optimized, so Cursor deploys each updated checkpoint to users before collecting data for the next training step.

3. 3

   The new default Tab model makes 21% fewer suggestions than its predecessor while achieving a 28% higher accept rate for the suggestions it does make.