DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

Summary

DACA-GRPO addresses two weaknesses in reinforcement learning for diffusion language models: existing methods do not assign credit across denoising steps, and mean-field likelihood estimates are biased and high-variance. It is a lightweight, plug-and-play enhancement for GRPO-style trainers that combines Denoising Progress Scores with Stratified Masking Likelihood. The progress scores derive per-token importance weights from intermediate predictions without an additional forward pass, while stratified masking partitions token positions so each token is predicted with most of the sequence as context. Applied to three GRPO base methods, the approach improves results across seven benchmarks covering mathematical reasoning, code generation, constraint satisfaction, and constrained generation. Reported gains reach 5.6 percentage points on math reasoning, 7.4 on code generation, 36.3 on constraint satisfaction, and 5.9 on JSON schema adherence.

Takeaways

1. 1

   Denoising Progress Scores provide per-token importance weights from intermediate predictions without requiring an additional forward pass.

2. 2

   Stratified Masking Likelihood partitions token positions into strata, giving each token most of the sequence as context and reducing mean-field likelihood bias.

3. 3

   Across seven benchmarks and three GRPO base methods, reported gains reach 36.3 percentage points for constraint satisfaction and 7.4 points for code generation.

FAQ