RISED: Rubrics for Agentic Multi-Environment Selection and Self-Distillation

Summary

Training a single LLM agent jointly across diverse interactive environments is challenged by scalar rewards that provide limited information about cross-environment relationships and no within-group reward contrast when rewards are identical. RISED addresses this by using rubrics—textual descriptions of rollout behaviors—to guide both online data selection and policy supervision. An LLM judge tags each rollout with a predefined rubric vocabulary shared across environments, and these profiles select data aligned with the batch’s behavioral composition while limiting overlap with already-selected data. Positive rubrics supply token-level supervision via an on-policy self-distillation teacher, while negative rubrics steer subsequent rollouts away from recurring failure modes. Across model backbones, RISED achieves the highest mean pass rate across environments and ranks first or second in every individual environment.

Takeaways

1. 1

   Scalar rewards in multi-environment RL provide limited information about cross-environment relationships and no within-group reward contrast when rewards are identical, motivating richer textual feedback such as rubrics.

2. 2

   RISED uses an LLM judge to tag each rollout with a predefined rubric vocabulary shared across environments, and these profiles guide data selection that aligns with the batch’s behavioral composition while limiting overlap with already-selected data.

3. 3

   Across model backbones, RISED achieves the highest mean pass rate across environments and ranks first or second in every individual environment, demonstrating the effectiveness of rubric-based guidance.

FAQ