REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs

Summary

REFACTOR-VLA addresses the weakness of monolithic vision-language-action (VLA) models, which generate raw commands or short action sequences and struggle with long-horizon tasks and interpretability. Its wake/sleep architecture uses a Behavioral-Equivalence Kernel (BEK) to cluster motor-program segments by outcomes from a learned latent world model, then produces typed lambda terms for a library-conditioned rectified-flow action decoder. Skills are admitted only when they satisfy both Minimum Description Length (MDL) and return-preservation gates, while training proceeds through world-model warmup, wake-phase policy optimization, and sleep-phase skill discovery. On the full LIBERO benchmark, expanding the world model from 188 million to 430 million parameters worsened results on all four suites, showing that scale alone was not beneficial. Adding supervised contrastive InfoNCE loss during warmup substantially improved sleep-phase clustering, with reported NMI scores of 0.462, 0.867, 0.915, and 0.754 for the Object, Spatial, Goal, and LIBERO-10 suites, respectively, under n = 3 multi-seeding.

Takeaways

1. 1

   Increasing the latent world model from 188 million to 430 million parameters worsened performance on all four LIBERO benchmark suites.

2. 2

   Adding supervised contrastive InfoNCE loss during Phase A improved the quality of skill clustering in the sleep phase.

3. 3

   Reported NMI under n = 3 multi-seeding was 0.462 ± 0.021 for Object, 0.867 ± 0.025 for Spatial, 0.915 ± 0.013 for Goal, and 0.754 ± 0.010 for LIBERO-10.