---
title: "REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs"
description: "REFACTOR-VLA addresses the weakness of monolithic vision-language-action (VLA) models, which generate raw commands or short action sequences and struggle with long-horizon tasks and interpretability. Its wake/sleep architecture uses a Behavioral-Equivalence Kernel (BEK) to cluster motor-program segments by outcomes from a learned latent world model, then produces typed lambda terms for a library-conditioned rectified-flow action decoder. Skills are admitted only when they satisfy both Minimum Description Length (MDL) and return-preservation gates, while training proceeds through world-model warmup, wake-phase policy optimization, and sleep-phase skill discovery. On the full LIBERO benchmark, expanding the world model from 188 million to 430 million parameters worsened results on all four suites, showing that scale alone was not beneficial. Adding supervised contrastive InfoNCE loss during warmup substantially improved sleep-phase clustering, with reported NMI scores of 0.462, 0.867, 0.915, and 0.754 for the Object, Spatial, Goal, and LIBERO-10 suites, respectively, under n = 3 multi-seeding."
---

# REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs

[Apple](https://yomu.fyi/company/apple) · Sep 2, 2026

**Type:** Benchmark

## Summary

REFACTOR-VLA addresses the weakness of monolithic vision-language-action (VLA) models, which generate raw commands or short action sequences and struggle with long-horizon tasks and interpretability. Its wake/sleep architecture uses a Behavioral-Equivalence Kernel (BEK) to cluster motor-program segments by outcomes from a learned latent world model, then produces typed lambda terms for a library-conditioned rectified-flow action decoder. Skills are admitted only when they satisfy both Minimum Description Length (MDL) and return-preservation gates, while training proceeds through world-model warmup, wake-phase policy optimization, and sleep-phase skill discovery. On the full LIBERO benchmark, expanding the world model from 188 million to 430 million parameters worsened results on all four suites, showing that scale alone was not beneficial. Adding supervised contrastive InfoNCE loss during warmup substantially improved sleep-phase clustering, with reported NMI scores of 0.462, 0.867, 0.915, and 0.754 for the Object, Spatial, Goal, and LIBERO-10 suites, respectively, under n = 3 multi-seeding.

## Context

Most current VLA models are monolithic, producing raw motor commands or short action sequences without reusable behavioral abstractions. The source says this leads to poor performance on long-horizon tasks and makes learned behavior difficult to interpret. Existing skill-discovery methods either cluster contrastive embeddings or use LLM judgments that are not calibrated to the robot's dynamics.

## Approach / What changed

REFACTOR-VLA uses a wake/sleep architecture. During sleep, a Behavioral-Equivalence Kernel clusters motor-program segments according to outcomes from rollouts in a learned latent world model, Mφ. During wake, typed lambda terms from a Hindley–Milner-inspired vocabulary condition a rectified-flow action decoder. Skills must pass both an MDL criterion and a return-preservation gate, with training divided into world-model warmup, policy optimization, and skill discovery.

## Takeaways

- Increasing the latent world model from 188 million to 430 million parameters worsened performance on all four LIBERO benchmark suites.
- Adding supervised contrastive InfoNCE loss during Phase A improved the quality of skill clustering in the sleep phase.
- Reported NMI under n = 3 multi-seeding was 0.462 ± 0.021 for Object, 0.867 ± 0.025 for Spatial, 0.915 ± 0.013 for Goal, and 0.754 ± 0.010 for LIBERO-10.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Model Evaluation](https://yomu.fyi/topic/model-evaluation)

- Source: [Apple](https://machinelearning.apple.com/research/refactor-vla-motor-programs)
- Source URL: https://machinelearning.apple.com/research/refactor-vla-motor-programs
- Ingested by Yomu: 2026-09-03T11:17:14.567Z

[Read original post](https://machinelearning.apple.com/research/refactor-vla-motor-programs)