Bringing Robotics AI to Embedded Platforms: Dataset Recording, VLA Fine‑Tuning, and On‑Device Optimizations

Summary

Deploying Vision–Language–Action policies on embedded robots is constrained by compute, memory, power, and real-time control requirements; synchronous inference can leave the arm idle and cause oscillatory corrections. NXP presents dataset-recording practices for a tea-bag placement task, including fixed cameras, controlled lighting, a gripper-mounted view, calibration backups, and diverse training and validation splits with recovery episodes. The example uses 120 episodes, three 640×480 cameras at 30 fps, batch size 8, and checkpoint selection based on training and validation success rather than loss alone. For deployment, the VLA is decomposed into vision, LLM backbone, and action-expert stages, with selective quantization and asynchronous scheduling constrained by inference time being shorter than action execution. On the i.MX 95, optimized ACT latency reached 0.32 seconds with 0.89 global accuracy, while ONNX FP32 ACT reached 0.96 global accuracy at 2.86 seconds.

Takeaways

1. 1

   A gripper-mounted camera provides a close, task-relevant viewpoint for fine manipulation and helps ensure operators rely only on inputs available to the policy at runtime.

2. 2

   For the tea-bag task, the dataset contained 120 episodes across 10 clusters, with cluster 6 withheld for validation and about 20% of training episodes representing recovery behavior.

3. 3

   On the i.MX 95, optimized ACT reduced inference latency from 2.86 seconds with ONNX FP32 to 0.32 seconds, while global accuracy changed from 0.96 to 0.89.