Smol2Operator: Post-Training GUI Agents for Computer Use

Summary

Lightweight vision-language models often lack the precise visual grounding and action execution capabilities required for graphical user interface automation. To resolve this deficiency, the authors constructed a post-training pipeline that evolves base models into agentic GUI coders through supervised fine-tuning. The process standardizes disparate mobile and desktop dataset action spaces into a unified functional format utilizing normalized coordinates relative to image size. Training is structured into two sequential phases that first establish basic visual perception and subsequently introduce explicit reasoning steps with masked assistant loss. On the ScreenSpot-v2 perception benchmark, fine-tuning SmolVLM2-2.2B-Instruct increased element localization accuracy to 61.71 percent, while a compact 460-million-parameter model achieved approximately 58 percent accuracy.

Takeaways

1. 1

   Using normalized coordinates scaled between 0 and 1 prevents action coordinates from breaking when vision-language models resize input screenshots during processing.

2. 2

   Adding explicit reasoning steps in Phase 2 fine-tuning raised SmolVLM2-2.2B-Instruct ScreenSpot-v2 accuracy from 41% to 61.71%.

3. 3

   The two-phase fine-tuning methodology scaled down effectively to the nanoVLM-460M model, which attained approximately 58% accuracy on ScreenSpot-v2.