# Smol2Operator: Post-Training GUI Agents for Computer Use

[Hugging Face](https://yomu.fyi/company/hugging-face) · Amir Mahla, merve, Sergio Paniego, Vaibhav Srivastav, Lewis Tunstall · Sep 23, 2025

**Type:** Problem & solution

## Summary

Lightweight vision-language models often lack the precise visual grounding and action execution capabilities required for graphical user interface automation. To resolve this deficiency, the authors constructed a post-training pipeline that evolves base models into agentic GUI coders through supervised fine-tuning. The process standardizes disparate mobile and desktop dataset action spaces into a unified functional format utilizing normalized coordinates relative to image size. Training is structured into two sequential phases that first establish basic visual perception and subsequently introduce explicit reasoning steps with masked assistant loss. On the ScreenSpot-v2 perception benchmark, fine-tuning SmolVLM2-2.2B-Instruct increased element localization accuracy to 61.71 percent, while a compact 460-million-parameter model achieved approximately 58 percent accuracy.

## Context

Lightweight vision-language models typically lack grounding capabilities for graphical user interface (GUI) automation. Furthermore, existing GUI automation datasets suffer from fragmented, inconsistent action spaces with disparate function signatures, parameter formats, and coordinate representations.

## Approach / What changed

The authors built a data transformation pipeline that standardizes heterogeneous desktop and mobile actions into a unified format with normalized coordinates between 0 and 1. They applied a two-phase supervised fine-tuning strategy using TRL on SmolVLM2-2.2B-Instruct, progressing from perception to agentic reasoning with masked loss on non-assistant tokens, and open-sourced all code, datasets, and models.

## Takeaways

- Using normalized coordinates scaled between 0 and 1 prevents action coordinates from breaking when vision-language models resize input screenshots during processing.
- Adding explicit reasoning steps in Phase 2 fine-tuning raised SmolVLM2-2.2B-Instruct ScreenSpot-v2 accuracy from 41% to 61.71%.
- The two-phase fine-tuning methodology scaled down effectively to the nanoVLM-460M model, which attained approximately 58% accuracy on ScreenSpot-v2.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source)

[Read original post](https://huggingface.co/blog/smol2operator)