Smol2Operator: Post-Training GUI Agents for Computer Use

Summary

Lightweight vision-language models often lack the precise visual grounding and action execution capabilities required for graphical user interface automation. To resolve this deficiency, the authors constructed a post-training pipeline that evolves base models into agentic GUI coders through supervised fine-tuning. The process standardizes disparate mobile and desktop dataset action spaces into a unified functional format utilizing normalized coordinates relative to image size. Training is structured into two sequential phases that first establish basic visual perception and subsequently introduce explicit reasoning steps with masked assistant loss. On the ScreenSpot-v2 perception benchmark, fine-tuning SmolVLM2-2.2B-Instruct increased element localization accuracy to 61.71 percent, while a compact 460-million-parameter model achieved approximately 58 percent accuracy.

Context

Lightweight vision-language models typically lack grounding capabilities for graphical user interface (GUI) automation. Furthermore, existing GUI automation datasets suffer from fragmented, inconsistent action spaces with disparate function signatures, parameter formats, and coordinate representations.

Approach / What changed

The authors built a data transformation pipeline that standardizes heterogeneous desktop and mobile actions into a unified format with normalized coordinates between 0 and 1. They applied a two-phase supervised fine-tuning strategy using TRL on SmolVLM2-2.2B-Instruct, progressing from perception to agentic reasoning with masked loss on non-assistant tokens, and open-sourced all code, datasets, and models.

Takeaways

• Using normalized coordinates scaled between 0 and 1 prevents action coordinates from breaking when vision-language models resize input screenshots during processing.
• Adding explicit reasoning steps in Phase 2 fine-tuning raised SmolVLM2-2.2B-Instruct ScreenSpot-v2 accuracy from 41% to 61.71%.
• The two-phase fine-tuning methodology scaled down effectively to the nanoVLM-460M model, which attained approximately 58% accuracy on ScreenSpot-v2.

Related reading

• huggingface.co · Dec 8, 2021

  Training CodeParrot 🦜 from Scratch

  Training code generation models from scratch requires careful data filtering, tokenization, and compute management. To build CodeParrot, a 1.5-billion-parameter Python generation model, engineers extracted 20 million Python files from BigQuery and cleaned the resulting 180 GB corpus down to 50 GB after discovering extreme duplication. A custom GPT-2 tokenizer was trained over streamed samples, and the GPT-2 large architecture was initialized with layer-scaled, full-precision attention mechanisms. The training pipeline leveraged Hugging Face Accelerate alongside gradient checkpointing and a streaming iterable dataset designed to yield fixed-length concatenated token sequences. CodeParrot was trained on roughly 25 to 30 billion tokens and evaluated on coding tasks using the pass@k metric.

  Leandro von Werra

• MongoDB · Sep 23, 2025

  Build AI Agents Worth Keeping: The Canvas Framework

  Enterprise AI agent initiatives frequently stall after pilot phases due to technology-first thinking, governance gaps, infrastructure complexity, and poor alignment with business needs. To bridge this divide, development teams are shifting away from data-first pipelines toward a product-first methodology structured as product, agent, data, and model. The Canvas framework provides a phased workflow moving from quick proof-of-concept validation to model orchestration and operational hardening. In Phase 4, developers focus on API management, external provider orchestration, cost optimization, and evaluation pipelines. Phase 5 adds necessary governance, compliance, user experience, and security layers required to transform working agent prototypes into sustainable production deployments.