---
title: "Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation"
description: "Current active learning strategies for vision transformers in semantic segmentation require extensive pixel labeling, impractical when only one pixel per image is labeled. TEAL is the first pixel-level active learning framework built around ViT representations, using a frozen backbone and diversity-based selection directly on the native token lattice to avoid interpolation artifacts. It applies a two-level MaxHerding strategy over multi-layer token descriptors for representative candidate selection, followed by margin-based uncertainty refinement on a decoder grid. Across CamVid, Cityscapes, ADE20K, and Pascal Context, TEAL improves over the strongest baseline by up to +24.36 mIoU with a single labeled pixel per image and reaches 97% of fully supervised performance with just 10 pixels. The improvements persist when all baselines use the same DINOv3 backbone, showing that token-space selection with decoder-grid refinement makes ViT token geometry effective for low-budget active segmentation."
---

# Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation

[Apple](https://yomu.fyi/company/apple) · Oct 21, 2026

**Type:** Problem & solution

## Summary

Current active learning strategies for vision transformers in semantic segmentation require extensive pixel labeling, impractical when only one pixel per image is labeled. TEAL is the first pixel-level active learning framework built around ViT representations, using a frozen backbone and diversity-based selection directly on the native token lattice to avoid interpolation artifacts. It applies a two-level MaxHerding strategy over multi-layer token descriptors for representative candidate selection, followed by margin-based uncertainty refinement on a decoder grid. Across CamVid, Cityscapes, ADE20K, and Pascal Context, TEAL improves over the strongest baseline by up to +24.36 mIoU with a single labeled pixel per image and reaches 97% of fully supervised performance with just 10 pixels. The improvements persist when all baselines use the same DINOv3 backbone, showing that token-space selection with decoder-grid refinement makes ViT token geometry effective for low-budget active segmentation.

## Context

Research paper introducing TEAL, a token-space active learning framework for low-budget semantic segmentation using frozen ViT backbones.

## Approach / What changed

TEAL employs a frozen ViT backbone, diversity-based selection on the token lattice via two-level MaxHerding over multi-layer token descriptors, and margin-based uncertainty refinement on a decoder grid.

## Takeaways

- TEAL operates directly on the ViT token lattice, avoiding interpolation artifacts from converting token embeddings to dense pixel resolutions.
- With a single labeled pixel per image, TEAL improves over the strongest baseline by up to +24.36 mIoU on standard segmentation datasets.
- TEAL reaches 97% of fully supervised performance with just 10 labeled pixels per image, demonstrating extreme label efficiency.

## FAQ

### How does TEAL select informative pixels?

TEAL uses a two-level MaxHerding strategy over multi-layer token descriptors for diversity-based selection, followed by margin-based uncertainty refinement on a finer decoder grid.

### What datasets were used to evaluate TEAL?

TEAL was evaluated on CamVid, Cityscapes, ADE20K, and Pascal Context, consistently outperforming baselines under extreme label scarcity.

### Does TEAL require a fine-tuned ViT backbone?

No, TEAL uses a frozen ViT backbone, avoiding the need for fine-tuning the encoder while still achieving strong performance.

**Tags:** [Active Learning](https://yomu.fyi/topic/active-learning), [Semantic Segmentation](https://yomu.fyi/topic/semantic-segmentation)

- Source: [Apple](https://machinelearning.apple.com/research/pixels-tokens-active-learning)
- Source URL: https://machinelearning.apple.com/research/pixels-tokens-active-learning
- Ingested by Yomu: 2026-11-11T00:00:15.805Z

[Read original post](https://machinelearning.apple.com/research/pixels-tokens-active-learning)
