---
title: "Falcon Perception"
description: "Falcon Perception is a 0.6B-parameter early-fusion Transformer for open-vocabulary grounding and segmentation from natural-language prompts. It processes image patches, text, and task tokens in one sequence, using hybrid attention so image tokens receive bidirectional context while text and task tokens decode causally. Its Chain-of-Perception interface predicts each instance as coordinate, size, and segmentation tokens, with specialized Fourier-feature and dot-product heads for dense outputs. The post also introduces PBench, which profiles attributes, OCR-guided identification, spatial understanding, relations, and crowded scenes. Falcon Perception reaches 68.0 Macro-F1 on SA-Co versus 62.3 for SAM 3, while presence calibration remains weaker at MCC 0.64 versus 0.82; the 0.3B Falcon OCR model scores 80.3 on olmOCR and 88.6 on OmniDocBench."
---

# Falcon Perception

[Hugging Face](https://yomu.fyi/company/hugging-face) · FalconPerception · Apr 1, 2026

**Type:** Announcement

## Summary

Falcon Perception is a 0.6B-parameter early-fusion Transformer for open-vocabulary grounding and segmentation from natural-language prompts. It processes image patches, text, and task tokens in one sequence, using hybrid attention so image tokens receive bidirectional context while text and task tokens decode causally. Its Chain-of-Perception interface predicts each instance as coordinate, size, and segmentation tokens, with specialized Fourier-feature and dot-product heads for dense outputs. The post also introduces PBench, which profiles attributes, OCR-guided identification, spatial understanding, relations, and crowded scenes. Falcon Perception reaches 68.0 Macro-F1 on SA-Co versus 62.3 for SAM 3, while presence calibration remains weaker at MCC 0.64 versus 0.82; the 0.3B Falcon OCR model scores 80.3 on olmOCR and 88.6 on OmniDocBench.

## Context

Open-vocabulary perception systems often use modular pipelines with separate vision, fusion, decoding, matching, and post-processing components. The post identifies difficulty scaling these designs cleanly, attributing improvements to specific components, and avoiding accumulated complexity. It also states that existing referring-expression benchmarks can be saturated and may conflate failures across capabilities.

## Approach / What changed

Falcon Perception uses one autoregressive Transformer over image patches, text, and task tokens with a hybrid attention mask. Chain-of-Perception predicts coordinate, size, and segmentation tokens in sequence; lightweight specialized heads decode continuous geometry and generate masks by dot product with upsampled image features. Training uses multi-teacher distillation from DINOv3 and SigLIP2, large-scale data with hard negatives, and a three-stage recipe. PBench provides capability-specific diagnostics, while FlexAttention and a paged inference engine support serving.

## Takeaways

- The hybrid attention mask lets image tokens attend bidirectionally to image tokens while text and task tokens attend causally to the visual prefix and preceding text.
- Chain-of-Perception avoids token-by-token high-resolution mask generation by producing one segmentation embedding per instance and combining it with upsampled image features.
- Falcon Perception’s SA-Co Macro-F1 is 68.0 versus 62.3 for SAM 3, but its presence-calibration MCC is 0.64 versus 0.82.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance)

- Source: [Hugging Face](https://huggingface.co/blog/tiiuae/falcon-perception)
- Source URL: https://huggingface.co/blog/tiiuae/falcon-perception
- Ingested by Yomu: 2026-08-29T01:47:16.667Z

[Read original post](https://huggingface.co/blog/tiiuae/falcon-perception)