Loading post…
Hugging FaceFalconPerceptionAnnouncement
Falcon Perception
Summary
Falcon Perception is a 0.6B-parameter early-fusion Transformer for open-vocabulary grounding and segmentation from natural-language prompts. It processes image patches, text, and task tokens in one sequence, using hybrid attention so image tokens receive bidirectional context while text and task tokens decode causally. Its Chain-of-Perception interface predicts each instance as coordinate, size, and segmentation tokens, with specialized Fourier-feature and dot-product heads for dense outputs. The post also introduces PBench, which profiles attributes, OCR-guided identification, spatial understanding, relations, and crowded scenes. Falcon Perception reaches 68.0 Macro-F1 on SA-Co versus 62.3 for SAM 3, while presence calibration remains weaker at MCC 0.64 versus 0.82; the 0.3B Falcon OCR model scores 80.3 on olmOCR and 88.6 on OmniDocBench.
Context
Open-vocabulary perception systems often use modular pipelines with separate vision, fusion, decoding, matching, and post-processing components. The post identifies difficulty scaling these designs cleanly, attributing improvements to specific components, and avoiding accumulated complexity. It also states that existing referring-expression benchmarks can be saturated and may conflate failures across capabilities.
Approach / What changed
Falcon Perception uses one autoregressive Transformer over image patches, text, and task tokens with a hybrid attention mask. Chain-of-Perception predicts coordinate, size, and segmentation tokens in sequence; lightweight specialized heads decode continuous geometry and generate masks by dot product with upsampled image features. Training uses multi-teacher distillation from DINOv3 and SigLIP2, large-scale data with hard negatives, and a three-stage recipe. PBench provides capability-specific diagnostics, while FlexAttention and a paged inference engine support serving.
Takeaways
The hybrid attention mask lets image tokens attend bidirectionally to image tokens while text and task tokens attend causally to the visual prefix and preceding text.
Chain-of-Perception avoids token-by-token high-resolution mask generation by producing one segmentation embedding per instance and combining it with upsampled image features.
Falcon Perception’s SA-Co Macro-F1 is 68.0 versus 62.3 for SAM 3, but its presence-calibration MCC is 0.64 versus 0.82.
Source and freshness
- Source
- huggingface.co
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.