# Gaia2 and ARE: Empowering the community to study agents

[Hugging Face](https://yomu.fyi/company/hugging-face) · Clémentine Fourrier, Grégoire Mialon, Maxime Lecanu, Pierre Andrews, Adrien Carreira, frere thibaud, Avijit Ghosh, Romain Froger, Dheeraj Mekala, Caroline Pascal, Ulyana Piterbarg · Sep 22, 2025

**Type:** Benchmark

## Summary

Existing AI agent evaluation environments are often tightly coupled to specific tasks and fail to model real-world challenges such as API failures, spontaneous events, and asynchronous conditions. To address this limitation, the Gaia2 benchmark and the Meta Agents Research Environments (ARE) framework introduce interactive read-and-write evaluation. Gaia2 incorporates 1,000 human-created scenarios spanning multi-step execution, cross-source search, ambiguity handling, adaptability, temporal reasoning, agent collaboration, and noise tolerance. Using a simulated smartphone interface equipped with 101 tools, evaluations showed that GPT-5 with high reasoning scored highest overall, while Kimi K2 was the leading open-source model. The results demonstrated that instruction following and search do not reliably predict performance on closer-to-real-world tasks.

## Context

Existing agent evaluation environments are tightly coupled with specific tasks, lack real-world flexibility, and do not reflect open-world conditions like asynchronous chaos, spontaneous events, and failed page loads, while earlier GAIA benchmark tasks have become too easy for newer models.

## Approach / What changed

Introducing Gaia2, a read-and-write benchmark with 1,000 human-created scenarios across seven task groups, paired with the open-source Meta Agents Research Environments (ARE) framework that simulates a smartphone environment with 101 tools, persona histories, and JSON trace recording.

## Takeaways

- Gaia2 shifts from read-only evaluation to interactive read-and-write tasks featuring noise tolerance, time sensitivity, ambiguity, and API failures.
- Evaluating multiple frontier models showed that while basic execution and search are near solved, temporal reasoning remains the hardest split across all models.
- ARE automatically records structured execution traces including tool calls, API responses, latency metrics, and model thoughts for offline JSON export.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://huggingface.co/blog/gaia2)