Gaia2 and ARE: Empowering the community to study agents

Summary

Existing AI agent evaluation environments are often tightly coupled to specific tasks and fail to model real-world challenges such as API failures, spontaneous events, and asynchronous conditions. To address this limitation, the Gaia2 benchmark and the Meta Agents Research Environments (ARE) framework introduce interactive read-and-write evaluation. Gaia2 incorporates 1,000 human-created scenarios spanning multi-step execution, cross-source search, ambiguity handling, adaptability, temporal reasoning, agent collaboration, and noise tolerance. Using a simulated smartphone interface equipped with 101 tools, evaluations showed that GPT-5 with high reasoning scored highest overall, while Kimi K2 was the leading open-source model. The results demonstrated that instruction following and search do not reliably predict performance on closer-to-real-world tasks.

Takeaways

1. 1

   Gaia2 shifts from read-only evaluation to interactive read-and-write tasks featuring noise tolerance, time sensitivity, ambiguity, and API failures.

2. 2

   Evaluating multiple frontier models showed that while basic execution and search are near solved, temporal reasoning remains the hardest split across all models.

3. 3

   ARE automatically records structured execution traces including tool calls, API responses, latency metrics, and model thoughts for offline JSON export.