Gaia2 and ARE: Empowering the community to study agents

Summary

Existing AI agent evaluation environments are often tightly coupled to specific tasks and fail to model real-world challenges such as API failures, spontaneous events, and asynchronous conditions. To address this limitation, the Gaia2 benchmark and the Meta Agents Research Environments (ARE) framework introduce interactive read-and-write evaluation. Gaia2 incorporates 1,000 human-created scenarios spanning multi-step execution, cross-source search, ambiguity handling, adaptability, temporal reasoning, agent collaboration, and noise tolerance. Using a simulated smartphone interface equipped with 101 tools, evaluations showed that GPT-5 with high reasoning scored highest overall, while Kimi K2 was the leading open-source model. The results demonstrated that instruction following and search do not reliably predict performance on closer-to-real-world tasks.

Context

Existing agent evaluation environments are tightly coupled with specific tasks, lack real-world flexibility, and do not reflect open-world conditions like asynchronous chaos, spontaneous events, and failed page loads, while earlier GAIA benchmark tasks have become too easy for newer models.

Approach / What changed

Introducing Gaia2, a read-and-write benchmark with 1,000 human-created scenarios across seven task groups, paired with the open-source Meta Agents Research Environments (ARE) framework that simulates a smartphone environment with 101 tools, persona histories, and JSON trace recording.

Takeaways

• Gaia2 shifts from read-only evaluation to interactive read-and-write tasks featuring noise tolerance, time sensitivity, ambiguity, and API failures.
• Evaluating multiple frontier models showed that while basic execution and search are near solved, temporal reasoning remains the hardest split across all models.
• ARE automatically records structured execution traces including tool calls, API responses, latency metrics, and model thoughts for offline JSON export.

Related reading

• Hugging Face · Oct 7, 2025

  BigCodeArena: Judging code generations end to end with code executions

  Assessing AI-generated code via static inspection is error-prone and fails to reflect whether code functions correctly across real-world environments. BigCodeArena addresses this evaluation bottleneck as a human-in-the-loop platform that executes model outputs in real time inside isolated sandbox environments across ten programming languages and eight execution frameworks. Users interact directly with live applications, refine implementations through multi-turn conversations, and submit pairwise preference votes to generate Elo ratings via the Bradley-Terry model. In five months of community testing covering over 14,000 conversations, proprietary models including o3-mini and o1-mini led overall Elo rankings, while open models like Qwen2.5 and Llama-3.3-70B trailed closed competitors. The project has open-sourced its full evaluation pipelines, conversation datasets, and related benchmarks to support community-driven code generation assessment.

  Terry Yue Zhuo

• Hugging Face ·