BigCodeArena: Judging code generations end to end with code executions

Summary

Assessing AI-generated code via static inspection is error-prone and fails to reflect whether code functions correctly across real-world environments. BigCodeArena addresses this evaluation bottleneck as a human-in-the-loop platform that executes model outputs in real time inside isolated sandbox environments across ten programming languages and eight execution frameworks. Users interact directly with live applications, refine implementations through multi-turn conversations, and submit pairwise preference votes to generate Elo ratings via the Bradley-Terry model. In five months of community testing covering over 14,000 conversations, proprietary models including o3-mini and o1-mini led overall Elo rankings, while open models like Qwen2.5 and Llama-3.3-70B trailed closed competitors. The project has open-sourced its full evaluation pipelines, conversation datasets, and related benchmarks to support community-driven code generation assessment.

Context

Evaluating code generation is challenging because mentally simulating and reading raw source code without execution is cognitively demanding and error-prone, particularly for complex UI applications or long programs. Furthermore, traditional benchmarks like HumanEval rely on predefined test cases that cover only a tiny fraction of real-world programming tasks.

Approach / What changed

BigCodeArena provides a human-in-the-loop evaluation platform where users submit coding tasks, run side-by-side code generations in isolated real-time sandboxes, interact with live application outputs, and vote on the better result. The system supports 10 programming languages and 8 runtime environments, enables multi-turn iterative testing, and computes Bradley-Terry Elo ratings with bootstrap confidence intervals from pairwise votes.

Takeaways

• BigCodeArena collected over 14,000 multi-turn sessions across 500+ users, with web design (36%), problem solving (23%), and game development (16%) representing the most common coding scenarios.
• Elo rating analysis of 4,731 filtered pairwise execution votes showed o3-mini and o1-mini consistently leading across all evaluation settings, followed closely by Claude-3.5-Sonnet.
• The platform supports execution across 10 languages and 8 environments, including React, Vue, Core Web, Streamlit, Gradio, PyGame, and Mermaid diagrams.

Related reading

• Hugging Face · Sep 22, 2025

  Gaia2 and ARE: Empowering the community to study agents

  Existing AI agent evaluation environments are often tightly coupled to specific tasks and fail to model real-world challenges such as API failures, spontaneous events, and asynchronous conditions. To address this limitation, the Gaia2 benchmark and the Meta Agents Research Environments (ARE) framework introduce interactive read-and-write evaluation. Gaia2 incorporates 1,000 human-created scenarios spanning multi-step execution, cross-source search, ambiguity handling, adaptability, temporal reasoning, agent collaboration, and noise tolerance. Using a simulated smartphone interface equipped with 101 tools, evaluations showed that GPT-5 with high reasoning scored highest overall, while Kimi K2 was the leading open-source model. The results demonstrated that instruction following and search do not reliably predict performance on closer-to-real-world tasks.

  Clémentine Fourrier, Grégoire Mialon, Maxime Lecanu, Pierre Andrews, Adrien Carreira, frere thibaud, Avijit Ghosh, Romain Froger, Dheeraj Mekala, Caroline Pascal, Ulyana Piterbarg

• Hugging Face