# BigCodeArena: Judging code generations end to end with code executions

[Hugging Face](https://yomu.fyi/company/hugging-face) · Terry Yue Zhuo · Oct 7, 2025

**Type:** Announcement

## Summary

Assessing AI-generated code via static inspection is error-prone and fails to reflect whether code functions correctly across real-world environments. BigCodeArena addresses this evaluation bottleneck as a human-in-the-loop platform that executes model outputs in real time inside isolated sandbox environments across ten programming languages and eight execution frameworks. Users interact directly with live applications, refine implementations through multi-turn conversations, and submit pairwise preference votes to generate Elo ratings via the Bradley-Terry model. In five months of community testing covering over 14,000 conversations, proprietary models including o3-mini and o1-mini led overall Elo rankings, while open models like Qwen2.5 and Llama-3.3-70B trailed closed competitors. The project has open-sourced its full evaluation pipelines, conversation datasets, and related benchmarks to support community-driven code generation assessment.

## Context

Evaluating code generation is challenging because mentally simulating and reading raw source code without execution is cognitively demanding and error-prone, particularly for complex UI applications or long programs. Furthermore, traditional benchmarks like HumanEval rely on predefined test cases that cover only a tiny fraction of real-world programming tasks.

## Approach / What changed

BigCodeArena provides a human-in-the-loop evaluation platform where users submit coding tasks, run side-by-side code generations in isolated real-time sandboxes, interact with live application outputs, and vote on the better result. The system supports 10 programming languages and 8 runtime environments, enables multi-turn iterative testing, and computes Bradley-Terry Elo ratings with bootstrap confidence intervals from pairwise votes.

## Takeaways

- BigCodeArena collected over 14,000 multi-turn sessions across 500+ users, with web design (36%), problem solving (23%), and game development (16%) representing the most common coding scenarios.
- Elo rating analysis of 4,731 filtered pairwise execution votes showed o3-mini and o1-mini consistently leading across all evaluation settings, followed closely by Claude-3.5-Sonnet.
- The platform supports execution across 10 languages and 8 environments, including React, Vue, Core Web, Streamlit, Gradio, PyGame, and Mermaid diagrams.

**Tags:** [Developer Experience](https://yomu.fyi/topic/developer-experience), [LLMs](https://yomu.fyi/topic/llm), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://huggingface.co/blog/bigcode/arena)