BigCodeArena scores coding models by running their code, not reading it

Judging code generation by how clean the comments look is like judging a car from the brochure. In real life what matters is whether it starts, whether it brakes in time, and whether it is pleasant to use. The authors of BigCodeArena take exactly that practical view: their open platform compares large language model (LLM) solutions not by string overlap but by whether they run, how they respond to interaction, and how they behave on screen. A user sees two anonymous contenders, runs both solutions in a sandbox, can click through the interface, enter data and debug, and only then votes for A, B, a tie, or "both are bad."

What this is for

Long code often looks right to the eye, then fails on launch, breaks at runtime, or misses the prompt entirely. The authors show empirically that execution and interactivity sharply improve the reliability of judgements — for humans and for LLM judges alike. The effect is clearest in UI tasks, games and web pages, where dynamics, event handling and small UX details decide the outcome.

How the platform is built

The user interface is built on Gradio; the server side is a modular E2B system with isolated sandboxes. It supports 10 languages (Python, JS/TS, HTML, C/C++, Java, Go, Rust, Markdown) and 8 environments (React, Vue, Core Web, Streamlit, PyGame, Gradio, Mermaid, Interpreter). The system automatically pulls code out of markdown blocks, installs dependencies, compiles or runs it, and returns every artifact: logs, screenshots, the interactive UI.

There is a paired mode, in the spirit of Chatbot Arena, and a one-sided mode for testing a single model. To rule out a speed bias, neither side's result is shown until generation and execution have finished on both. Model pairs are balanced by weighted sampling: newcomers get a temporary weight boost so they accumulate statistics faster.

What has been collected so far

Over 5+ months: 14,123 sessions from 500+ unique IPs. From those, 4,731 multi-turn paired examples with preference votes were extracted. Each of the 10 models has at least 700 votes. Topics recur across six areas: Web Design, Game Development, Diagram Creation, Creative Coding, Scientific Computing, Problem Solving. The data shows Python and interpreter scenarios dominating, but UI frameworks take a large share too.

How the models were compared

Votes are aggregated with a Bradley–Terry model, bootstrapped over 100 repeats with 95% confidence intervals. Three regimes were considered: all data; matching environment; matching language. The rankings agree across all three. In live voting, o3-mini and o1-mini lead consistently, followed by Claude-3.5-Sonnet. The middle tier is GPT-4o, o1, Gemini-2.0-Pro/Flash; weaker are Qwen2.5 and Llama-3.3-70B. By language and environment the picture is uneven: Gemini-2.0-Pro is strong in Rust, for instance, while o3-mini holds up in React, Streamlit, Gradio and Core Web.

On judging and reward models

BigCodeReward, a set for training and evaluating models on practical code, is built on 4.7K human preferences. It has three classes (A better, B better, tie) and two modes: without and with execution results (text, screenshots, interaction). The conclusion is blunt: access to execution usually makes judges more accurate. The largest gain is in diagrams and game development, the smallest in applied programming. Individual LLM judges are also unstable, which matters for the work ahead.

An automatic arena with no humans

AutoCodeArena is a fully automated ranking. It takes 600 representative prompts, runs the code in Docker, uses Claude-3.7-Sonnet as the judge and GPT-4.1 as the baseline model, and compares pairwise against that baseline with bootstrap intervals. Here the new leader is GPT-5; Claude-Opus-4 and Claude-Sonnet-4 perform strongly. Among open models, Kimi-K2, GLM-4.5 and Qwen3-Coder show notable progress.

Why this matters for the community

• Running the code is the key to an honest reading of correctness, robustness and fidelity to the prompt.
• The platform itself is open, with transparent infrastructure and data. You can check the methodology, reproduce the results, build your own metrics.
• The accumulated interaction traces (clicks, keyboard, scrolling) open a path to more agentic judges and to better models that account for how interfaces actually behave.

The authors clean the data and run everything in isolated sandboxes, which lowers the risk of PII leaks and malicious code. They are honest that the risks are not fully eliminated. Inference is offloaded to external endpoints, which reduces the direct energy footprint and simplifies transparency.

What comes next

The team plans to add languages and environments, build "live" versions of BigCodeReward and AutoCodeArena, develop multi-agent judges, account for user scenarios and improve reward models for code. From a practical standpoint the main lesson is already clear: without execution, any evaluation of code generation will limp. With it, the leaders become visible and progress becomes measurable.

💾 Code