LLMs score 70+ on game code but under 25 on how the game looks

Making a game is more than getting code to run. It takes mechanics a player can grasp, art that looks decent, smooth animation and a steady 60 FPS. Large language models handle algorithmic problems confidently, but evaluations of their code rarely account for playability or aesthetics. The authors of V-GameGym set out to fill that gap: they assembled a realistic benchmark for visual game generation in Python with Pygame, and showed how to measure the quality of that work objectively.

What they built

V-GameGym is 2,219 carefully vetted game samples spread across 100 thematic clusters and drawn from 2,190 real repositories. Each sample pairs a natural-language requirement with reference code that is guaranteed to run and to produce visual artifacts: screenshots and video of a short demo. On top of that sits a multimodal evaluation that checks not only the code but what the player actually sees.

How the dataset was built

The source projects came from OpenCoder and The Stack v2, filtered down to those using Pygame. To avoid monotony, the authors described each program with feature vectors (structure, API calls, length, pattern frequencies) and clustered them with MiniBatchKMeans. From every cluster they took the structurally most complete sample. That bought both variety and a high run rate.

Then an automated LLM pipeline took over. It analyzed the intent of the original game, added autonomous behavior (a fixed-length demo with no keyboard input), verified that it ran in a sandbox, fixed failures from the logs, and wrote clear requirements in the voice of a product manager. Eight graduate students then reviewed the code and the visual output in a sandbox UI and signed off on the final set.

How models are scored

The system measures three modalities and averages them into a single score:

• code: functionality, quality and implementation technique;
• screenshots: interface completeness and visual readiness;
• video: animation, motion, gameplay coherence.

Two strong models do the automatic judging: Qwen3-Coder-480B-A35B-Instruct grades the code, Qwen2.5-VL-72B the visual artifacts. The result is a rating from Poor to Excellent and an honest indicator of playability rather than of syntax alone.

What the experiments showed

The authors tested 70 models, open and commercial. The picture is telling.

• Closed models lead: the best overall score is 45.0 (GPT-5). Among open models the top spots belong to the 400B+ giants, Qwen3-Coder-480B and DeepSeek-V3.
• The gap between code and picture is wide: code scores are often 70+, while screenshots and video land below 25. Models write working Pygame confidently, but are far weaker at conveying aesthetics and motion.
• Scale helps but does not settle it: the number of games solved grows logarithmically with parameter count, and architecture and training contribute noticeably.
• A long tail of difficulty: most tasks sit at Fair and Poor, and Excellent is rare. A game is the integration of logic, graphics and timing, and that is where models keep tripping.

Why it matters

V-GameGym pulls code-generation evaluation back toward reality. In games what counts is not only functions and classes but pace, responsiveness and how readable the screen is. The new benchmark holds models accountable for the whole experience, from project structure to the frames on screen.

That opens room for research in three directions:

• planning and a world model for the scene and its rules;
• graphics and animation consistent with the game logic;
• performance and frame stability in a constrained environment.

It also gives an honest measure of progress: a leaderboard that shows who manages dynamic scenes and who manages tidy code.

Where the limits are

The set is built around Pygame, so complex 3D worlds and specialized engines are out of scope for now. Autonomous demos are no substitute for a live player. LLM judging, reliable as it is, can carry biases; the authors push back on that with validation, retries and manual review.

Even so, the infrastructure holds up: 100% of the reference samples run, video covers every sample, and each game comes with ten screenshots.

The authors see promise in multimodal training, where a model plans the mechanics, draws the assets, sets the pacing and tunes FPS at once. Multi-agent systems in the roles of designer, programmer and tester would help, along with asset generation on the fly and learning from human play sessions. And, of course, extending the set beyond Pygame.

The bottom line

V-GameGym carefully closes the gap between code correctness and the quality of the play experience. We now have a large, varied and reproducible way to check whether LLMs can not only write programs but build games worth playing.