Rewriting an image as SVG code costs GPT-5 15 points of accuracy

Today's vision-language models see an image as an array of pixels. But to actually understand a picture, they need to work with symbols rather than pixels — the way they work with code. Pixels are fine for recognition and poor for carrying an image into an LLM's context. And pixels do not reliably tell you how objects relate to one another, or how many of each thing is in the frame.

The researchers propose converting the image into SVG code, which carries more information and holds structured data about the scene. SVG code is compact, easy to read and easy to execute — and, most importantly, easy to verify by simply running it.

What the authors built

The authors introduce a visual coding benchmark called VCode. It measures how well a model generates SVG code from an image while preserving the symbolic structure of the original scene — a description of the objects and their relations, written as code. The task is not to copy the picture into SVG but to carry the symbolic structure across: the objects, their signatures, their relative positions and their ordering in depth.

The authors also propose CodeVQA, a method that scores visual coding by surrogate execution. The model generates SVG code from the image, but the question is then answered from the rendered SVG rather than from the original. If the answer matches the one given for the original, the model carried the scene's symbolic content across.

The authors run an agent called VCoder, which applies two key strategies on top of current vision-language models:

1. Reasoning with revision: the model generates SVG code, renders it, compares the render against the original and fixes what came out, step by step. This works around a certain "blindness" and gets the structure of the code as close to the original as possible.
2. Acting with visual tools: segmenters, detectors and OCR. You can plug in the Florence-2 object detector, which reports where the objects are and what categories they belong to. The SAM-2 segmenter returns object masks as polygons. OpenOCR detects text in the image and locates it. All of that context is then converted into SVG elements.

VCode pulls tasks from three domains: MM-Vet (general visual common sense), MMMU (image-based questions across disciplines) and CV-Bench (perception). That comes to 464 tasks, each a question to be answered from a picture.

Scoring runs through CodeVQA: the more correct answers, the better. The authors also report the SigLIP score — the cosine similarity between embeddings of the original image and the SVG render.

What came out of it

Models are still bad at generating SVG code, and worse at preserving its symbolic content. The authors use GPT-5 as the baseline: 72.3 on SigLIP and 46.8 overall on CodeVQA, below the 61.7 the same questions get when answered from the original image. There is room to grow.

VCoder built on Claude-4-Opus does best, finishing at 54. The gain is sharpest on MM-Vet (+16.7), with gains in the other domains as well — MMMU (+6.2) and CV-Bench (+11.4).

The ablation shows that every visual tool added contributes something. Object detection brings more semantics and more precise positions, masks preserve spatial layout, and OCR matters for anything that hinges on text.

Iterative revision matters too: the first pass almost always improves the result, and after that it depends on the model's "patience".

Another interesting piece is the length of the SVG code. Shorter turns out to be worse — presumably because it is poorer in symbolic objects. Systems that carry over more objects and more of their relations perform better. The final SVG code can run past 2,000 tokens.

Telling examples

On MM-Vet the hardest questions are the knowledge ones, which need extra context pulled in — culture, history — and none of it can be lost on the way into code.

On MMMU everything rests on carrying the logic of formulas, diagrams, labels and charts across correctly; OCR and faithful geometry are what save you here.

On CV-Bench the 3D cues sometimes decide it — depth and proximity. The symbolic layer has to capture depth ordering or relative distances.

Takeaways

SVG code is a working format for reasoning and for acting: code can be edited, executed, layered with tools, and talked through with the prompt across revisions.

The study shows a wide gap between writing text code and writing visual code. The researchers offer workable benchmarks for measuring it, and they confirm that combining revision with pluggable tools — segmenters and OCR — improves how much meaning survives the trip out of pixels. The next step is training specialized models for visual coding end to end.

Going from pixels to code is a step toward AI that does not merely "see" a picture but understands it as a system of meanings and relations between objects.

💾 Code