JanusCoder learns to see the interface its own code renders

Scientific plots, interactive interfaces, animated walkthroughs of theorems — all of it is, at bottom, code rendered as something you look at. Yet today's AI systems work in the text modality alone. They have no notion of what the code will look like on screen, or how it will behave inside a running application.

The researchers propose a single multimodal code model, JanusCoder. It accepts text, an image, or a combination of the two, and returns code that, when executed, reproduces the visual — the picture it was asked to recreate, the interactive elements of a web page, or an animation.

Why this is hard

The main obstacle to systems of this kind is the amount of multimodal data available for programming interfaces: you need not only the instruction and the code, but also the image the code produces when it runs, or even a screenshot of the web page. Data like that is hard and expensive to synthesize and verify. So the field builds a separate dataset for each narrow task — plot generation, visual generation for web interfaces, animation — and transfers little knowledge between domains along the way. The pipeline proposed here rests on a mix of strategies for selecting and improving data, automatic code execution, and judgements from several large vision-language models.

What the work builds

The researchers present a pipeline made up of the following steps:

• Guided evolution - Takes data in "instruction-code" form and derives sharper variants of the task and its solution, verified in the execution environment.
• Recontextualization - Rewrites the instruction so that it captures the intent and the edge cases of the given code more fully.
• Reverse instruction - Uses code fragments in different languages to generate plausible instructions.
• Bidirectional translation - Ports a solution between two adjacent programming domains — Manim ↔ Mathematica, for instance — to close the data gap in thinly covered areas.

Every new version is checked for executability, after which reward models select the variant with the highest score. The reward model uses a vision model to compare the image against the task, and a text model to grade the quality of the code and how well it matches the instruction.

What's in the data

The authors assembled JanusCode-800K, the largest dataset of its kind to date, holding "correct" examples of visual programming. It is already balanced between text-centric tasks, where the instruction and the code carry the weight, and visual-centric tasks, which additionally use a rendered image or a screenshot. Inside: Python visualization and plot edits, web pages and web-page editing tasks, SVG, tasks covering a range of algorithms, scientific tasks, Manim animations and Wolfram Mathematica projects. The authors also stress the value of carrying data across domains — other visualization languages such as R and Matlab, and HTML and SVG for interactive scientific artifacts.

The resulting models

On top of the dataset the authors trained two models: JanusCoder, for text-only code generation, and the multimodal JanusCoderV, which understands images. Both are positioned as a single interface for visual programming: they take a textual or visual request, write or edit code, and check the visual result by executing it. Different backbone architectures are used depending on model size; the authors show that the benefit of their data carries over across architectures and scales.

On PandasPlotBench, JanusCoder-14B produces invalid code 9.7% of the time, comparable to GPT-4o. On ArtifactsBench, which covers interactive artifacts, JanusCoder beats GPT-4o. On DTVBench the models come close to proprietary ones on the quality of animations and mathematical plots.

The multimodal JanusCoderV clearly outperforms open counterparts on ChartMimic, WebCode2M and DesignBench, and on ChartMimic it even beats GPT-4o. The model holds on to its code generation ability on BigCodeBench and LiveCodeBench while staying strong on visualization — a rare combination.

What this means for us

So JanusCoder and JanusCoderV open a new stage in programming — one where AI does not merely write code, but sees and understands how that code looks on screen, turning syntax into something closer to visual intelligence.

The authors are pushing their own user-facing interface, which matters for analysts, for web developers, and for anyone building scientific and educational visualizations. It is a step toward systems where an LLM drives the execution environment and takes its feedback from the screen. The authors are releasing the code and the models, along with a new benchmark, DTVBench.

💾 Code