PaperTalker builds a paper's explainer video and beats humans on informativeness

A short two-to-ten-minute explainer video has become close to mandatory for a paper: it goes on the project page, it gets shown at seminars, it gets forwarded to colleagues. But making one means hours of slide prep, recording a voice track and a talking head, then editing and revisions. And it is not the same problem as "natural" video generation: here you have to carry the paper's long context accurately, combine several channels at once — slides, speech, subtitles, cursor — and make the result clear and professional.

What the authors actually propose

Paper2Video takes on the whole problem. First, it gives the field its first open benchmark: 101 paper–author-video pairs with metadata, including the original slides and the presenter's face and voice. Second, it proposes a set of metrics that score not only the picture but knowledge transfer and memorability. And finally, it presents PaperTalker, a multi-agent system that assembles an academic talk end to end: it generates slides, writes subtitles, synchronizes speech, drives the cursor, and creates a talking head that imitates the author.

Why this is hard

A talk is a long multimodal object. You have to understand a paper full of figures and tables, compress it into 10–20 slides, lay it out so nothing spills over, and keep voice, subtitles and pointer in sync. Even strong video models still struggle with small on-screen text, long context and exact timing — which is why this calls for a modular architecture and metrics that mean something.

How PaperTalker works

The system is a set of agents, each responsible for one channel.

• Slides. Beamer LaTeX generated straight from the paper's text, then auto-debugged. The key idea is Tree Search Visual Choice: the system enumerates visual variants (figure scales, font sizes), renders the candidates, and uses a vision-language model to pick the one where nothing overflows and everything stays readable.
• Subtitles and attention cues. From the rasterized slides the system builds a short script: per-frame captions plus hints about where the cursor should point.
• Cursor. Positions on the slide are grounded through a GUI interaction model; WhisperX supplies exact word-level timings, so the pointer moves between phrases instead of twitching inside them.
• Speech and talking head. F5-TTS reproduces the author's voice, while models like Hallo2 and FantasyTalking produce realistic facial motion and upper body. One detail matters a lot here: generation runs in parallel across slides, which speeds the whole process up by more than six times.

How they measure quality

There are no standard metrics for videos like these, so the team proposed four:

• Meta Similarity — how close the slides and subtitles come to a human-made talk.
• PresentArena — a pairwise "which is better" comparison of two videos on clarity, structure and engagement.
• PresentQuiz — a test of knowledge transfer: the video should help a viewer answer questions about the paper correctly.
• IP Memory — how memorable the author and the work are after a short clip, which is closer to what an actual conference feels like.

What it looks like in practice

On the Paper2Video benchmark, PaperTalker beats the baselines consistently. It wins more often in PresentArena, posts the best Meta Similarity, and gets high speech-synthesis quality out of voice personalization. On the PresentQuiz informativeness test it even comes out 10% ahead of human videos: less filler, more facts. The cursor and the presenter help noticeably — they add accuracy and raise the win rate in pairwise comparisons. On IP Memory, PaperTalker is the best of the lot: face, voice and a deliberate structure together make the video stick.

And it does not burn unlimited resources doing it: slide-level parallelism yields more than a sixfold speedup in production at modest inference cost. Visually it is the closest thing to a human talk — no blurred text, a clean layout — unlike end-to-end video generation, which is usually capped at a few seconds and cannot hold small type.

A word on slide layout

Tree Search Visual Choice deserves its own mention. It is a small but very practical improvement: enumerate a few carefully chosen layouts and pick the best one automatically. That kills overflows, fixes captions, and makes the final frames far cleaner.

Why this matters for the community

The work closes three gaps at once: it supplies a reference dataset, proposes metrics with substance, and shows a system ready to be used. For researchers it is a new testbed, where you can measure not only how pretty the picture is but knowledge transfer and the memory effect. For practitioners it is a way to get a solid video for a paper quickly, without the misery of editing. And, valuably, the architecture is modular: every component can be improved on its own — from script generation to precise cursor grounding.

Limitations and what comes next

There is room to grow: presenter personalization depends on the quality of the source material, and pulling key figures and tables out of complicated LaTeX projects automatically still takes care. But the results already come close to author-made talks, and open code and data should let an ecosystem grow around the benchmark quickly.

💾 Code