SpyRL turns open-ended tasks into a spy hunt with a checkable reward

When a task has no right answer

Large language models have an old problem. They learn well wherever the answer can be checked exactly: math, coding problems, formal puzzles. The answer is either right or it isn't. The machine gets a clean signal and improves.

But the moment a task turns open-ended — write a story, summarize a document well, produce a coherent explanation — everything gets harder. Who decides that a text is “good”? A human? Another model? A list of criteria? Every one of those brings noise, bias and extra inference cost.

The authors propose an unexpected move: instead of measuring the quality of an open-ended answer directly, rebuild the task itself so that a verifiable signal appears inside it. Out of that idea came a new approach — RLSVR, reinforcement learning with self-verifiable rewards — and a concrete implementation, SpyRL. In effect, the model is trained by playing “spot the spy.”

The results suggest this is a way to carry verifiable-reward training over into open-ended tasks.

What RLSVR is

The ordinary verifiable-reward setup is simple: the model solves a task, a checker compares the result against the correct answer, the model gets a reward. That is convenient, cheap and it scales. But only where a “correct answer” exists at all.

RLSVR takes a different route. If a task has no built-in checker, one can be constructed by transforming the task. The idea comes from self-supervision: there are no labels there either, but researchers invent a proxy task in which the labels arise automatically out of the data itself.

Roughly the same thing happens here, except that instead of labels you get rewards.

The scheme looks like this:

🟠 Take the original open-ended task — summarizing a long text, say.

🟠 The environment hides part of the information in advance and records exactly what it hid.

🟠 The models perform the same task, but under slightly different conditions.

🟠 Then a further question comes up, one that can be answered exactly: who was working with incomplete information?

🟠 That answer is checked against the environment's record, which means the reward arrives without a human and without an external judge.

The crucial part: the proxy task has to be tied to real quality. If a bad text doesn't help expose the hidden information gap, the whole scheme falls apart. So the authors show how to make that link tight.

SpyRL: the spy game

The concrete implementation is called SpyRL. Here several copies of the model play a social deduction game.

One player is assigned the role of spy. The rest are “civilians.” The civilians get the full input. The spy gets a corrupted version: part of the text is hidden, the context trimmed or masked. After that, everyone performs the same target task: writing a summary, a story or a math solution.

Then comes the second stage: everyone looks at everyone else's answers and votes on which of them is the spy.

And this is where the verifiable reward appears.

The environment knows the spy's identity in advance. Which means you can count exactly:

🟣 Who correctly picked out the spy

🟣 Who drew the most suspicion

🟣 How well the spy “blended into the crowd”

🟣 How convincingly the civilians told a bad answer from a good one

For the detection stage the reward is blunt: name the spy and you get a point, miss and you get zero. For the generation stage the reward depends on the number of suspicion votes. If you're often taken for the spy, your answer probably looked worse than the others.

That is the central idea: judging the quality of a text turns into detecting a hidden information deficit.

Why it should work

The main question: is suspicion of being the spy really connected to text quality? Maybe the players are just guessing from surface cues.

The authors test exactly that.

They run 100 game episodes on creative writing and summarization tasks. For each answer they count how many suspicion votes it drew. Separately, they ask GPT-4o to rank the answers by quality, best to worst.

That comes close to a direct test of the method: if more votes go with lower quality, the internal reward really is hitting its target.

And that is what the chart shows. More suspicion, worse quality. Not perfectly linear, but a stable relationship. For work of this kind that counts for more than an elegant explanation: the internal mechanism doesn't merely exist, it agrees with an outside assessment.

Most open-ended tasks today are trained either on human preferences or on a judge model. And there is always the risk that what you're optimizing is not the quality of the text but the judge's taste or its weaknesses.

SpyRL offers a different source of signal: a game situation where the correct answer is built into the environment itself.

What the experiments covered

The authors tested the method on three kinds of task:

🟠 Summarization of long documents

🟠 Creative writing

🟠 Mathematical reasoning

In summarization and writing, the spy saw the text with fragments masked out. In math, part of the problem's context was taken away. Everyone then had to produce a result, and from those results work out who the spy was.

Training alternated between two modes:

🟣 First the task-performing stage improves

🟣 Then the spy-detection stage improves

🟣 Then the cycle repeats

This is an important detail. Update the “performer” and the “detector” at the same time and the system becomes too unstable: one module shifts the distribution of answers while the other is trying to adapt to it. The authors show that joint updating is noticeably worse than alternating.

Another technical detail is the correction for roles. The spy and the ordinary players are not on equal terms: the spy starts from a worse position. So raw rewards can't simply be added together. For that, the method carries a separate role-conditioned advantage estimate, so the model doesn't learn to punish itself merely for having been the spy.

What it did in practice

On open-ended tasks the effect is clear.

In summarization, SpyRL consistently beats the base models and two self-play baselines, R-Zero and Absolute Zero. On Qwen3-8B its average win rate over its own base version in A/B comparisons reaches 75.4%. For Qwen3-4B it is 73.9%.

In creative writing the picture is more interesting still. On Qwen3-8B the method takes about 77.3% of the wins against the base model on one set and 78.1% on another. And the advantage shows up not only in “overall quality” but in slipperier categories such as novelty and emotional impact.

The authors also checked human ratings. Ten graduate students compared anonymized model texts. SpyRL leads there too, on almost every dimension.

On math the method doesn't fall down either, even though ordinary verifiable rewards already exist there. SpyRL wins here as well. For Qwen3-4B the average gain across seven benchmarks was 8.97%, for Qwen3-8B 6.16%. The jumps are largest on harder exam-style problems such as AIME.

An approach invented for open-ended tasks helps even where verifiable answers were already available. Apparently what matters is not only that a reward exists, but how rich and competitive it is.

What the ablations showed

The authors ran a set of ablations, and they make clear what holds the method up.

🟠 Train only the task-performing stage, without improving the detector, and the gains plateau fast.

🟠 Train only the detector, and the target ability barely improves at all.

🟠 Remove the spy — that is, the information asymmetry — and quality again stops improving before long.

🟠 Remove the correction for spy versus ordinary player, and in places the results drop below the base model.

One more useful detail is group size. The best jump comes from going from 3 players to 5. After that the returns shrink.

That is a good practical finding: there's no need to keep inflating the multi-agent setup. It's enough to reach the level of difficulty at which answers are already hard to tell apart but the detector can still be trained stably.

Why this matters

If you train models on math, things are more or less clear: there's a checker, there's an answer, there's a reward. But most of the interesting product tasks live in another zone. Summarization, writing, research, drafting, long explanations — none of them has a one-line ground truth.

Today the industry leans mostly on three things in those places:

🟣 Human preferences

🟣 Reward models

🟣 Judge models

All three have their limits. Humans are expensive and slow. Reward models inherit the biases of their training data. Judge models cost money on every inference and often grade worse than you'd like.

RLSVR points to another route: rather than hunting for the perfect external judge, change the structure of the task itself so that the signal is born out of the environment. This doesn't remove every problem. The game has to be designed carefully, the players have to be watched in case they start keying on surface cues, and the game reward has to be checked against real quality. But the direction here is clear and practical.

The takeaway

Open-ended tasks can acquire a verifiable reward if the environment is rebuilt the right way. That is the main point of RLSVR.

SpyRL does it through a simple mechanic: one player sees a corrupted input, the others see the full one, and then everyone tries to work out the spy from the output. The spy's identity is known in advance, so the reward is exact. And the game itself pushes the model to write better, because a bad answer gives away the missing information.

Three conclusions follow from this.

🟠 Reinforcement learning doesn't have to be confined to math and coding problems.

🟠 Open-ended tasks can be trained without leaning constantly on an external judge.

🟠 Transforming the task becomes as important a tool as the model's architecture or the choice of optimization algorithm.

If you want to understand where the next generation of LLM self-training might go, this is the place to look: not only at smarter judges, but at smarter game environments where the right signal is born on its own.