Reinforcement Learning with Verifiable Rewards (RLVR) has become the engine behind modern reasoning models. The recipe is straightforward: let a model generate answers to math problems or coding challenges, then check those answers against a deterministic verifier. Correct answers earn rewards, wrong ones don’t. This simple loop has driven dramatic gains in mathematical reasoning, code generation, and logical problem-solving. But it hits a wall the moment you step outside domains where correctness can be deterministically checked.
How do you verify the quality of a creative story? A text summary? An open-ended essay? Traditionally, you bring in human annotators (expensive and slow), train a reward model to approximate preferences (introduces bias), or use an LLM as a judge (limited by the judge’s own capabilities). Each approach trades the clean, scalable supervision of RLVR for noisy, costly approximations. A recent paper from researchers at Duke University, Adobe, Oregon State University, Penn State, NUS, and Amazon, accepted at COLM 2026, proposes an elegant way around this bottleneck — and the mechanism is inspired by a party game.
The Self-Supervised Learning Insight
The paper, titled “From RLVR to RLSVR,” draws a direct analogy to self-supervised learning. In self-supervised learning, you don’t try to approximate missing labels — you transform the task so that labels emerge automatically from the data itself. Masked language modeling doesn’t try to predict semantic categories; it masks tokens and asks the model to fill them in. The “label” is the original token, available for free.
The authors apply the same principle to reinforcement learning. Their framework, called RLSVR (Reinforcement Learning with Self-Verifiable Rewards), transforms an open-ended task into a proxy environment where the reward signal is generated by the environment’s own internal rules — not by an external judge. The key insight: if you can design a task transformation where the ground truth is known by construction, you get deterministic, unlimited, and free supervision for any task.
SpyRL: A Social Deduction Game for Model Training
The authors instantiate RLSVR with a concrete method called SpyRL, inspired by the social deduction game “Who Is the Spy?” The setup works like this: take an open-ended task — say, summarizing a government report. Assign multiple agents to complete it, but give one agent (secretly designated as the “spy”) a degraded version of the input with 20% of the text masked. The other “civilian” agents get the full, unmodified input.
All agents produce their outputs. Then comes the detection stage: each agent examines everyone’s outputs and votes on who they think is the spy. Because the spy identity was predetermined by the environment, whether each vote is correct is trivially verifiable — it’s a deterministic check against a known ground truth.
The crucial link is that the spy, working with degraded information, tends to produce lower-quality outputs. When outputs are poor, other agents notice and vote the spy out. The paper validates this correlation empirically: outputs that receive more suspicion votes consistently rank lower in quality when evaluated independently by GPT-4o. This means the voting mechanism converts the otherwise unverifiable notion of “output quality” into a verifiable identity-recognition problem.
How the Game Drives Improvement
The reward design is zero-sum between the spy and civilians. The spy is penalized for getting caught — incentivizing it to produce the best possible output despite degraded information. Civilians are rewarded when the spy is caught but penalized if they themselves attract suspicion — incentivizing them to produce high-quality work. This creates a self-play loop where both sides continuously improve: performers learn to generate better outputs, while detectors learn to identify subtler quality differences.
The optimization alternates between training the performing policy and the detection policy. This prevents the premature convergence that plagues fixed self-play setups: when detectors get too good, training shifts to performers; when performers get too deceptive, training shifts back to detectors. The two stages chase each other around the model’s current capability frontier.
The framework uses GRPO-style group-relative advantage estimation, normalizing rewards within each group of agents rather than against a learned critic. With a default group size of five agents (four civilians, one spy), the method maintains stable learning signals without requiring a separate value network.
Results That Span Domains
The researchers evaluated SpyRL on three task domains using Qwen3-4B and Qwen3-8B as base models: text summarization (GovReport), creative writing (WritingPrompts), and mathematical reasoning (Nemotron-CC-Math-v1). The results are striking across all three.
On summarization, SpyRL achieved the highest ROUGE-L scores across five benchmarks for both model sizes. On Qwen3-8B, it achieved a 75.4% GPT-4o pairwise win rate against the untrained baseline — meaning an independent judge preferred SpyRL’s summaries three out of four times.
On creative writing, the gains were even more pronounced: 77.3% overall win rate for Qwen3-8B, with the largest margins in novelty and emotional depth — exactly the subjective dimensions that traditional RLVR cannot touch. A blinded human evaluation with ten PhD-level evaluators confirmed these gains, with SpyRL winning 80% of overall comparisons against the base model.
Perhaps most surprisingly, SpyRL also improved performance on verifiable math benchmarks — the domain where standard RLVR already operates. On Qwen3-4B, GSM8K accuracy jumped from 84.5% to 93.4%, and AIME 2025 accuracy tripled from 6.7% to 20.0%. The group-based competitive pressure apparently produces finer-grained learning signals than solitary RLVR training, encouraging the model to generate more rigorous and complete reasoning.
The Cost Advantage
One of the most compelling findings is the cost comparison. The paper benchmarks SpyRL against rubric-as-reward methods that use external LLMs (Qwen3.5-27B and GPT-4o) as evaluators during training. Those methods incurred approximately $200 and $900 in verifier costs respectively. SpyRL incurred zero — it needs no external judge at all. And despite the cost asymmetry, SpyRL matched or exceeded both methods on creative writing quality.
This matters for practical deployment. Training models on open-ended tasks has traditionally required either expensive human annotation or costly LLM-as-judge inference. SpyRL’s self-play loop generates training signal from the game structure itself, making it economically viable to extend RL training to subjective domains.
Cross-Task Transfer and Limitations
The paper also examines whether skills learned through SpyRL transfer across domains. The answer depends on domain similarity: a model trained on summarization showed positive transfer to creative writing (59.1% win rate against the base model), and vice versa. But a model trained on mathematical reasoning showed negative transfer to both summarization and writing. This suggests that the method’s benefits are tied to the nature of the target task — quantitative rigor doesn’t necessarily improve narrative quality.
The ablation studies confirm the importance of the full system. Removing the information asymmetry (no spy) collapses performance. Training only the performing or only the detection stage yields marginal gains compared to the coupled optimization. The zero-sum competitive structure between performers is what drives sustained improvement.
Why This Matters
RLVR’s limitation to math and code has been one of the most significant constraints on scaling reinforcement learning for LLMs. Every attempt to extend it — RLHF, DPO, LLM-as-judge — replaces deterministic verification with approximate signals, trading scalability for noise. RLSVR reframes the problem entirely: instead of approximating the missing verifier, transform the task so that verification emerges from the environment’s own rules.
The code and trained models are publicly available on GitHub, making it straightforward for other researchers to build on the approach. The framework itself is general — SpyRL is one instantiation of RLSVR, but the task-transformation principle could be applied with entirely different game structures. Any transformation that creates a known ground truth from the data itself, while preserving capability overlap with the target task, could work.
This is a genuinely novel angle on a problem that has frustrated the field. By treating reward verifiability as a designable property of the training environment rather than an intrinsic property of the task, RLSVR opens a path to scalable self-improvement in domains that have long resisted it.