1 Foundations

1.1 Chapter Map
- Define RLVR as learning from verifiable reward signals and explain verifiable tasks.
- Analyze why RLVR became ubiquitous in reasoning models and preview the structure of the book.
1.2 What RLVR Is
RLVR is reinforcement learning on tasks where some meaningful part of correctness can be checked directly. The check you implement can be exact, as in symbolic math or formal proof, or the result of an executable, as in unit tests. It can be partial, i.e. grounded question answering or tool-using agents where only some parts of the trajectory can be reliably scored. The unifying idea is the availability of a success notion, because a task/environment possessing useful correctness signals can be used to improve search at test time, or RL’d against to optimize a model, resulting in systems that improve far beyond what static supervised fine-tuning alone produces.1
We use verifier as the default term for the mechanism that checks output and produces a signal; related terms include checker, scorer, and sometimes judge.


1.3 Origins of RLVR
In some sense RLVR is akin to the “OG” reinforcement learning paradigm, since it learns from direct reward rather than preference comparison (the basis of reinforcement learning from human feedback, or RLHF), just like the classic RL environments, e.g. CartPole; what is new is the application to LLMs through verifiers that can check answers, code, proofs, and traces.
I personally reflect back on the advent of reasoning models and reinforcement learning through a strange amnesia of an idea so simple with hindsight, but which took two years after ChatGPT to discover. This assessment, however, is unfair in the sense that the idea to make models think step by step long predates the 2024 reasoning-model wave.2 The broader prompting paradigm emerged across late 2021 and early 2022: scratchpads for intermediate computation appeared first, chain-of-thought prompting then formalized the use of intermediate reasoning traces, and the exact prompt “Let’s think step by step” was popularized a few months later.
Before the reasoning-model wave of 2024, code generation had already explored reinforcement learning against executable verifiers: CodeRL (July 5, 2022), PPOCoder (January 31, 2023), and RLTF (July 10, 2023) all trained language models using unit tests or execution feedback as objective reward signals.3
Math-Shepherd was a landmark mathematical-reasoning RL paper, using a learned process reward model to train language models with step-by-step PPO (Wang et al. 2024). DeepSeekMath introduced GRPO and was published on February 5, 2024; it stands as the first paper to apply critic-free RL to mathematical reasoning at LLM scale.
Things heated up in September 2024, when OpenAI published Learning to Reason with LLMs (o1), indicating that they had used a train-time and test-time compute strategy to enhance model reasoning through reinforcement learning in math and coding tasks.4 The name “Reinforcement Learning with Verifiable Rewards” (RLVR) was coined in the Tulu 3 paper from November 22, 2024.5 Finally, there was DeepSeek-R1 at the start of 2025, which demonstrated the full verifier-driven RL formula for bootstrapping reasoning models (DeepSeek-AI et al. 2025). To quote someone describing the atmosphere at Meta after R1 launched, “Engineers are moving frantically to dissect DeepSeek and copy anything and everything we can from it,” and according to Fortune, there were war rooms assembled at Meta to understand how a Chinese lab with substantially fewer resources was beating them.6
1.4 Verifiable Tasks
Verifiable tasks separate better behavior from worse at acceptable cost (imagine a task that required an expert chemist to read the agent traces and assign a score manually; this is likely not an acceptable cost). Indeed, chemistry was not the first domain to be scaled via verifiable rewards; math, code, and formal proof became central to RLVR because correctness is tractable. For a math problem, one might normalize an answer and compare it with a reference answer, while code and formal proof use programs to verify correctness (Shao et al. 2024; Liu et al. 2023; Xin et al. 2024).
1.5 RLVR and Reasoning
RLVR is a training paradigm, and reasoning is a downstream capability/artefact, e.g. multi-step breakdown, search, planning, tool use, etc. The marriage between the two occurs because the most successful reasoning domains are the ones which leverage strong verifiers. It’s therefore understandable that RLVR and reasoning are conflated, since verifier-friendly domains are the best places to scale reasoning performance.
1.6 Verifiable versus Complete
It is important to monitor the verifier’s blind spots, since designing verifiers that are robust across all possible inputs, especially during prolonged training runs, is non-trivial. Consider two examples of verifier weaknesses:
- Unit tests cannot check every possible regression that an agent may introduce in software.
- A math reward may depend on brittle extraction.
These examples raise important questions to consider in applying RLVR:
- what is being checked?
- what is being missed?
- how expensive is the check?
- how easily can the signal be gamed?
1.7 What We Cover
The next chapters move from the general paradigm to the main reward regimes in practice. Chapters 2 through 4 cover outcome rewards, process rewards, programmatic, learned and hybrid verification pipelines. Chapter 5 demonstrates turning a verifier into a learning signal. Chapter 6 turns to search and test-time verification, and Chapter 7 covers reward hacking. Chapter 8 documents the 2026 incidents in which models from four frontier labs reached real systems outside their sandboxes, in large part due to the optimization pressure in scaling RLVR. Chapters 9-11 discuss real frontier RLVR recipes, agents, RL environments, and open problems.
Whereas supervised fine-tuning is bottlenecked by data quality, if one has an RLVR environment which is robust and difficult in interesting ways, models can become superhuman, which is an important paradigm shift.↩︎
A useful compressed lineage runs from scratchpads in late 2021, to chain-of-thought prompting in January 2022, to the exact zero-shot prompt “Let’s think step by step” in May 2022 (Nye et al. 2021; Wei et al. 2022; Kojima et al. 2022).↩︎
CodeRL was submitted on July 5, 2022 and used unit tests and a critic model to guide program synthesis (Le et al. 2022). PPOCoder was submitted on January 31, 2023 and used execution-based feedback with PPO (Shojaee et al. 2023). RLTF was submitted on July 10, 2023 and used online unit-test feedback of multiple granularities for code LLMs (Liu et al. 2023).↩︎
OpenAI’s writeup states that
o1performance improved with both more reinforcement learning, which they describe as train-time compute, and more time spent thinking at test time (OpenAI 2024).↩︎DeepSeekMath introduced GRPO and used RL to improve mathematical reasoning in an open model (Shao et al. 2024). Tulu 3 later introduced the name “Reinforcement Learning with Verifiable Rewards (RLVR)” for this broader training pattern (Lambert et al. 2024).↩︎
The quoted line was reported as an anonymous Teamblind post summarized by TMTPOST, while the claim that Meta created four “war rooms” was reported by Fortune, citing The Information (TMTPOST Global 2025; Quiroz-Gutierrez 2025).↩︎