2  Outcome Rewards

M. C. Escher, Castrovalva (1930).

2.1 Chapter Map

  • Explain outcome verifiers.

2.2 How outcome verifiers are implemented

A useful abstraction of outcome-verification pipelines is in three steps:

  1. Extract.

    • Parse the model’s raw text to isolate the checked artifact. This depends on the output contract: the <answer> tags in our scaffold, \boxed{} in many math benchmarks, the final code block in a generation task, or the proof term in a formal system.
  2. Canonicalize.

    • Map the extracted artifact to a representation that is stable under harmless surface variation. In math this can mean parsing (2,3), {3,2}, and x=2, x=3 into the same set object.
  3. Reward.

    • Assign a reward value. The simplest version is binary: 1 if correct, 0 otherwise. Partial credit for passing some but not all tests, or a continuous score from a symbolic similarity metric are possible too.

Current RLVR frameworks differ in how they represent the extract -> canonicalize -> reward interface, and it also depends on the domain you’re working in; see the Hugging Face reinforcement learning environments explainer for a comprehensive rundown (Kolavi et al. 2026). If we consider math-style tasks, those reward functions often delegate most work to answer-verification libraries such as Math-Verify, whose documented grading architecture is explicit: answer extraction, conversion to a common representation, and gold comparison (Kydlicek 2025).

2.3 A Rollout

A rollout is a sample from the current policy on a prompt, example:

Prompt

Solve the equation: \(x^2-5x+6=0\).

Completion

<think>
We can factor the quadratic: x^2 - 5x + 6 = (x-2)(x-3).
Set each factor to zero:
x - 2 = 0 -> x = 2
x - 3 = 0 -> x = 3
The solutions are 2 and 3.
</think>

<answer>
(2,3)
</answer>

Outcome reward check (for RLVR).

The verifier reads between <answer>...</answer>, normalizes/canonicalizes the answer, and checks it against the ground-truth set.1

\[ r(\text{input},\text{output})= \begin{cases} 1 & \text{if } \operatorname{canon}\bigl(\operatorname{extract}_{\mathrm{ans}}(\text{output})\bigr)=\{2,3\},\\ 0 & \text{otherwise.} \end{cases} \tag{2.1}\]

If the model deviates from the output contract e.g. omits <answer>...</answer>, the verifier can assign an incorrect reward even when the underlying solution is correct.

Figure 2.1: The above function is many-to-one.

2.4 A minimal outcome verifier

A toy math verifier for the quadratic example looks like this:

import re

ANSWER_RE = re.compile(r"<answer>\s*(.*?)\s*</answer>", re.DOTALL)

def extract_answer(completion: str) -> str | None:
    match = ANSWER_RE.search(completion)
    return None if match is None else match.group(1).strip()

def canonicalize_answer(answer: str) -> tuple[str, ...]:
    text = answer.strip().lower()
    text = re.sub(r"\\?[{}()]", "", text)
    text = re.sub(r"^roots?\s+are\s+", "", text)
    text = re.sub(r"\bx\s*(?:=|\\in)\s*", "", text)
    text = re.sub(r"\b(?:or|and)\b", ",", text)
    pieces = [piece.strip() for piece in text.split(",")]
    number = re.compile(r"-?\d+(?:\.\d+)?")
    if not pieces or any(not number.fullmatch(piece) for piece in pieces):
        return ()
    return tuple(sorted(set(pieces), key=float))

def outcome_reward(completion: str, gold: tuple[str, ...] = ("2", "3")) -> float:
    answer = extract_answer(completion)
    if answer is None:
        return 0.0
    candidate = canonicalize_answer(answer)
    expected = tuple(sorted(set(gold), key=float))
    return float(candidate == expected)

2.5 Outcome check, full-trajectory update

Although verifiers only consider the outcome, the optimizer updates the entire trajectory. In REINFORCE-style algorithms (including GRPO), the scalar reward (or an advantage derived from it) is used to upweight or downweight the log-probability of every token in the completion. If the advantage relative to the sampled group is positive, the whole chain of reasoning that produced it becomes more likely; the converse is also true!

To quote Andrej Karpathy on his October 17, 2025 appearance on the Dwarkesh podcast (Patel 2025):

Every single one of those incorrect things you did, as long as you got to the correct solution, will be upweighted as “Do more of this.” It’s terrible. It’s noise.

You’ve done all this work only to find, at the end, you get a single number of like, “Oh, you did correct.” Based on that, you weigh that entire trajectory as like, upweight or downweight. The way I like to put it is you’re sucking supervision through a straw. You’ve done all this work that could be a minute of rollout, and you’re sucking the bits of supervision of the final reward signal through a straw and you’re broadcasting that across the entire trajectory and using that to upweight or downweight that trajectory. It’s just stupid and crazy.

A human would never do this. Number one, a human would never do hundreds of rollouts. Number two, when a person finds a solution, they will have a pretty complicated process of review of, “Okay, I think these parts I did well, these parts I did not do that well. I should probably do this or that.” They think through things. There’s nothing in current LLMs that does this. There’s no equivalent of it. But I do see papers popping out that are trying to do this because it’s obvious to everyone in the field.

One cannot deny the success of outcome-based RLVR, but in this regard, it is a blunt instrument that assigns one scalar per completion. This works well in practice because over many rollouts and many problems, tokens that consistently appear in correct trajectories get reinforced and tokens that appear in incorrect trajectories get suppressed, but it also means that outcome rewards cannot isolate one step as good or bad. That distinction is what process rewards (Chapter 3) provide.

sampled option unsampled options
Figure 2.2: Example from Section 2.3 continued.

2.6 Domain-specific considerations

Domain What we’re verifying Typical verifier Main bottleneck
Math2 Final answer or structured mathematical object Answer extraction, canonicalization, symbolic equivalence, or reference comparison Output contracts and normalization
Code3 Program, patch, or execution result Sandboxed tests, hidden tests, timeouts, and optional static checks Test coverage and flaky infrastructure
Formal proof4 Proof term, tactic trace, or proof state Proof-assistant kernel e.g. Lean Search, decomposition, and formalization burden

2.7 Failure modes

  • An extractor can reward obedience to formatting conventions more than correctness.
  • A canonicalizer can fail to merge equivalent answers or merge distinct answers into one canonical form.
  • A verifier can evaluate the wrong capability due to reward hacking.

  1. DeepSeek-R1-Zero uses <think>/<answer> separators and applies task-specific response-shape constraints for reward parsing, including boxed final outputs when useful for deterministic math verification (DeepSeek-AI et al. 2025).↩︎

  2. (Shao et al. 2024; DeepSeek-AI et al. 2025).↩︎

  3. (Le et al. 2022; Shojaee et al. 2023; Liu et al. 2023).↩︎

  4. (Xin, Guo, et al. 2024; Xin, Ren, et al. 2024).↩︎