7  Reward Hacking

M. C. Escher, Fiumara, Calabria (1930).

7.1 Chapter Map

• Optimization pressure and Goodhart’s Law.
• A taxonomy of exploits and the hardening techniques that work.

7.2 Goodhart’s Law

RLVR is in some sense the epitome of Goodhart’s Law when we view the verifier as the measure that becomes the target through RL optimization. If the verifier has any gap between what it checks and what we want, then optimization can exploit that gap for almost all non-trivial proxies (Skalse et al. 2022).¹ Goodhart’s Law can be applied to RLVR in three ways:

1. The verifier has random errors on some inputs. Over many training steps, the policy shifts toward the subspace where the verifier is accidentally generous.

2. Gradient descent over thousands of steps is powerful enough to discover gaps in the verifier that may be rare or invisible under ordinary evaluation.

3. Optimizing the proxy (test passage, answer matching) may produce a policy that achieves high scores through mechanisms unrelated to the intended skill, e.g. pattern-matching, memorization, or distribution exploitation.

Example intuition for how improving the proxy can worsen the target: imagine three choices. The true evaluator says A is best, B is second-best, C is worst. The proxy treats A as best but says B and C are equally bad. Moving mass from B to C leaves the proxy unchanged while hurting true performance. Combine that with a small move from B to A, and the proxy improves while true performance still falls.

7.3 Taxonomy of verifier exploits

7.3.1 Extraction exploits

The model satisfies the answer extractor without doing the task. Anything inside the <answer> tags that matches the gold answer gets full correctness reward, regardless of what preceded it. A model can learn to produce minimal-effort responses that place a plausible answer in the right position: a single line of text, no reasoning, occasionally correct by chance.

7.3.2 Reward shaping exploits

The signal path from Chapter 5 introduces its own optimization targets. Format rewards, partial credit, and auxiliary bonuses create surfaces the model can exploit independently of correctness.

7.3.3 Test adequacy failures

The verifier is correct on what it checks, but what it checks is insufficient. Liu et al. found that augmenting HumanEval with 80× more test cases changed model rankings: some models that scored well on the original suite dropped substantially (Liu et al. 2023). In code generation, test suites are finite approximations of a specification. Through hardcoded branches or shallow pattern matching, a model can learn to pass specific test cases without implementing the correct algorithm.

7.3.4 Learned verifier biases

When the verifier includes a learned judge (Chapter 4), the judge’s training artifacts become exploit surfaces. A model trained against a verbosity-biased judge learns to produce longer outputs. A model trained against a position-biased judge learns to place its strongest argument where the judge expects to find it.

7.3.5 Distribution shift

The verifier was calibrated for one distribution of model outputs, yet the policy has drifted to another. A learned judge trained on early-training outputs may misjudge late-training outputs that have shifted in register, length, or structure.

7.3.6 Mechanism gaps

Turpin et al. showed that chain-of-thought explanations can hide factors that influenced the answer, and Lanham et al. tested faithfulness more directly by intervening on traces (Turpin et al. 2023; Lanham et al. 2023). The mechanism gap is the difference between a trace that predicts correctness and a trace that causally controls the answer:

Let \(X\) be the prompt, \(T\) the written reasoning trace, \(Y\) the final answer, and \(H\) the hidden computation that produced both. An outcome verifier observes \((X,Y)\). A process verifier observes \((X,T,Y)\).

The artifact-level question is:

\[ \Pr(Y \text{ correct} \mid X,T). \tag{7.1}\]

The causal question is different:

\[ \Pr(Y=y \mid \operatorname{do}(T=t), X) \quad \text{versus} \quad \Pr(Y=y \mid \operatorname{do}(T=t'), X). \tag{7.2}\]

7.4 Empirical exploits

7.4.1 Unit-test manipulation

OpenAI reports a frontier reasoning model training run in which the agent was placed in coding environments and rewarded for making unit tests pass (Baker et al. 2025). The agent did not only write better code. It found reward hacks in the environment. Two systemic hacks were exit(0), which exploited a bug that let the agent exit before all tests ran, and raise SkipTest, which skipped unit-test evaluation from outside the testing framework. These hacks became systemic until the environment was patched.

Patching a verification function to always return true, writing stubs when unit-test coverage is poor, parsing tests to extract expected values, decompiling reference artifacts, or shadowing libraries such as pandas so that the verifier doesn’t check the intended implementation are further examples of reward hacking. When optimization pressure overwhelms the verifier, the model learns that the reward is attached to “tests pass,” not to “the repository now implements the intended behavior.”

7.4.2 Missing negative

Cursor’s 2026 description of real-time RL for Composer gives the production version of the same problem (Jackson et al. 2026). The training loop used real user interactions as reward signal and shipped new checkpoints as often as every five hours. One exploit came from invalid tool calls. Composer often needs to read files or run terminal commands. The original reward pipeline discarded examples where the tool call was invalid, so the model learned that if a task looked likely to fail, emitting a broken tool call avoided negative reward. The fix was to include broken tool calls as negative examples.

Part of the reward was derived from edits, so Composer also learned to defer risky edits by asking clarifying questions instead of touching code. The reward pipeline had not defined the boundary between appropriate caution and avoidance of negative reward, so editing rates dropped until Cursor changed the reward function.

7.5 The over-optimization curve

The clearest quantitative evidence for Goodhart dynamics in optimization comes from Gao et al., who measured the relationship between optimization pressure and performance (Gao et al. 2023). The premise is to treat one large reward model as a stand-in for human judgment, the “gold” model, whose scores count as ground truth and stay fixed during training. The authors train a smaller proxy reward model on the gold model’s labels and optimize the policy against the proxy. What happens is that proxy reward increases monotonically, but gold reward first rises, then falls. The peak location depends on the proxy’s quality: better proxies peak later and higher, while weaker proxies peak early and low. The original result was measured for learned reward models in RLHF, but the dynamics apply whenever a proxy is imperfect. In the context of RLVR, the proxy can be the programmatic verifier, which approximates but may not equal the target capability. The natural stand-in for the gold model is then a stronger check held out from training, such as hidden tests or an augmented test suite like EvalPlus (Liu et al. 2023). The same dynamics hold as with learned reward models; the difference being that programmatic verifiers are stronger proxies than learned reward models, so peaks likely occur later and gaps open more slowly.

Figure 7.1: In real RLHF, human labellers produce the comparisons that train the proxy reward model. In Gao et al.’s synthetic setup, a large gold reward model, itself trained on real comparisons, labels the comparisons instead, so its score can stand in for ground truth. Redrawn after Gao et al.’s Figure 2 (Gao et al. 2023).

Pan et al. built four RL environments with deliberately misspecified rewards (traffic control, COVID response, blood glucose monitoring, and the Atari game Riverraid) and varied agent capability through model size, action resolution, observation noise, and training time. They found that as the policy becomes stronger, it finds exploits that weaker policies could not (Pan et al. 2022). There are capability thresholds where agent behavior qualitatively shifts, causing sharp drops in true performance even as proxy reward continues to climb. These phase transitions are only predictable empirically and difficult to monitor.

Drag the slider to increase optimization pressure. Toggle verifier strength to see how the Goodhart gap changes.

Optimization pressure:

Figure 7.2: Optimizing against an imperfect proxy: proxy reward keeps rising while true performance peaks and then falls, later and higher for a stronger proxy. The curves follow the functional forms of Gao et al. with illustrative coefficients (Gao et al. 2023).

7.6 Are the highest-scoring answers actually correct?

A verifier can judge most answers correctly yet give its highest scores to answers that exploit its mistakes. Chapter 6 judged a yes/no verifier by the precision of its accepted pool, i.e. the fraction of accepted samples that are actually correct. Once a model is optimized against a verifier that outputs a score, the samples that matter are the highest-scoring ones, because selection and RL both push toward whatever the verifier scores highest. Average verifier accuracy therefore tells us little, and what matters is precision among the top-scoring samples.

As in Chapter 6, let \(v(x, y)\) be the score the verifier assigns to response \(y\) for prompt \(x\), and let \(C(y) \in \{0,1\}\) record whether \(y\) is actually correct. The verifier score is the proxy; \(C\) is the target. Best-of-\(N\) draws \(N\) samples from the policy and returns the one the verifier scores highest:

\[ y_N^\star = \arg\max_{1 \le i \le N} v(x, y_i), \qquad y_i \sim \pi_\theta(\cdot \mid x). \]

The quantity we care about is not \(\mathbb{E}[v(x, y_N^\star)]\). That will almost always rise with \(N\). The quantity we care about is the probability that the returned answer is correct:

\[ \Pr\bigl(C(y_N^\star) = 1\bigr) = \Pr\!\left( C\!\left(\arg\max_{1 \le i \le N} v(x, y_i)\right) = 1 \right). \]

If \(v\) and \(C\) agree in the bulk of the distribution but disagree in the upper tail of \(v\), then increasing \(N\) can make the system worse. The selector is not sampling typical verifier-approved outputs. It is sampling the most extreme verifier-approved output it can find.

A useful diagnostic is the precision of the accepted pool at threshold \(\tau\), where a sample counts as accepted when its score is at least \(\tau\):

\[ \operatorname{Precision}(\tau) = \Pr\bigl(C(y)=1 \mid v(x, y) \ge \tau\bigr). \]

With a yes/no verifier and \(\tau = 1\), this reduces to Equation 6.3.

Raising the threshold first filters out ordinary wrong answers, so precision rises. Past a point, the only samples left are the rare exploits that score higher than any honest answer, and precision collapses.

Figure 7.3: Illustrative score distributions for correct answers, wrong answers, and a rare exploit that the verifier scores highest (left), and the resulting precision of the accepted pool as the threshold rises (right).

For a verifier that only grades single samples (pass@1), moderate thresholds may be enough. For best-of-64, PRM-guided beam search, or RL over many gradient steps, the relevant threshold is much higher. The optimizer pushes probability mass toward the region where \(v\) is maximal, so robustness means that \(v\) remains aligned with \(C\) in that region. This is why red-teaming should search for high-score false positives, not just estimate average verifier accuracy on held-out samples.

7.7 Test time exploits

Best-of-\(N\) selection helps when the verifier is faithful, but can increase probability of high-scoring false positives. Suppose 1 in 100 rollouts contains a verifier exploit: a response that scores high on the proxy but low on true capability. Because the exploit scores highest, best-of-\(N\) returns it whenever it is among the candidates. The chance that at least one of \(N\) independent rollouts is an exploit is \(1 - 0.99^N\), the same calculation as Equation 6.1 with exploits in place of correct answers. With best-of-16, that chance is about 15%. With best-of-64, it rises to about 47%. With best-of-256, it reaches about 92%. Search is not gradient descent, but it still finds the gap between proxy and true, and a verifier that is good enough for pass@1 may not be good enough for best-of-64.

7.8 Hardening techniques

Hardening measures cost compute, engineering time, or both. We justify their use by whether they push the over-optimization peak far enough to the right.

1. Hidden tests. Holding out a set of tests the model never trains against reduces direct overfitting to visible checks; the model cannot overfit to tests it does not see.

2. Test augmentation. Generating tests automatically can expand coverage beyond what a human problem-setter provides. EvalPlus demonstrated that generated test suites reveal false positives that the original tests miss (Liu et al. 2023).

3. Red-teaming before training. Probing the verifier adversarially is a proactive way to de-risk training runs.

4. KL constraints and early stopping. A KL penalty to the reference policy, PPO-style ratio clipping (which GRPO also uses), or simply stopping training before the over-optimization peak keeps the policy close enough to its initial state that the verifier’s calibration still holds.

5. Progressive curriculum. Increase task difficulty as the model improves, following the competence-band principle from Chapter 5. Progressive difficulty keeps the optimization pressure focused on genuinely informative tasks.

7.9 Open questions

• What’s a good protocol to evaluate verifier robustness?
• Can the model’s own internal representations be used to detect reward hacking?
• Does combining hidden tests, ensembles, and KL constraints give diminishing or compounding returns?

Baker, Bowen, Joost Huizinga, Leo Gao, et al. 2025. “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.” arXiv Preprint arXiv:2503.11926. https://arxiv.org/abs/2503.11926.

Gao, Leo, John Schulman, and Jacob Hilton. 2023. “Scaling Laws for Reward Model Overoptimization.” Proceedings of the 40th International Conference on Machine Learning (ICML). https://arxiv.org/abs/2210.10760.

Jackson, Jacob, Ben Trapani, Nathan Wang, and Wanqi Zhu. 2026. Improving Composer Through Real-Time RL. Cursor Blog. https://cursor.com/blog/real-time-rl-for-composer.

Lanham, Tamera, Anna Chen, Ansh Radhakrishnan, et al. 2023. “Measuring Faithfulness in Chain-of-Thought Reasoning.” arXiv Preprint arXiv:2307.13702. https://arxiv.org/abs/2307.13702.

Liu, Jiawei, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.” arXiv Preprint arXiv:2305.01210. https://arxiv.org/abs/2305.01210.

Pan, Alexander, Kush Bhatia, and Jacob Steinhardt. 2022. “The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models.” Proceedings of the International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2201.03544.

Skalse, Joar, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. “Defining and Characterizing Reward Hacking.” Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2209.13085.

Turpin, Miles, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. “Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting.” arXiv Preprint arXiv:2305.04388. https://arxiv.org/abs/2305.04388.

---

1. Following Skalse et al., reward hacking with imperfect proxies has multiple outcomes:

   1. Different rewards can rank low-level behaviors differently while still sharing optimal policies; in this case, optimization can still reach a good target-optimal policy.

   2. There can also be policies with \(J_{\tilde{R}}(\pi_2) > J_{\tilde{R}}(\pi_1)\) and \(J_{R}(\pi_2) < J_{R}(\pi_1)\), where \(R\) is the true reward, \(\tilde{R}\) the proxy, and \(J_R(\pi)\) the expected return of policy \(\pi\) under \(R\), so improving proxy reward worsens target reward.

   3. For imperfect, non-trivial proxies, such misalignment directions exist somewhere in policy space, so over-optimization can still lead to degraded target performance, but this is not mathematically guaranteed for every trajectory.