10.1 Chapter Map

Why frontier labs now train across many harnesses and generate their own environments.
What long context and multimodality change in RLVR.

10.2 From reward functions to harnesses

Long-context and tool-using settings break the clean RLVR picture from earlier chapters. A rollout may now include repository state, browser state, images, shell transcripts, tool arguments, observations, generated files, runtime failures, partial progress markers, and a termination decision.

The harness decides what the policy is allowed to observe, what actions it can take, what state changes get logged, and which artifacts become reward-bearing. In a math problem, this surface can be tiny: prompt in, final answer out, exact checker at the end. In long-context, multimodal, and agentic tasks, the harness becomes sacrosanct. It may log evidence spans, retrieved images, browser actions, shell output, patches, timeouts, and tool failures.

Figure 10.1: The trajectory, environment state, and verifier stack part of the training interface.

10.3 Long context

Long-context RLVR keeps the verifier unchanged, since final-answer verification is independent of context length; nevertheless, it stretches everything before the check, as the evidence the model needs sits somewhere in tens of thousands of input tokens. QwenLong-L1 found that applying RL naively to long inputs trains inefficiently and unstably, so it adapts a short-context reasoning model in stages: a supervised warm-up, then RL phases on progressively longer inputs, with hard examples from earlier phases sampled again later (Wan et al. 2025).

Long rollouts finish anywhere from seconds to hours after starting, meaning batching is king, since an unsophisticated trainer that greedily batches rollouts fills its early batches with short, easy tasks and its later batches with hard ones. MiniMax reports greedy batching causes gradient oscillation and optimization instability, and for its M2 models uses what it calls windowed FIFO: the trainer is greedy only inside a sliding window at the head of the generation queue, and a finished rollout beyond the window must wait until the window reaches it. Accordingly, the trainer never runs more than one window ahead of the oldest unfinished rollout; the effect is that short rollouts beyond the window cannot crowd slow ones out of a batch. The window’s size is a tunable fraction of the queue, which MiniMax sets to 30% (MiniMax 2026). DeepSeek-V4.1-Flash counters the same bias by capping how many rollouts each dataset can have in flight and discarding the short samples that return first at the start of training when the length bias is strongest, until batches reach their steady-state length distribution (DeepSeek-AI 2026).

Figure 10.2: Windowed FIFO. Rollouts enter the queue in the order they start, and the trainer may take any finished rollout inside the window (blue), but none beyond it (red cross); the window advances by the oldest rollout. Inspired by Figure 5 of the MiniMax-M2 report (MiniMax 2026).

10.4 Multimodal

Despite being multimodal, when applying RLVR to visual settings, we tend to keep the verifier text-based; the alternative cases put vision inside the loop, either as something the policy acts on or as something the verifier looks at.

10.4.1 Vision in the policy’s loop

Kimi K3 trains visual reasoning over STEM problems, visual puzzles, and charts as an agentic task where each trajectory runs in a Python interpreter sandbox, and the model writes and executes code to crop, zoom, transform, compute, or check intermediate results in a REPL loop. The reward is still a checkable final answer, but the image is an environment the model manipulates over many steps rather than a fixed input (Kimi Team 2026). This iterative approach seems more scalable, as K3 scores 94.3% on Math-Vision without tools and 97.8% with Python, and 23.0% on ZeroBench-main (pass@5) without tools and 41.0% with Python (Kimi Team 2026).

10.4.2 Vision in the verifier

Kimi K3’s web-development environments combine deterministic checks with LLM judges such that the deterministic checks test application behavior and, for tasks that replicate a reference, score structural and pixel-level similarity. The judges then inspect the source code or look at and interact with the rendered output, and the reward is zeroed when the project fails to build, runs with errors, or fakes the artifact instead of implementing it, a hard gate against the reward hacking of Chapter 7 (Kimi Team 2026). MiniMax takes the judge approach further for its M2 models, as its Agent-as-a-Verifier deploys a generated application in a sandbox, then has a verifier agent drive it with Playwright, clicking buttons and completing workflows, before judging visual quality. For slides, a visual scorer renders the deck and judges it, and trajectories built with several different slide libraries are mixed in so that the policy does not overfit to one rendering toolkit (MiniMax 2026). In both reports the verifier is a hybrid stack in the sense of Chapter 4, with hard execution gates first, followed by learned visual judgment.

10.5 DeepSWE

DeepSWE is a software coding agent trained with the rLLM platform that uses Qwen3-32B as the harness policy and is trained with GRPO++, a modified GRPO. Training took six days on 64 H100 GPUs, and on SWE-bench Verified the result is 42.2% pass@1, 71.0% pass@16, and 59.0% when hybrid test-time scaling selects among 16 rollouts (Agentica Team and Together AI 2025).

The training environment is a subset of R2E-Gym, i.e. dockerized and executable software-engineering tasks with natural-language task descriptions, repositories, unit tests, and reward calculation by running tests (Jain et al. 2025). DeepSWE used 4,500 of these tasks, after removing any drawn from the same repositories as SWE-bench Verified to avoid contamination, and each RL iteration spawned 512 Docker containers in parallel: a batch of 64 tasks with 8 rollouts each (Agentica Team and Together AI 2025).

A DeepSWE-style rollout has this shape:

The task is a natural-language issue against a repository at a fixed commit.
The model searches for relevant symbols and files.
The model views code and accumulates local evidence in a long context window.
The model edits or creates files.
The model executes commands or tests and observes failures.
The model revises the patch until it stops or hits the step limit.
The harness records the trajectory, output patch, exit reason, timeout state, and reward.
The RL trainer masks the loss for trajectories that hit the maximum context length, a 20-minute generation timeout, or the step limit, and updates the policy from the rest.

The crux here is that the harness itself shapes the policy (the Qwen model we are post-training) through the observations it returns, unique tools, valid action syntax, and timeout rules. That is to say, putting the post-trained model in an equivalent harness with different tool names would lead to worse results, since the tokens the model would need to generate in order to tool-call would be farther out of distribution. Kimi K3’s report makes the same observation: training in a single fixed harness can make a model overfit to its tool schema, system prompt, context management, and interaction protocol. Given Kimi is a model which is used across many third-party harnesses and cannot afford to be overfit to one harness in the same sense that a model trained mainly for its lab’s own harness, such as Claude Code or Codex, can, K3 is therefore trained across many harness configurations, including ones that mimic Claude Code and Codex (Kimi Team 2026).

10.6 MiniMax

During MiniMax’s training, the policy (the M2 model being trained) runs inside an agent scaffold, i.e. a program such as Claude Code that sends a prompt to a model API, calls tools, rewrites the prompt, and repeats. To learn from one of these calls, the trainer needs the exact prompt the model saw and the tokens it returned, since RL raises or lowers the probability of that response given that prompt. Yet every agent rewrites its prompts in its own way, whether by truncating old tool outputs, summarizing, or delegating to sub-agents, so rebuilding the prompts from outside would require reimplementing each agent’s logic. MiniMax’s solution is a gateway, think of it as a middleman that forwards each request and keeps a copy. The agent sends each of its requests to the gateway’s model API instead of directly to a model; the gateway passes the call to the policy being trained, returns the policy’s answer to the agent, and saves the exact prompt and response as a training example. The trainer thus needs to know nothing about how the agent built each prompt. MiniMax says the gateway design has been validated across hundreds of distinct agent scaffolds and thousands of tool-call formats (MiniMax 2026).

Many agent harnesses send their requests to one gateway, which forwards them to the model; the model and the training loop feed each other. — Figure 10.3: MiniMax’s gateway: many agent harnesses call one model API, and the gateway forwards each call to the policy, returns the answer, and records the exact prompt and response as a training example.

For example, take an agent fixing a bug over three calls. In the first call, the model sees the issue and asks to read a file. In the second, it sees the issue, its request, and the file’s 2,000 lines, and runs the tests. Before the third call, the agent replaces the 2,000 lines with a two-line summary to save space, so the model sees the issue, its requests, the summary, and the test output. The gateway records three training examples, each holding exactly what the model saw at that call, including the summary in the third. A trainer that rebuilt the context from the agent’s full history would instead train the third step on 2,000 lines the model never saw at that point.

MiniMax distinguishes two kinds of agent by how much of this rewriting the trainer can see. A white-box agent runs its context management, such as sliding-window truncation or periodic summarization, inside the training framework, so the trainer sees each transformation and can build training sequences that match the states the policy faces at inference. A black-box agent is opaque: the trainer sees only what each call exposes, namely the context sent to the model, the model’s response, and the tool results, as in the bug-fixing example. That is enough to train on, and it lets MiniMax plug in agents with deep thinking loops, aggressive context rewriting, or hierarchical multi-agent coordination without modifying them. The gateway serves both; a white-box agent differs only in also registering its context-management operations (MiniMax 2026). The black-box interface is what enables training across many harnesses, and that diverse training can help produce a broadly capable policy.

10.7 DeepSeek

DeepSeek scales RL along training compute and the number of scaffolds; on DeepSWE v1.1, a DataCurve benchmark unrelated to the DeepSWE agent, DeepSeek-V4.1-Flash resolves between 65.5% and 74.2% of tasks across eight configurations from six scaffold families, including Claude Code, Codex, and OpenCode, a robustness DeepSeek says is consistent with the diversity of environments, tool schemas, and interaction formats in its synthesized training data (DeepSeek-AI 2026).

10.8 Where environments come from

Ilya Sutskever¹ proclaimed at NeurIPS 2024 that “pretraining as we know it will unquestionably end”, because compute keeps growing while “we have but one internet”: “the fossil fuel of AI” (Sutskever 2024). Environments are the RL counterpart of pretraining’s internet, and high-quality environments are harder to scale than pretraining data. DeepSWE trained on 4,500 R2E-Gym environments, which were curated semi-automatically from real GitHub commits, and although expert data will only become more important over time, automating environment construction will inexorably grow, because of the cost of man-made environments and as a by-product of recursive self-improvement (RSI). DeepSeek is aware of this and consequently runs a synthetic environment effort, which defines tasks as a triplet:

a problem,
an environment, and
a verification system.

They score each task on difficulty and correctness and use those scores as rewards to train the model to build better tasks (DeepSeek-AI 2026). For coding, specialized agents check whether a repository can build and run in a container, choose a starting commit or session turn, design implementation directions, specify fail-to-pass and pass-to-pass evaluation points, and then turn them into test code in an isolated container.

Figure 10.4: Fail-to-pass (top) and pass-to-pass (bottom) tests.

Writing the patch that resolves an issue is hard, while running tests against a patch is cheap, so a model can write the check for a task it cannot yet solve. Jason Wei calls this difference the asymmetry of verification: “some tasks are much easier to verify than to solve” (Wei 2025). As is standard in reinforcement learning environment creation, for general agents DeepSeek builds mocked tools that reproduce, at train time, the interfaces and behaviors of real tools seen in employee and partner usage, and turns failure cases submitted by employees into new environments that replay them (DeepSeek-AI 2026). Anchoring task generation in real workflows and real failures is one clever way to guard against model collapse.²