Training Environments Need a Reward Contract, Not a Guess

Executable environment compilation and reward-contract engineering decides whether a training environment that runs deterministically under code actually behaves like the real system it imitates. A simulator built from documentation, traces or source code can execute cleanly and still teach an agent the wrong lesson, because a wrong deterministic oracle trains a policy just as efficiently as a correct one. Treat a compiled environment as code plus a stated set of assumptions plus a reward contract, and fund the audit of the last two as carefully as the first.

What it is

This area bundles two disciplines: compiling a documented or observed system into an executable environment, and writing the reward contract that scores what an agent does inside it. An environment exposes a reset that returns the world to a known state from a seed and fixture, and a step that returns an observation, the reward's components, and whether the episode ended by task resolution or a time limit. A time cap is not proof of failure.

Four principles hold. Code-backed does not mean world-true, since a generated simulator is itself a model whose errors must be measured, not assumed away because the code runs. Every inferred transition must be logged as an assumption, separate from source fact. Reward is a contract with stated precedence, not one number: a booking with the wrong date must fail even when local checks pass. Whatever the policy should not see, including any grading-only field, must stay outside its observation, or the environment trains it to read the answer key.

Organizations are using generated environments to scale training past what humans can hand-build, and a policy that satisfies a flawed environment looks successful until it meets the real system. Three 2026 systems show the spread. EnvHarness, posted August 20, 2026 and read here at evidence grade B, wraps an existing environment by changing its starting states and task chains while keeping the backend and verifier in place, a narrower claim than reliable compilation from documentation alone. Agent World Model, posted February 2026 and revised in May, and Safe and Scalable Web Agent Learning via Recreated Websites, posted March 2026, describe hundreds of synthetic or recreated environments with transfer results, but both are read here at evidence grade C, abstract and metadata only, neither independently replicated.

The architecture

The operating model here is the environment admission contract: a compiled candidate passes through four gates before it can generate training data.

The assumption ledger gate logs every inferred transition, separated from source fact. The reset and replay gate requires a reset to restore database state, caches, the random-number generator, clock assumptions, permission fixtures and queues, checked by replaying a snapshot and confirming state hashes match. The solvability gate requires a reference policy to find at least one successful path, while a separate adversarial pass tests for reward tampering, out-of-range actions, unreachable goals and shortcuts that pay out without doing the task. The reward audit gate has a validator with no authority to rewrite the objective check the reward contract against its stated precedence, confirming an exact database predicate is evidence about that database, not user success.

An admitted environment is versioned as one unit, its code hash, reward version, checkpoint, tool adapter, fixture and seed recorded together, since any one changing invalidates prior trajectories. One that fails any gate is quarantined, and every trajectory it already generated is traced back through that record to decide whether the fix needs a rescore, a regeneration, or quarantining a trained checkpoint.

How to lead it

Split ownership: the engineers who build the environment should not also run the independent audit. Fund reset and replay checks, the adversarial search and the reward audit at the same level as building the environment, since verification time can exceed build time. Measure reset determinism rate, solvability witness coverage, reward audit pass rate and version-record completeness on one dashboard, and stop accepting a reward curve alone as evidence of readiness.

Two decisions belong at the executive level only: setting what admitted means for a category of environment before results are visible, since the bar differs for a low-stakes workflow versus anything touching payment, health or legal outcomes, and approving any change to a reward contract's precedence, since that silently redefines what the training run is optimizing.

What it is worth

The money shows up in three places: compute spent training against a reward that never converts into a capability that works outside the simulator, audit cost left out of the budget because it does not look like training, and the cost of discovering after deployment that a policy satisfied its environment rather than the task, which means regenerating trajectories, retraining and an incident review at once. None of that is visible from a reward curve.

Measure lifecycle cost, build, verification, training, deployment and failure recovery, against task completion an independent evaluator confirms outside the environment, not reward-curve movement. The evidence supports funding an inspectable, code-backed environment over a narrative one wherever backend access allows it, and funding an independent adversarial audit before any reward is trusted at scale. It does not support a universal solvability certificate, a guaranteed reset for arbitrary real-world effects, one fidelity metric across every task, or any specific return figure. This essay draws on three primary sources, one at grade B and two at grade C, and its own search of the literature remains partial.