The Co-Design Ledger: Proving a Joint Harness and Weight Gain

Joint harness, weight and policy co-design tunes an agent's scaffold, prompts, tools, routing and memory together with its model weights, instead of treating the two as separate programs. It sounds like it should beat improving either lever alone. The 2026 evidence does not yet support that assumption, and the failure mode is not the idea but how the reported gain gets measured.

What it is

An agent has two genuinely different things that can change. The harness is everything that shapes behavior without touching a trained parameter: instructions, tool descriptions, routing rules, memory and retrieval policy, and the execution graph. The weights are the trained parameters a training run adjusts. Two more objects need independent version control: the reward that scores an attempt, and the environment the agent acts in. A reward that drifts alongside the system it grades no longer measures what the business approved.

One phrase, self-evolution, covers at least four mechanisms. Rejection-sampling training retrains the model on its own successful attempts without estimating a reward gradient. Policy-gradient reinforcement learning needs the model's own log probabilities per step. Harness search proposes and runs a candidate prompt, tool or execution graph with no weight change. A fourth regime freezes the model and optimizes only the documents around it.

My organizing principle: a reported gain from training harness and weights together is not evidence the two interact, until a design isolates the interaction. It can be nothing more than a stronger critic, more attempts, or removal of a tool-format bottleneck either lever alone would have fixed.

2026 produced the first concrete recipes, and each audit trail is thinner than its headline. WHALE, made public August 31, 2026, alternates rejection-sampling training and harness search on 2 billion and 4 billion parameter models, but selects its final pair by accuracy on the test set itself. SafeEvolve, made public September 2, 2026, pairs prompt and skill edits with harness-focused training and reports gains on AgentDojo, but only its abstract was available, leaving its ablations and durability against later training unknown. A related Anthropic report, dated August 28, 2026, found automated researchers can outperform a human comparison on alignment experiments, but recorded limited search budgets and cheating attempts, so any safety gain is not proven durable.

The architecture

The framework I use to make a co-design decision auditable is the co-design ledger: four checkpoints a candidate harness-and-weight pair must clear before its reported gain counts as real.

First, the manifest: before training starts, the team lists every independently versioned object, instructions, tool descriptions, execution graph, skills, memory rules, adapters, router, reward and evaluator, with the reward and evaluator held apart from the rest.

Second, the paired run. Alternating training with harness search, or evolving both together against live agent experience, are both legitimate. The promoted pair must be chosen on a validation set training never touched, and proven on a sealed test opened once after every phase decision is final, the checkpoint WHALE's own algorithm fails by picking its final pair on test accuracy.

Third, the factorial audit. One alternating run cannot show whether harness and weights interact or one lever did all the work. The ledger requires all four combinations, original harness with original weights, optimized harness alone, trained weights alone, and both together, at matched budgets. Only when the combined cell beats the best single-lever result by more than the two levers' separate gains added, with an uncertainty interval excluding zero, does the ledger record a genuine interaction.

Fourth, the immutable core and its compatibility report. Authorization, network scope, any externally effective action, and the evaluator's own integrity sit in a boundary no training loop can touch. Every promoted pair also carries a compatibility report, re-run against the current provider, tokenizer and tool schema, since a provider update can break instruction priority while accuracy scores hold steady.

How to lead it

One named co-design owner holds the ledger, with standing authority to decline a promotion regardless of how good the pair looks. Fund the full factorial audit, a genuinely sealed test set, and a compatibility report triggered by every provider or tool-schema change. Measure the interaction estimate itself, safety behavior on a protected evaluation the training loop never sees, and total cost including failed rollouts and the serving change the pair needs. Stop promoting any pair whose test set touched checkpoint selection, and stop treating API-only search as equivalent to a weight-trained result.

Two decisions belong to an executive alone: how much of the model's weights to expose to training, and whether the immutable safety core sits outside every future training run before the first cycle begins, since retrofitting it later means auditing every change made since.

What it is worth

Money and time show up in proposal generation and failed rollouts, teacher labels for a warm start, training and evaluator-call cost, the serving change the pair needs, and a compatibility report each provider update. Risk shows up as a pair promoted on a contaminated test set, or a reward that drifted alongside the system it scored.

Measure with the ledger's four-cell design, so the reported gain is the interaction itself, not the best single lever's score. A program earns back what it cost once the promoted pair's serving-quality advantage, discounted for the complexity it adds, exceeds what the four checkpoints cost to run, across enough volume before the next provider update forces a re-audit.

The 2026 evidence supports that alternating harness and weight optimization is workable, and that a safety-oriented version is being tried in the open. It does not establish a reproduced interaction effect, a safety guarantee surviving further training, or a return figure a business can adopt directly. Both recipes reached only an author-claim evidence grade, with no independent replication yet.