The Manifest Ledger: Proving AI Development Actually Compounds

Harness-of-harness, artifact evolution and evidence continuity is the discipline of keeping an AI development loop's proof of work valid while the underlying project keeps changing. A fixed AI coding agent placed inside a loop that versions its own evidence can make incremental work cumulative, so a team stops re-proving what it already tested. What is not yet settled, even in the study most associated with the idea, is whether adding supervision and role hierarchy actually buys better quality once total compute, tools and review time are held equal.

What it is

An artifact is a versioned bundle: requirements, source, dependencies, datasets, environment assumptions, acceptance criteria and user-visible behavior, treated as one identity, not files edited by hand. Evidence is a result bound to one specific version of that bundle and one test protocol, not a floating fact that survives every future change.

Four kinds of change move through a project, each changing what earlier evidence still means: artifact edits to requirements or source, harness edits to prompts, routing or tool orchestration, numerical policy updates to model parameters under a training contract, and environment transformations to the tasks or dynamics the agent is tested against. Two things follow: a result is only as good as the dependencies it was tested against, so a build hash alone is not sufficient proof the parts underneath still work, and a different prompt run on the same underlying model is not an independent check.

The architecture

I call the design that holds up under those caveats the manifest ledger: an identity for the work, an independent gate that decides what counts, and a permanent record that keeps every attempt, not only the winners.

The manifest is a hash over everything a result actually depends on: requirements, source tree, dependency lock, data schema, container, evaluator and permission policy. Two builds with the same manifest are the same artifact for evidence purposes. Change any one field and the identity changes, and evidence tied to the old identity needs an explicit check before anyone reuses it.

A candidate carries its base version, the parts of the artifact it writes, the parts it reads, and its spending budget, and is evaluated only against the tests its declared dependencies could affect.

The gate is the one authority that decides accept or reject, sitting outside the document proposing the work. Accepted changes commit atomically and the manifest moves forward. Rejected attempts change nothing about the project state but are appended to the ledger anyway, since remembering only winners cannot tell a rare success from a lucky one, and best-of-many scoring measures selection, not quality.

A September 2026 paper from Shanghai AI Laboratory tested a multi-day software development benchmark and reported a three-pass configuration scoring 71.52 against a vanilla single-agent baseline's 58.24, at 8.41 million tokens against 6.33 million, with a two-pass configuration at 64.84 and 5.67 million tokens. The comparison ran one pass per task with no reported reproducible seeds, evidence graded B. The authors kept the harness fixed for the study, contrary to the name. A separate September 2026 study tested six creator models building their own agent harnesses across four domains and 2,207 task instances, finding the harnesses uneven in quality and weaker than mature, hand-built references, evidence graded C.

How to lead it

Ownership sits with whoever controls the gate. Leaders funding role hierarchy on the Shanghai demonstration should ask for the matched-budget comparison, since matching total token spend and matching pass count are different experiments.

Fund manifest-hash tooling covering every dependency, isolated candidate environments, a dependency-aware test selector that avoids full re-runs, and an append-only ledger keeping rejected attempts alongside accepted ones. Run scheduled interruption drills checking whether the recovered project reconciles its files, its stated intentions, its evidence and any external effects already caused, since a restored checkpoint does not undo a message already sent or a purchase already cleared.

Two decisions stay with the executive alone: the stopping rule, when to accept the current state as the minimum acceptable product because further expected improvement is worth less than its cost, and authorization, which of artifact edits, harness edits, policy updates and environment changes a given team may make, since an environment transformation needs its own validation authority and should never inherit sign-off from an artifact-edit process.

What it is worth

The financial exposure sits in three places: time spent re-verifying work a stale summary claimed was already proven, cost from a regression a noisy or repeatedly queried test score let through, and unrecoverable exposure from an external action, a message sent or a record written elsewhere, that a checkpoint restore cannot undo. An August 2026 study of agent rollback documented exactly that category of failure across internal state, external state and irreversible effects, evidence graded C, thin but pointed.

Measure before and after on acceptance rate, regression rate and interruption-recovery success at matched budget on a fixed set of tasks, not total throughput, which rewards attempts rather than accepted quality. The evidence supports that a fixed base agent inside a manifest-bound, evidence-versioned loop is workable and can make incremental work more cumulative. It does not support that hierarchy itself, rather than the evidence discipline underneath it, is the reason quality improves at equal total budget, and the dossier behind this essay is itself a partial-scope review, not a completed literature survey. No return figure or adoption timeline follows from it.