Auditing an AI Program When Every Piece Passes Its Own Test

Expanded coverage audit, contradiction resolution and cross-layer synthesis is the discipline of checking a large AI program for duplicated work, unresolved conflicts between sources, and the exact seams where components that each pass their own test can still fail once connected. One recent audit checked fourteen research tracks in full and the internal consistency of thirteen already-written specialist reports, and was honest that this counts as a self-consistency check, not outside verification. That distinction, and the seams it protects, is what a board should demand before funding the next phase.

What it is

The area covers how to audit an AI program once it has grown past a single team's memory: many tracks, many claims that a component is faster, safer or more capable, and a real risk no one checks where those claims meet. It does not claim a numeric completeness score, average incompatible benchmark scores into one number, or count a repeated citation as corroboration.

Five principles carry the discipline. Ownership before content: every cache, checkpoint, memory store or evaluation set gets one accountable owner, and a similar-sounding title elsewhere is an interface to confirm, not evidence the work is done. Attribution survives correction: a source's affiliation, date and version must be checked directly, since a headline assumption can be wrong even when the result is real. A local pass is not a system pass: a component can pass its own test while the assembled system fails at the boundary where components hand off. Grade the evidence: preliminary, first-party or confounded work earns a pilot, never a rollout, however strong its headline number. The auditor is not independent of itself: reviewing its own prior output catches contradictions, not outside replication.

Several August and September 2026 studies each modify how an AI agent operates itself, and reading them as one trend is how a board overfunds a shared assumption. The audit corrects citations rather than repeating them: Harness-of-Harness's artifact evolution belongs to Shanghai AI Laboratory, not the model or harness assumed.

The architecture

I call the practice the seam ledger, because the real risk in a compound AI system lives at the handoff between two components, not inside either one. It has three parts: a component claim, the point of local success, a seam, the moment authority, evidence or a budget changes hands, and a system outcome, what happens once the claim crosses into a use it was not tested inside.

Five seams recur, each with a documented way to pass locally while the system fails. Authority: a resumed action carries a checkpoint valid at save time whose approval was revoked before it replayed, a failure no signature or unit test catches alone. Evidence: a test result is truthfully reported for one build while a dependency change ships a different binary under the same name, so the evidence is stale and launched at once, both truthfully. Protected evaluation: a metric generator never sees test labels and a downstream updater only sees rollback decisions, so the system adapts to the held-out set though each half looks clean alone. Propagation: a source record is deleted correctly but a summary cache or trained adapter keeps the content, so the removal claim is false though every component reports success. Resource: two components each read the same shared budget within what they saw, yet the total exceeds what was available, because reading a number is not reserving it.

None of these five failures shows up in a component's own test suite, only when someone deliberately checks the seam. One owner, not a committee, should hold the ledger as a living document, updated at every release. Fund the work in order: deterministic identities and dependency manifests first, no model required, then the five counterexamples above against the real system, then one adaptive mechanism piloted against a simple baseline it must beat, then the integrated system evaluated against protected, unseen histories, then promotion for what stayed inside scope. Two decisions belong to an executive alone: setting which failure classes are unacceptable before the pilot runs, not after, and refusing claims the evidence does not support, a universal exactly-once guarantee, privacy from data merely hard to read, or correctness from something compiling.

What it is worth

Money and risk show up at the seams, not inside any single component's invoice. The audit's own illustrative accounting: a simpler workflow at 1.3 units per attempt and 90 percent success costs roughly 1.44 per successful task, a more elaborate one at 1.8 units and 95 percent success costs roughly 1.89, and the extra 0.5 a unit of complexity costs earns its place only once harm avoided by that five point gain exceeds ten times its cost. Those are hypothetical units, not a return figure for a business case.

The same source is direct about compound risk. Two layers that each fail on the same tenth of cases still fail together a tenth of the time, while two layers with disjoint failure cases fail a fifth of the time, and a 90 percent score on each layer alone does not say which world you are in. That overlap is what to ask a vendor for.

What the evidence does not support is a general reliability multiplier for adding a seam ledger to any program. Measure cost per successful task and the overlap in failure cases before and after, and treat that, not a borrowed figure, as the return.