The Evaluation Chain of Custody for Trustworthy AI Claims
A passing benchmark score is not proof. The evaluation chain of custody defines, freezes, grades and gates the evidence behind an AI release decision.
Yash Sharma5 min readEvaluation and assurance
Research area 31: Evaluation, benchmark and statistical-measurement engineering
In brief
- A benchmark score is not evidence until you name the population, unit and denominator behind it.
- The evaluation chain of custody defines, freezes, grades and gates every AI release claim.
- Skipping evaluation discipline turns a clean report into a production incident, not a return figure.
An evaluation score is not evidence until you can name the population it estimates, the independent unit it counts and the denominator behind it. Evaluation, benchmark and statistical-measurement engineering turns a benchmark result into a claim a board can act on. Most AI programs skip straight to the score, which is why a clean evaluation report so often arrives just before an incident.
What it is
Evaluation is applied statistics aimed at one question: does this system, under this resource policy, do what the business needs on the population it will actually meet. A benchmark is the task collection and execution protocol. An estimand is the population quantity a study is trying to estimate, such as the true rate at which an agent completes a class of claims correctly. A metric is the calculation, a score one observed value of it, and an evaluator the whole apparatus around that value.
Four principles matter most: define the estimand and denominator before a single run, treat repeated runs of the same task as nested observations of one unit rather than new independent evidence, keep metrics separate since pass at one, pass at k, consistency, completion, groundedness and harm answer different questions, and treat every grader, executable test, rubric, language model or person, as an instrument with its own error rates, validated on a blind sample before its verdicts are trusted.
Agents now act, not just answer, raising the stakes. A 2026 study ran seven API models across 36 expert tasks, each two to twelve expert hours, collected 756 trajectories in a shared harness, and found trajectory level metrics expose differences a single aggregate score hides. A January 2026 Anthropic account recommends starting a suite from twenty to fifty real failure cases, useful for beginning but not a power calculation or a certification of reliability.
The architecture
My rule is to run every evaluation claim through what I call the evaluation chain of custody: four links that each must hold or the claim is not admissible for a release decision. Define sets the population, the independent unit and the denominator. Freeze locks the exact system version, harness, tools, context policy and budget under test, and records when the system was available, when the report went public and when the experiment ran. Grade routes evidence through validated instruments, layering executable checks, rubrics, language model judges and human adjudication. Gate turns graded evidence into a release decision using a margin and an uncertainty bound set before the evaluation started, not a point score chosen after the fact.
Each link has a failure mode, and any one breaks the chain. Define fails when three seeds on ten tasks get counted as thirty independent samples instead of ten clusters. Grade fails in a way a March 2026 judge debiasing study shows directly: optimizing a judge against one bias benchmark cut that judge's bias score roughly in half, from about 20.7 to 10.8, while degrading its performance on grading tasks it was not optimized against. Gate fails whenever a fixed sample size or leaderboard threshold is treated as a guarantee rather than the output of a stated risk tolerance. A defensible release decision holds all four links at once, and an explicit finding of insufficient evidence is a legitimate output of the chain, not a failure of it.
How to lead it
Ownership works best outside the team optimizing the system being measured, the separation a finance function expects between the desk that trades and the desk that marks the trade to market. Give that lead three artifacts: a benchmark card recording each test set's population, source and known omissions, a grader card recording each grader's validated error rates, and a release gate protocol stating the margin and power calculation a candidate must clear before shipping. Report missing traces and abandoned tasks alongside successes, since a dashboard of completed runs alone is biased by construction.
Two decisions belong to the executive alone: setting the tolerated risk and smallest improvement worth shipping before the evaluation runs, since a margin chosen after seeing the result is not a margin, and accepting an honest finding of insufficient evidence as legitimate rather than pressuring the team toward a number. Version the model, prompt, tools, policy, data and environment independently, and keep the map from every accepted claim back to what it tested.
What it is worth
Money and risk show up in the gap between what a passing score implies and what it actually provides. Thirty runs with zero observed failures looks like proof of near perfect reliability, but the one-sided ninety five percent upper bound on the true failure rate is still about 9.5 percent, and pushing that bound to 1 percent takes roughly 300 trials, not 30. Programs that fund the thirty and skip the other 270 are pricing a guarantee they have not purchased.
Risk also hides in the grading step: a judge tuned narrowly against one bias benchmark can look better on the watched metric while getting worse at ordinary grading, so a program tracking only that metric can regress with no internal alarm.
The reviewed evidence supports trajectory level grading and judge bias trade-offs as real, bounded findings, not a universal sample size or a return figure, so none is claimed here. What it is worth is the difference between a decision that would survive a second look and one that would not.
Questions leaders ask
- What is the single biggest mistake companies make when evaluating an AI agent?
- Treating a single benchmark score as a release certificate instead of naming the population it represents, the independent unit being counted and the denominator behind the number. A score without those three declared is not evidence a board can rely on, since a different population, unit or denominator can produce a very different real world result even when the printed score looks identical.
- How many test runs are enough to prove an AI system is safe?
- There is no universal number, and thirty clean runs prove less than most teams assume. Zero observed failures across thirty independent trials still carries a one-sided ninety five percent upper bound on the true failure rate of about 9.5 percent. Driving that bound down to 1 percent takes roughly 300 independent, representative trials, and the real number depends on the variance in your own data and the risk you can accept.
- Can a language model judge replace human evaluators entirely?
- Not safely on its own. Language model judges scale semantic grading well, but a March 2026 study found that optimizing a judge against one bias benchmark improved that specific score while quietly degrading its performance on ordinary grading tasks. The safer design layers executable checks, rubrics, judge models and human adjudication, and validates each one's error rate independently rather than trusting any single grader by default.
- Who inside a company should own AI evaluation?
- An evaluation lead who sits outside the team optimizing the system being measured, similar to the separation between a trading desk and the function that marks its positions. That lead owns the benchmark card, the grader card and the release gate protocol, and reports the full picture, including timeouts, denied actions and abandoned tasks, not only the successful runs that make a dashboard look clean.
- What does an insufficient evidence finding actually mean?
- It means the available sample cannot support the claim being asked of it, not that the system has failed. The honest response is to narrow the claim to what the evidence supports or gather more evidence before a release decision, rather than stretching a small sample past what a power calculation would allow. Treating this as a legitimate outcome, not a failure, is a decision only an executive sponsor can protect.
