Reliability Engineering for AI Agents: Contain Before You Recover

Reliability, resilience and failure-containment engineering is the discipline of proving an AI agent's success under repetition, not assuming it from one clean run. A retried task that finally returns the right answer does not prove the original attempt's effect on the world was undone. Executives funding agents to take real actions need to fund containment before they fund recovery, because recovery only ever restores what the system can see, not what already happened outside it.

What it is

Reliability, resilience and failure-containment engineering treats an AI agent's behavior as a repeated, perturbed and recoverable process, not a single demonstration. It separates properties an executive can otherwise blur into one word, "reliable": consistency, robustness under perturbation, predictability, recoverability and containment of impact when something goes wrong. A system can be available and wrong, correct but late, or recovered on a computer while a duplicate payment sits uncorrected in the outside world.

Failures are infrastructure, semantic or common-cause: an infrastructure fault, a timeout, is local and repeatable, a semantic fault survives a retry unchanged because the same reasoning produces it again, and a common-cause fault, the same poisoned document read by a generator and its own verifier, defeats redundancy that looks independent on paper. Execution state, semantic claims and external effects are three separate things.

Why it matters now

Agents now take actions with real consequences, and 2026's evidence shows genuine gains next to real gaps. A February 2026 study running fifteen models across repeated executions of GAIA and airline tasks found reliability dimensions improving unevenly rather than together.

A production enterprise agent studied in July 2026 lifted four hundred spreadsheet tasks from 80.25 to 91.25 percent correct once a verification loop was added, but the loop's own confusion matrix recorded eight true flags, zero false flags and thirty-two false confirmations, meaning most interventions confirmed an already-wrong answer. AgentRewind, an August 2026 workspace-recovery system, raised one paired recovery rate from 8 to 30 percent across fifty failed tasks, restoring only tracked workspace state, not a network call outside that boundary. A separate August 2026 review of 147 incidents across 81 runs on one software-delivery platform documented a run of fifty-four consecutive successful tool calls without real progress.

The architecture

I call this the three-ledger model: one ledger tracks whether the process ran, a second tracks whether its claimed conclusion is true, and a third tracks what actually changed outside the system. None may stand in for the others.

The execution ledger records whether a task is prepared, dispatched, committed or lost. The semantic ledger holds a claim as provisional until something outside the agent's own trace confirms it. The effect ledger is the one most systems skip: a durable record, created before an action is dispatched, of what was attempted against the outside world and under what identity.

The decision point sits at the moment a response goes missing. An agent that retries under a new identifier after no confirmation can cause the original action to commit twice, and no checkpoint can undo that. The correct move is to mark the effect ledger unknown, then reconcile it against the remote system's own record before retrying under the same durable identity.

Skipping this produces familiar failures: a duplicate external action from retrying past an unknown state, and a shared misconception that passes a verification loop because the generator and its verifier drew on the same wrong premise. Bounding permissions, assigning a durable action identity and reconciling unknown effects before any retry keep these out of production.

How to lead it

Ownership belongs to whoever runs the platform the agent executes on, typically a reliability or platform engineering leader, working against objectives a business owner has signed off on. That leader owns the ledgers, the reconciliation logic, and the containment controls: budget limits, scope limits and circuit breakers.

Fund durable action identity and reconciliation infrastructure before a bigger model or a faster verifier, since neither helps once an effect is already ambiguous. Measure verified completion, duplicate external effects, unauthorized actions and recovery time as separate figures, never blended into one score.

Two decisions should stay with an executive: how much authority an agent gets for an irreversible action, and when a stuck reconciliation gets escalated to a person rather than left to retry. Both are risk appetite calls.

What it is worth

The clearest 2026 evidence here is mathematical, not financial, and it argues for skepticism about a clean track record. Three hundred zero-failure test runs only support a failure rate below roughly one percent at ordinary confidence, and pushing that bound to one in ten thousand requires close to thirty thousand independent, representative trials.

Retry budgets carry a similar trap. If ninety percent of tasks succeed independently on a given attempt while the remaining ten percent are simply impossible with current tools, three retries produce about 89.9 percent success at 1.3 attempts each. Treating the blended rate as equally retriable instead predicts 99.3 percent success at 1.23 attempts, overstating the outcome and the cost of tasks no retry will ever fix.

No source behind this essay reports a dollar return figure for reliability investment, and none is claimed here. The evidence does support where to look for risk: common-cause failures shared between a generator and its own checker, rare severe events no test volume can rule out, and external effects a technically successful recovery leaves untouched.