The Four-Claim Model: What AI Interpretability Actually Proves
A readable AI explanation is not proof of a mechanism. The four-claim model shows what interpretability and steering evidence actually establishes.
Yash Sharma5 min readEvaluation and assurance
Research area 35: Interpretability, representation-steering and behavioral-analysis engineering
In brief
- A readable AI explanation only proves a signal is present, not that the model used it.
- The four-claim model sorts every interpretability finding by the test it actually passed.
- Most 2026 evidence supports narrow, bounded effects, not full visibility or proven safety.
Interpretability, representation-steering and behavioral-analysis engineering asks a narrower question than most vendor pitches imply: not whether an AI system can be explained, but what any single explanation proves. A readable label for an internal signal does not prove the model used it, and a technique that changes behavior does not prove it changed only the behavior targeted. Executives funding this as one all-purpose transparency layer are buying four different kinds of evidence, and 2026 research makes the difference impossible to ignore.
What it is
This area covers what a team can learn or control by reading and intervening on a model's internal activations, plus the behavioral tests that check whether a reading was actually used. Four tool families do the work. Probes are cheap classifiers showing information is present in recoverable form, nothing more. Attribution and lens methods connect an output to inputs or components under a method-specific definition of importance, an approximation rather than ground truth. Patching, ablation and steering are interventions that replace, remove or nudge an internal state and observe the effect. Sparse feature learning and natural-language decoders compress activations into a dictionary or generated text at the cost of fidelity.
Four principles govern what to fund. Access is not negotiable: patching and ablation need writable internal state, available on open-weight models and generally not closed APIs, so a proprietary study shows what its authors could instrument, not what a customer can reproduce. Superposition is real: one internal direction can carry several unrelated concepts, and good reconstruction does not make a feature dictionary uniquely correct. Distribution shift matters more here than almost anywhere in AI engineering: a direction found on one prompt set does not automatically hold on the next. A generated explanation is itself a model output shaped by training, not a window into computation, and an intervention is not selective until its effect on fluency, accuracy, refusals and tool choice has also been measured.
Production systems increasingly generate their own explanations at scale, and a plausible account is, from the outside, indistinguishable from a faithful one. A study of 1,517 claims about a single Qwen2.5-7B layer, first public July 22, 2026, found only 2.1 percent met its criterion for the claimed concept being verifiably encoded, despite .84 average reconstruction on those claims. A study first public in late August 2026 tested 25 open models from 4 billion to 744 billion parameters and found they could measurably alter a monitored activation signal during a bounded, harmless copying task.
The architecture
The clearest way to run this work is what I call the four-claim model: every finding is classified into one of four claim types before anyone acts on it.
The first type, predictive correlation, is a probe or readable label showing information is present and recoverable, nothing more. The Qwen2.5-7B layer study shows why this matters: high reconstruction accuracy and a verifiably-encoded concept were different things in nearly all 1,517 claims tested. Treat correlation as a lead, never as proof the model relied on it.
The second type, an intervention effect, is patching, ablation or steering changing behavior in a controlled test. It only counts once it clears real controls: a matched-norm random direction as comparison, a declared list of collateral outcomes measured alongside the target metric, and retesting after any model or prompt-distribution change. The activation-control study is intervention-type evidence of exactly this kind, real and bounded, not shown to generalize past the ten concepts and copying task it tested.
The third type, a constrained mechanistic account, predicts behavior across held-out interventions and counterfactuals, not one experiment. A Jacobian-lens study, published July 6, 2026, built an averaged lens across 1,000 prompts to read and intervene on what its authors call a global workspace, earning this tier while calling its own account an incomplete approximation.
The fourth type, a useful incomplete approximation, is sparse feature dictionaries and natural-language decoders that compress activations into something readable, valuable for exploration, not self-certifying. A May 7, 2026 study of such decoders reported variance reconstruction around .6 to .8, a measure of recovered information, not truth, and found confabulation with no benefit over a search-based baseline. Good reconstruction does not make a decoded description true. Every claim that reaches a decision should carry its tier label.
Ownership belongs with a team that did not build the system being read, the same logic that puts verification in independent hands. That team owns the probes, protocols, collateral-effect suite and the claim-tier label attached to every finding. Fund a collateral-effect suite run alongside every steering or ablation experiment, negative controls and matched-norm random directions run by default, and a revalidation trigger tied to model version, prompt distribution and language.
Two decisions belong to the executive alone: whether a specific claim is strong enough to gate an irreversible action, and what the organization will say publicly about what its interpretability program can and cannot detect, since overstating that coverage to a board or regulator is a liability of its own.
What it is worth
Money and risk show up in three places: engineering time spent building and validating internal-state tooling, the cost of a decision made on a claim weaker than presented, and the exposure created when a monitoring signal is trusted past what it was tested for. The Jacobian-lens and decoder methods are not cheap, and their diagnostic value should be weighed against a simple probe or plain behavioral test run by the same team.
The 2026 evidence supports that targeted interventions can produce measured, controlled effects, and that natural-language decoding can produce useful, if unreliable, descriptions of activations. It does not support that any reviewed method gives complete visibility into a model, that a passed monitor establishes the absence of dangerous behavior, or that steering is stable across languages, model versions and adversarial pressure. None of the four studies behind this essay carries an independently verified replication.
Questions leaders ask
- Does a readable AI explanation prove the model actually used that signal?
- No. A probe or a decoded label only proves information is present in a recoverable form, what the dossier calls a predictive correlation. A 2026 study of a single model layer found that high reconstruction accuracy and a verifiably encoded concept were different things in nearly all the claims it tested. Treat a readable explanation as a lead to investigate, never as proof of how the model actually reasoned.
- Can activation steering change one behavior without affecting anything else?
- Not by default. A steering intervention has to be compared against a matched-norm random direction and measured against a declared list of collateral outcomes, including fluency, factual accuracy, refusal behavior and tool choice, not only the target metric. The 2026 evidence reviewed here supports that targeted steering can produce a measured effect. It does not yet support that any tested intervention is fully selective under realistic conditions.
- If an AI monitor shows no alarm, does that mean the system is safe?
- No. An absence of alarm is evidence about the specific tests that monitor has passed so far, not a general safety guarantee. A 2026 study of 25 open models found they could measurably alter a monitored activation signal in a bounded, harmless task, evidence that a monitored signal can be manipulated in at least some settings. Report a clean monitor as one input to a decision, never as proof of safety on its own.
- Which interpretability methods work on a closed, API-only AI model?
- Mostly the behavioral ones. Patching, ablation and direct activation steering require writable internal state, which a closed API generally does not expose, so these interventions stay largely limited to open-weight models a team can instrument directly. A proprietary internal study demonstrates what its own authors could instrument, not what an ordinary customer using the same product can reproduce, and that access gap should shape what any vendor claim is allowed to imply.
