The Substrate Gate: Where AI Systems Should Store Information
AI systems should route information through a gate that checks deletion and tenant isolation first, then let cost decide between context, memory and weights.
Yash Sharma5 min readContext, retrieval and memory
Research area 81: Information bottlenecks across context, memory, weights and harnesses
In brief
- Run each storage medium through deletion and isolation checks before cost decides where a fact lives.
- The substrate gate rules out media that cannot delete or isolate data, then cost picks among the rest.
- A purpose-built adapter beats simple retrieval only after about forty-five uses, and only once deletion is already solved.
Information bottlenecks across context, memory, weights and harnesses is the executive question of where an AI system should keep what it knows: the live prompt, an external record, a fast weight update, the model's trained weights, or the reviewed code that runs it. Getting it wrong burns money on tokens, training and migrations, or creates knowledge nobody can find, correct or delete when a regulator asks. The choice is not which medium performs best. It is which media are even allowed to hold it, and only then does cost decide.
What it is
Six kinds of storage compete for the same information. Current context is the live prompt: immediate, expensive per call, gone when the window closes. External records, episodic for what happened and semantic for what is believed true, hold volatile facts needing a version and an expiry date. Fast weights are session-specific, cheap to reuse but tied to their creating integration. Persistent weights amortize reusable patterns across every call but resist correcting or deleting a single fact. Skill documents are reviewed procedures, and harness policies are executable code that acts on the world, a different attack surface.
Sufficiency is relative to the question asked: keeping half of a two-bit fact answers questions about that half and nothing else, so no fixed summary answers every future question. Cost is not token count: a full accounting includes acquisition, compression, storage, retrieval, training, inference, migration, verification, deletion and human maintenance. Functional overlap is not operational equivalence: a context summary and a fine-tuned update can answer the same demo question yet differ in granularity, interference, tenant isolation and rollback cost.
A study of test-time training through context distillation, posted August 2026, found a causal window can write a residual signal into fast weights under aligned queries and no interfering key products: an eight-billion-parameter Llama model averaged 73.75 on RULER against 72.85 for continual pretraining after ten billion tokens, a real mechanism but not a deletion guarantee. A revised study of deployment-time memorization found raw-plus-resummarization, full purge and tombstoning each produced zero observed residue in a fifty-case test, correcting a narrower first version, though zero on fifty cases does not prove the wider population is clean. A study of information-restricted reasoning found a four-dimensional bottleneck that looked narrow until removing a raw-text pathway the model could also read cost 8.2 accuracy points on two tasks.
The architecture
The clearest way to run this decision is what I call the substrate gate: constraints eliminate a storage medium outright before cost or performance gets a vote, and only what survives competes on cost.
The gate asks four questions: can this substrate demonstrate itemwise deletion, does it keep one tenant's data out of another's answers, can it be refreshed before a decision goes stale, and can someone reconstruct why a fact was accepted. A substrate failing any one is out, however well it scores, because these are feasibility conditions, not penalties against a benchmark. Persistent weights fail deletion most often, cheap at inference but hard to correct without retraining, and fast weights inherit the same problems unless engineered in.
What survives competes on full cost against a fixed default: volatile private facts in versioned external records, task-local state in live context, stable procedures in reviewed harness code, reusable learning in weights only where access and deletion permit it. A derived summary or adapted weight is not separate from its source: it inherits the same lineage and deletion obligation, so tracking every record version but not what was derived from it reports a clean deletion that is not clean.
Ownership sits with whoever is accountable for the decision, jointly with each substrate's owner. Fund the fixed default first: versioned records for volatile or private facts, live context for one-off state, harness code for stable procedures, weight changes only where deletion and access are already solved.
Two calls belong to the executive alone: whether a candidate substrate is even permitted for a class of information, decided before anyone compares performance, since one that cannot demonstrably delete or isolate tenant data is ruled out regardless of how well it tests, and approving any move from the fixed default toward a learned allocator, since that hands a sensitive placement decision to a system optimizing its own proxy metric.
What it is worth
Money, risk and time show up in nine places: acquisition, compression, storage, retrieval, training, inference, migration, verification and deletion, plus ongoing human maintenance. An illustration in the underlying research makes the tradeoff concrete, not measured: external lookup at one unit per query against an adapter costing forty units to build plus a tenth of a unit per query turns cheaper only past roughly forty-five queries, and twenty rule changes needing twenty new adapters cost eight hundred in construction alone. That is illustrative, and beside the point if exact deletion is mandatory and the adapter cannot demonstrate it, because the gate decides first.
Measure before and after against the same baselines: full context, a fixed summary, query-conditioned retrieval, external memory with full lineage, a supported fast-weight method, under an equal budget. Track deletion against an independently audited set, since that number and a system's own lineage can differ by the copies nobody checked. The evidence supports testing fast weights under narrow assumptions and auditing a channel for bypasses. It does not support a validated dollar return or evidence that an adaptive allocator beats the fixed default.
Questions leaders ask
- Should information live in the prompt, in a database, or in the model's weights?
- Run it through the gate before comparing performance. Check whether the storage medium can demonstrate itemwise deletion, keep tenants isolated, stay fresh enough for the decision at hand, and be audited for why a fact was accepted. Only among the media that pass those checks does cost decide, and the fixed default is versioned external records for volatile facts, live context for one-off state, and weights only where deletion and access already work.
- Can an AI system prove it has deleted something?
- Only within the fifty-case tests a 2026 study actually ran, where raw-plus-resummarization, full purge and tombstoning each showed zero observed residue after a revision that corrected an earlier, narrower conclusion. A zero result on fifty audited cases is a real finding, not proof that a wider population is clean, and any deletion claim should say exactly which set it was checked against.
- Is a smaller, low-bandwidth memory automatically safer to govern?
- No. A 2026 study of information-restricted reasoning found that a narrow four-dimensional channel was not the true boundary of what the system could use, because a separate raw-text pathway remained available, and removing it cost 8.2 accuracy points on two tasks. A channel that looks narrow while an unaudited bypass sits next to it is not actually narrow, and governing only the visible channel misses where the real information is.
- When does it make sense to bake information into a fine-tuned weight update instead of retrieving it every time?
- Only once the gate clears it, meaning deletion and tenant isolation are already solved for that data. Past that point it is a cost question: an illustrative comparison in the underlying research has a purpose-built adapter beating simple external retrieval only after roughly forty-five uses, and losing badly again if the underlying rule changes often enough to need repeated rebuilding.
- What decision should stay with an executive rather than a technical team?
- Two decisions. Whether a candidate storage medium is even permitted for a class of information, decided before anyone looks at its performance, since a medium that cannot demonstrably delete or isolate data should be ruled out on that basis alone. And whether to approve any move from the fixed default allocation to a learned one, since that hands a sensitive placement decision to a system optimizing its own proxy metric.
