Why Semantic Caching Needs Authorization, Not Just Similarity

Semantic caching, invalidation and authorization-equivalence engineering decides when a cached answer, a retrieved passage or a stored model action can be reused safely, and when it must be recomputed. A fast cache hit is only valuable if it is also correct and permitted: a nearly identical request can carry a different date, jurisdiction, requester's rights, or a source since edited or revoked, and a cache that checks only similarity will not notice. Executives funding retrieval-augmented and agentic AI systems should treat this as a governance question with a profit and loss line, not a backend detail.

What it is

This is inventory management for AI-generated work: knowing what was produced, under what conditions, for whom, and whether those conditions still hold. Conflating the different kinds of reusable object is where the risk starts. Exact prefix caching reuses computation on an identical token sequence, narrow and low risk. Semantic answer caching reuses a full response judged equivalent to an earlier request, where meaning must be verified, not assumed. Retrieval, tool-result and workflow-checkpoint caching store search results, action observations and mid-process state. Shared model state, attention memory kept warm across requests, reuses computation, not meaning, and is never evidence an answer is still correct.

A cache hit is a compound claim: the requester is currently authorized, nothing the answer depends on has changed, it was produced under a compatible model and policy version, and the new request means the same thing as the old one. Similarity finds candidates, it never approves one. A 2026 audit of embedding-cosine similarity gates in agent systems tested a production-style gate against deliberately mutated queries and found it approved 56 harmful reversals, including one pair scoring a cosine similarity of .9608, comfortably inside the range most single-threshold designs treat as safe.

The architecture

I call the mechanism the four-tumbler lock: a cache hit should only turn when four independent checks align, the way a lock cylinder turns only when every pin lifts to the right height. The tumblers are authority, does the requester currently have the right to this artifact, version compatibility, was it produced under the same model, schema, policy and serving code, dependency freshness, has anything it traces back to changed or been revoked since caching, and equivalence, does the new request mean the same decision as the cached one. A similarity score can only inform the fourth tumbler, and even there it needs calibration, never a global cutoff.

Authority and version compatibility are cheap and deterministic, so they run first, inside an authorized partition, and either can reject a request before expensive matching begins. Dependency freshness runs next, checked against a graph recording what every derived object relies on, so revoking one source invalidates every summary and downstream decision built on it. Equivalence, the most expensive and least certain tumbler, runs last, once entitlement and currency are confirmed.

A miss at any tumbler is not a failure, it should fall through to plain recomputation with a bounded timeout, not a retry loop or a degraded answer. The common failure in the research is the opposite: a similarity score overriding a deterministic denial, or a fixed time-to-live that ages out an entry while missing a revocation landing moments after a refresh.

Ownership splits cleanly: platform engineering owns the mechanism, the cache, the dependency graph, invalidation events, version bookkeeping. The business owner of the decision being cached, a clinical, claims or pricing lead, owns the authority rules and the definition of equivalence, because only they know which changes are consequential. Fund the dependency graph before funding a learned equivalence model. Measure harmful false-hit rate and severity, stale-evidence rate, revocation failures and correct cache declines, and stop rewarding raw hit rate alone, it rewards the unsafe reuse this discipline exists to prevent.

Three decisions belong to an executive alone: which harm classes get near-zero tolerated error, health, legal, financial and safety among them, is a risk appetite call, not an engineering setting. Whether a single similarity threshold is ever acceptable across the business is a policy question, and the research answer is no, because tolerable error moves with the severity of a wrong answer. When to roll back an approximate reuse policy should sit above the team that built the cache.

What it is worth

The clearest, best-evidenced value is latency and compute cost avoided on repeated work: a 2026 measurement on an A100 GPU serving a 7-billion-parameter model found a cached response returned in 32.8 milliseconds against 149.6 milliseconds recomputed. That number will not transfer to another stack, but the shape of the saving, roughly an order of magnitude, is a reasonable planning assumption to test against your own traffic. Cache timing is itself an information channel an attacker can exploit even when a cache is correctly partitioned by tenant, and proposed defenses have so far been tested only in simulation, not live infrastructure.

The cost side is a ratio, not a fixed number. If the harm from a wrong reuse runs about 100 times the cost of recomputing, the tolerable error rate falls below roughly one in 100. Raise the harm to something like 10,000 times recompute cost, plausible for a health, legal or financial decision, and tolerable error falls to about one in 10,000. The same mechanism can be sound for a low-stakes lookup and unacceptable for a coverage decision.

What the evidence does not yet support is a return figure for a full deployment or a guarantee that revocation reaches every store. Budget for recomputation, not savings, until your own traffic says otherwise.