State-Space and Recurrent Models: Audit What They Really Keep

State-space, recurrent and hybrid sequence architectures replace standard attention to process long sequences without memory and compute growing with context length. That promise holds for the narrow recurrence at the center of these designs and routinely breaks for the complete system built around it: a design can have a fixed-size recurrent state and still carry an archive of stored history that grows exactly like the attention cache it was meant to replace. Replacing attention requires an audit of every tensor a system actually keeps, not the label on the paper.

What it is

These architectures carry a state forward and update it as each input arrives, rather than scanning the full history at every step. Three splits matter. State updates use a fixed transition learned once in training, or an input-dependent one gated by what just arrived. State shape is vector-valued or matrix-valued, which sets how much can be held before information collides. And designs are pure recurrence, hybrids that alternate attention and recurrence across layers, or a compact recurrence paired with attention over a stored archive of past summaries. Continuous-time state-space notation is not automatically exact: discretization preserves stability only for a specified linear system with negative transition eigenvalues and a positive step size. Outside that setting it is a parameterization, not a guarantee.

Attention's cost grows with context length, and every architecture here is sold on relief from that growth. Getting the accounting wrong means paying twice, once for the switch and again when the same memory profile reappears under a different name. A 2026 design called DART pairs a native recurrence with attention over an archive of stored per-chunk state increments that grows with chunks processed, so its retained state is not constant even though its recurrence is, evidence graded B: empirical, not independently reproduced. A separate August 2026 comparison found a 64-token window plus four sink tokens, with no extra training, retained roughly 99 percent of a teacher model's average score across six benchmarks, also graded B, an aggregate ratio, not a universal claim. The evidence base behind this essay is three papers, not a saturated literature.

The architecture

The discipline I use to evaluate every design in this family is what I call the retained-byte audit: count every tensor the system actually keeps, at deployment shapes, then ask whether the result clears a joint bar on quality, recall, latency and memory together.

The audit counts the native recurrent state, any per-chunk archive increments, projected keys and values kept as an explicit cache, prefix or sink tokens retained for exact local access, and temporary buffers used during parallel training scans, all at the actual dtype, batch size and sequence length a deployment will run. A design's published complexity class describes its recurrence, not its archive.

A design that clears the audit lands on a spectrum: pure recurrence bounds decode memory tightly at the cost of interference and a hard limit on exact recall, a chunk-state archive extends retrieval range but grows with sequence length by construction, alternating attention and recurrence gives a flexible quality-memory tradeoff while any attention cache layer still grows as it always has, a sliding window gives exact access to recent tokens while distant detail is lost, and full attention keeps the broadest access at the cost of memory and compute that scale with length. None is categorically superior: each is a different point on the tradeoff, and the audit tells a buyer which point a design occupies.

Two failure modes sit underneath all five. A fixed-size state has finitely many representable configurations, so it cannot losslessly distinguish an arbitrarily large number of independent histories, though semantic compression can stay useful past that point. Sequence boundaries also need explicit resets: a parallel scan can compose values in a different floating-point order than the serial recurrence it should match, and a packed batch of documents can leak one document's state into another's if a lane-level reset is missing.

How to lead it

Ownership sits with whoever is accountable for production inference cost and outage risk, under one shared contract with whoever validates exact-recall correctness on the workloads that need it. Fund the retained-byte audit as a standing gate before any such architecture ships, at equal state bytes, compute and latency against three baselines: pure recurrence, a byte-matched explicit cache, and a same-size archive of simpler summaries. Fund adversarial evaluation covering copying, counting, order swaps, repeated keys, code dependencies, rare facts, and long-distractor contradictions, with exact retrieval scored separately from semantic answering. Stop trusting a constant-memory label without the audit behind it, and stop treating scratch-trained and converted-from-attention results as the same kind of evidence, since a converted model carries the inductive biases of the architecture it came from.

Two calls belong to the executive alone: whether a workload requires exact retained records, which keeps a fallback in place regardless of architecture, and setting the joint bar itself, the quality, recall, latency and memory numbers a design must clear together before it replaces attention rather than supplements it, since no study here establishes one universal winner across model scale, conversion regime and task family.

What it is worth

Money, risk and time show up in inference cost and latency at long context, in conversion or retraining cost, and in the risk of an exact-recall failure the business assumed the new architecture could still handle. Measure before and after using the same three fixed baselines every time, not only against the attention setup being replaced, and track the sliding-window result as what it is, a 99 percent aggregate retention ratio on six benchmarks, not a guarantee for a from-scratch decision.

The evidence supports recurrence paired with selective explicit memory as a useful tradeoff, and that a compact native recurrence can still carry a growing archive, so compactness at the core is not proof of cheapness overall. It does not support a single best architecture at frontier scale, a proven way to guarantee constant memory without loss, or any specific return figure.