The Three-Ledger Method for Model Architecture Decisions
Architecture, pretraining objective and scaling bets need a three-ledger test that separates measured runs from fitted curves and extrapolated projections.
Yash Sharma4 min readModel architecture and training
Research area 13: Model-architecture, pretraining-objective and scaling engineering
In brief
- Sort every architecture, objective and scaling claim onto three ledgers before funding it: measured, fitted or extrapolated.
- The three-ledger method separates completed runs from fitted curves and unproven extrapolations before any bet is funded.
- Ignoring the ledgers means paying for extrapolated cluster bets and misattributed gains that ablations would have caught.
Model-architecture, pretraining-objective and scaling engineering decides what a model can represent, how it trains, and what a cluster budget actually buys before a dollar of compute is spent. Arguments over which architecture wins mix three kinds of claim: what has been measured, what a fitted curve predicts inside its tested range, and what someone has projected far past that range onto a bigger cluster. An executive who cannot tell those apart ends up funding a hypothesis and calling it a plan.
What it is
The area covers three linked choices. Architecture is the mechanism a model uses over tokens: a dense transformer, a sparse mixture-of-experts model routing each token to a subset of its parameters, a recurrent or state-space model compressing history into a fixed state, or a diffusion model revising a sequence rather than producing it left to right. Pretraining objective is what the model is optimized to predict: the next token in order, a masked observation, or several future tokens at once. Scaling is the empirical relationship between parameters, data and compute, and what it does and does not license past the range it was measured on.
Two principles hold. Families are not fixed products: hybrids mix attention, recurrence, routing and auxiliary objectives, and novelty should mean a changed equation, not a renamed component. The workload decides the architecture, not an aggregate loss number, and active and total parameters are different budgets.
Why it matters now
Three 2026 studies show the architecture question is no longer settled by one benchmark table. Sahoo and colleagues (February 16, 2026) compared generation-objective families, including a diffusion-family model called Duo, under a matched SlimPajama data set and tokenizer, across budgets from roughly six quintillion to one hundred quintillion floating-point operations: Duo came out ahead on a specific task after tuning, but the study cautions that likelihood bounds across objective families are not directly comparable.
Sarkar, Tang and Zha (August 11, 2026) fit roughly 150 runs spanning 104 million to 2.7 billion active parameters for sparse mixture-of-experts models: accounting for communication changes which configuration is fastest. Their headline recommendation, a 256-processor cluster running twenty days, sits roughly 150 times beyond the largest budget actually trained: a prediction, not a demonstrated run. Pandey (August 22, 2026) found that increasing experts in a mixture-of-experts state-space model worsened specific dynamical behavior in synthetic tasks, even under oracle routing. Treating either as a purchase order or a verdict on a whole model family commits budget to a claim the evidence does not support.
The architecture
I call this discipline the three-ledger method: every claim about an architecture, objective or scaling result belongs on one of three ledgers, and treating it as though it sits on a stronger ledger than the evidence supports is the most expensive mistake teams make here.
The first ledger is measured: numbers from runs that actually happened, on stated hardware. The second is fitted: values read from a scaling curve inside the tested range, trustworthy as interpolation and nothing more. The third is extrapolated: any operating point projected onto a bigger budget or an unbuilt configuration. The failure is moving a number from the third ledger to the first without saying so.
Before crediting an architecture change for a benchmark gain, hold data order, tokenizer and optimizer fixed, so the gain is attributed to the mechanism, not the whole recipe changing at once. Before committing a cluster allocation to a compute-optimal configuration, ask which ledger produced it, and validate an extrapolated one at an intermediate scale before the full budget moves. A synthetic task falsifies that task, not an entire family.
How to lead it
Ownership sits with a named architecture and scaling owner, not whichever team is training that quarter. That owner keeps a decision record for every model: workload, architecture family, parameter counts, training objective, hardware layout, and where the extrapolation boundary sits.
Fund matched-recipe ablations before crediting an architecture change, an intermediate-scale validation run before any extrapolated configuration gets full budget, and profiling of memory, communication and latency alongside loss. Stop funding a cluster commitment whose central number sits only on the extrapolated ledger, and stop letting a narrow negative result become a rule against an entire family.
Two decisions belong to an executive alone: how far past a fitted range the company will bet real budget, a statement of risk appetite rather than a curve-fit output, and when a forecast must be re-reviewed, such as after a hardware or data change, rather than assumed to still hold.
What it is worth
The money, risk and time show up in three places: cluster budget chasing an extrapolated optimum an intermediate-scale run would have caught, engineering time re-explaining a gain an ablation would have attributed correctly the first time, and serving costs that surprise a team when a lower-loss objective needs more inference passes than expected.
Measure before and after with the yardstick the evidence uses: report predicted and achieved numbers in separate tables, and rerun any comparison whenever hardware or data changes.
The 2026 evidence supports that generation-objective families behave differently enough to matter for specific tasks, that a systems-aware view changes what counts as an optimal sparse-model configuration, and that more experts can worsen a dynamical property in a bounded synthetic setting. It does not support a single best architecture, a general ranking across workloads, or any return figure.
Questions leaders ask
- What is the three-ledger method?
- It is a discipline for sorting every claim about a model's architecture, objective or scaling behavior onto one of three ledgers. Measured holds numbers from completed training runs. Fitted holds values read from a scaling curve inside the range it was actually tested on. Extrapolated holds any operating point projected onto a bigger budget or a different cluster. Fund only the first two ledgers as though they were proven.
- Should mixture-of-experts models be the default choice for a new large model?
- Not automatically. A 2026 systems-aware scaling study fit roughly 150 training runs and found that accounting for communication changes which configuration is actually fastest, not just which has the fewest active parameters. Its headline recommendation for a 256-processor cluster running twenty days sits about 150 times beyond the largest budget it actually trained, so treat that number as a hypothesis to validate at an intermediate scale, not a configuration to purchase directly.
- Do diffusion or masked-generation language models outperform autoregressive models?
- The clearest 2026 comparison trained generation-objective families under a matched data set, tokenizer and compute budget and found a diffusion-family model ahead on one task after supervised tuning. That is a real result, not a general verdict. The comparison's own authors caution that likelihood bounds across different objective families are not directly comparable, so a lower reported loss under one objective does not automatically mean better generation quality or lower serving cost.
- Does adding more experts always improve a model?
- No. A 2026 study found that increasing the number of experts worsened specific dynamical behavior in synthetic tasks built to test that property, even when routing was handled by an oracle rather than a learned router. That result is narrow and confined to synthetic dynamics, not a demonstrated failure across all mixture-of-experts language models, and it has abstract-level coverage only. Treat it as a reason to test dynamical properties directly, not as a blanket rule against more experts.
- Who inside a company should decide which model architecture to fund?
- A named architecture and scaling owner should run the ledger discipline day to day, but two calls belong to an executive alone. One is how far past a fitted scaling range the company is willing to bet real budget, since that is a risk-appetite decision, not a statistical output. The other is deciding when a scaling forecast must be re-reviewed, such as after a hardware, data or workload change, rather than assumed to still hold.
