Scaling laws are the most successful empirical result in modern machine learning and the most over-extrapolated. The core finding (cross-entropy loss falls as a smooth power law in parameters, data. Compute) has survived every replication since 2020, and Chinchilla's correction to the compute-optimal ratio was a genuine improvement, not a refutation. I don't dispute the curves. I dispute what people read into them.
A scaling law is a statement about next-token loss on a held-out distribution. Everything anyone actually cares about (can it debug a distributed system, will it fabricate a citation, does it fold into a workflow) sits several noisy transformations away from that number. The interesting action in 2025–2026 has been almost entirely in the gap between the curve and the capability. Four blind spots, in order of how much they've bitten.
The data quality wall
The raw-quantity story first: credible estimates put the stock of high-quality public text in the low tens of trillions of tokens, and frontier training runs have been bumping against it for two years. But the deeper issue is that tokens are not fungible, and the laws have no term for that. The strongest evidence is indirect and consistent:
- Data-curation interventions move the curve itself. Filtering, deduplication, and quality-weighting deliver improvements equivalent to multiples of raw data: the same loss at a fraction of the tokens. If token selection changes the constants of the law, the law was never really about token count.
- Small-model results made it undeniable. The Phi-lineage and its successors showed curated and synthetic "textbook-quality" data producing models that punch far above their parameter class on reasoning benchmarks. Read either as data quality mattering enormously or as benchmark proximity in disguise: both readings damage naive scaling extrapolation.
- Synthetic data works, conditionally. Where a verifier exists (math, code), model-generated data is now a standard, load-bearing ingredient. Where none exists, iterative self-training shows the drift and diversity-collapse failure modes the model-collapse literature predicted. Synthetic data is a verified-domains amplifier, not a general escape from the wall.
The operational consequence: the moat has moved from compute to data pipelines: and, for enterprises, to proprietary corpora that never appear in anyone's pretraining mix.
Post-training is where the product lives
Chinchilla-style accounting assigns essentially all capability to pretraining. Yet look at what changed user-visible performance since 2023: instruction tuning, RLHF and its cheaper successors like DPO. Above all RL on verifiable rewards, which took competition math and coding from mediocre to superhuman-tier using a rounding error of pretraining compute: DeepSeek-R1 made the cost asymmetry public and reproducible in early 2025.
Two facts sit awkwardly with curve-worship:
- Pretraining loss barely moves during post-training while benchmark performance moves enormously. Whatever the loss curve measures, it is not the thing post-training improves. The most parsimonious story: pretraining builds a capability manifold; post-training navigates it. Elicitation studies (base models matching RL-tuned models at high sampling k) support this more than the "new capabilities" story does.
- The returns per FLOP are wildly lopsided. When a few percent of total compute produces the majority of the perceived quality delta between model generations, "just scale pretraining" stops being a strategy and becomes a habit.
The equilibrium view I hold: pretraining sets the ceiling, post-training sets the floor you actually experience, and scaling laws only describe the first. Buyers comparing models by parameter count are reading the wrong spec sheet.
Architecture bets beyond the transformer
Scaling laws are architecture-conditional (every published curve is a curve for transformers) and the field routinely forgets the conditional. Through 2024–2026 the challengers earned real, bounded respect:
- State-space models (Mamba lineage) and hybrid attention-SSM stacks now match transformer quality at small-to-mid scale with linear-time inference, and hybrids have shipped in production models. The honest reading of head-to-head studies: SSMs win on throughput and long-sequence cost, lose on precise in-context recall and copying, and hybrids buy most of both: which is why nearly every "post-transformer" deployment is actually a transformer-plus.
- Mixture-of-experts stopped being an architecture bet and became bookkeeping. Sparse models decoupled parameters from FLOPs so thoroughly that "model size" is no longer one number, and naive parameter-based scaling comparisons across dense and sparse models are now category errors: a fact conveniently absent from most public model marketing.
- The exotic bets (diffusion LMs, recurrent-depth reasoning, memory-augmented architectures) remain research options, interesting precisely because inference economics, not loss, is where transformers hurt. Nobody has beaten the transformer on quality at frontier scale. Plenty have beaten it on cost per token served.
The strategic point: when capability gains per pretraining dollar flatten, architecture becomes an economics play before it becomes a capability play. That's the door SSMs and sparsity came through. It's the door the next thing will come through too.
Evaluation ceilings
The quietest problem is that our ability to measure capability has scaled far slower than capability itself, which means claims on both sides of the scaling debate are increasingly unfalsifiable.
The specifics are well documented and worth stating bluntly:
- Benchmark saturation and contamination are endemic. MMLU, GSM8K, HumanEval (the canon that scaling narratives were written in) are saturated, partially leaked into training corpora, or both. Contamination studies keep finding models acing canonical items and stumbling on freshly rephrased ones. Every headline scaling-produces-capability plot built on these axes inherits that noise.
- Emergence was partly a measurement artifact. The influential "mirage" line of work showed that many celebrated discontinuous jumps flatten into smooth curves under continuous metrics. Scaling looks lawful or magical depending on your choice of y-axis, which should unsettle anyone extrapolating either way.
- The tasks that matter resist scoring. Long-horizon agentic work, research judgment, real software maintenance: the frontier evals (SWE-bench variants, agentic suites, expert-written exams like HLE) are better, and they're also expensive, quickly gamed, and saturating on cycles measured in months. Meanwhile LLM-as-judge, the field's scaling answer to evaluation, imports the judged models' own biases (verbosity preference, self-preference, position effects) into the measurement instrument itself.
Here's the uncomfortable syllogism: scaling laws predict loss; we justify the spend by capability; capability is measured by evals, and the evals are the least rigorous link in the chain. The ceiling on what we can measure is becoming the ceiling on what we can credibly claim: and, for anyone allocating capital, on what they can credibly buy.
Turn a scaling forecast into a decision stack
An enterprise does not buy lower pretraining loss. It buys a task outcome under a service, risk and cost envelope. The investment argument must therefore cross several boundaries that a scaling curve does not contain. A plausible capability must transfer to the target population. The model must operate with the institution's data and tools. The resulting system must fit response-time, resilience and evidence requirements. The avoided work or risk must exceed total operating cost.
I separate those claims because each can fail independently. A model may improve on a fresh reasoning set and still lose in production because retrieval dominates the error. A fine-tune may improve average classification while degrading a small high-risk segment. A sparse architecture may reduce inference cost but require an operational stack the team cannot yet support. The curve is upstream evidence, not a downstream business case.
This decision stack is deliberately asymmetrical. Strong upstream evidence does not waive a downstream test. Poor upstream evidence can, however, stop an expensive programme early. If the model does not improve a fresh target-task set, there is little reason to build the complete platform merely to discover the same fact under load.
Data is a portfolio with rights and decay
Calling enterprise data a moat hides the work. Useful training or retrieval data needs ownership, lawful use, quality, lineage, temporal validity and a link to the target task. Those properties differ by source. Customer communications may be recent and behaviourally rich but restricted. Policy documents may be authoritative but version-sensitive. Historical decisions may contain valuable outcomes and inherited bias. Synthetic examples may cover rare cases but reflect the generator's blind spots.
A data portfolio should make those trade-offs visible. Assign each source a permitted use, quality tier, freshness rule and exclusion path. Record whether it supports pretraining, supervised tuning, preference or reward learning, retrieval, evaluation or none. Training rights do not imply evaluation independence. A case used for tuning cannot provide unbiased evidence of generalisation.
The most valuable data work often reduces model demand. Clean identifiers, explicit relationships and current policy metadata improve retrieval and deterministic validation. They can outperform another model increment on the workflow that matters. Scaling analysis rarely captures that alternative because it treats the model as the system and the data as training fuel.
Post-training needs its own return model
Post-training choices have different economics and evidence. Supervised fine-tuning teaches a target response distribution. Preference optimization shifts relative behaviour under the chosen pairs or feedback. Reinforcement learning with verifiable rewards can discover useful procedures when outcomes are checkable. Distillation transfers behaviour into a cheaper model but may lose the difficult tail. These are not interchangeable “customisation.”
| Intervention | Best fit | Primary failure risk | Decisive enterprise evidence |
|---|---|---|---|
| supervised fine-tuning | stable task format and labelled exemplars | memorising style or label artefacts | fresh slice performance and calibration |
| preference optimization | repeated trade-offs with reliable comparisons | reward or annotator preference becomes the product goal | blinded preference plus task and safety checks |
| verifiable-reward training | outcomes checked by code, rule or exact state | specification gaming and narrow transfer | held-out verified tasks and exploit tests |
| distillation | high-volume bounded tasks | weak tail and copied teacher errors | matched-quality cost curve by difficulty |
| prompt and retrieval change | knowledge or instruction failure | brittle context and hidden regressions | retrieval, answer and provenance ablation |
Measure the simplest intervention first. If retrieval repair captures most of the gain, tuning adds model lifecycle cost without enough value. If a deterministic calculation removes an error class, reinforcement learning is solving the wrong problem. The comparison must include change control, monitoring and rollback, not only training compute.
Architecture choices are serving choices
The architecture debate becomes concrete at inference. A long-context transformer may simplify application code while raising per-request memory and token cost. A state-space or hybrid design may improve throughput but weaken exact recall on a task that depends on identifiers. A mixture-of-experts model may offer a favourable active-compute profile while introducing routing behaviour and deployment constraints. The institution needs matched task quality before comparing infrastructure.
Tokens per second is not the denominator. Use successfully verified tasks, including retrieval, retries, batching delays, cache behaviour and verifier calls. Report the tail. A service that is cheap at average load and unstable at the close-of-business peak is not cheap in any operationally useful sense.
Evaluation as a managed asset
A fresh evaluation set loses value as teams tune against it. Treat it like a controlled asset with versions and exposure records. Keep a development set for daily iteration, a release set with restricted access and a rotating set of new cases. Add incidents, overrides and policy changes without turning the set into a museum of only extreme failures.
The scoring design should reflect the decision chain. Separate evidence retrieval, factual support, calculations, policy interpretation, tool choice, authority and final outcome. This produces a failure signature rather than one opaque score. Human ratings remain necessary for judgement-heavy work, but calibrate raters against adjudicated anchors and record disagreement.
Contamination is not only a public-benchmark problem. Internal leakage occurs when evaluation cases enter prompt examples, retrieval stores or tuning corpora. Store case identifiers and hashes in the data lineage so the release process can detect overlap. A private set is not independent merely because it is private.
A board-level decision matrix
The investment question is not “do we believe in scaling?” It is which uncertainty should be bought down next. If capability on the target task is unknown, fund evaluation. If capability is adequate but evidence retrieval is weak, fund context engineering. If quality is adequate but unit economics fail, fund routing, distillation, caching or architecture experiments. If the system performs but authority and recovery are weak, fund production control before wider autonomy.
This ordering protects capital. A frontier-model contract cannot repair an unevaluated workflow. A tuning programme cannot repair missing data rights. A serving optimization cannot justify a system whose residual error has no owner. Spend should follow the nearest unresolved decision, not the loudest research trend.
A worked portfolio choice
Suppose a bank is considering three investments for a document-heavy credit workflow. Option one upgrades every request to a larger frontier model. Option two repairs document classification, versioning and retrieval before keeping the existing model mix. Option three distils a bounded extraction and policy-mapping task into a smaller model while retaining the larger model for the ambiguous residual.
The first option has the cleanest procurement narrative and the weakest diagnosis. It may improve interpretation, but it spends on every case and leaves missing or stale evidence untouched. The second attacks the observed failure if trace analysis shows that relevant clauses or current financials are absent from the context. The third addresses unit economics if the task is already well specified and most cases are repetitive.
The discovery phase should quantify the error stack before choosing. Sample production-like cases and attribute each failure to evidence ingestion, retrieval, extraction, calculation, policy mapping, unsupported generation or action. Run the larger model with the same context, then with oracle context. If the gain appears only with oracle context, model scale was not the first constraint. If both models succeed with oracle context but the smaller model fails on the live bundle, retrieval may interact with model robustness. If the smaller model matches on routine slices, distillation or routing becomes credible.
Build three cost views at the same target quality. The model-upgrade view includes added input and output inference across the full volume. The context view includes ingestion, metadata, indexing, reranking and source stewardship. The distillation view includes dataset construction, training, validation, monitoring and the retained escalation path. Include human review and incident cost in all three. Comparing vendor rate cards alone biases the decision toward whichever cost surface is least visible.
The likely answer is a sequence rather than one option. Repair context until the system can distinguish model error from evidence error. Route a smaller or distilled model across the stable majority. Spend frontier inference and extended reasoning on the residual where measured consequence and difficulty justify it. The scaling curve still matters for that residual, but it no longer dictates the architecture of every request.
Governance of the forecast
Forecasts should be versioned artefacts with named assumptions. Record the source curve, model family, training regime, data distribution, target metric and extrapolation distance. Then list the transformations required to reach the business outcome: task transfer, system integration, human adoption and financial realization. Assign an evidence rating to each step.
Avoid a single probability of success. Separate technical feasibility, operational feasibility and benefit realization. A technically stronger model may be straightforward to test and difficult to integrate because data cannot leave a boundary. A working system may fail to realize benefit because downstream reviewers repeat all the work. Those are different risks with different owners.
Every forecast needs falsification conditions. Examples include no material improvement on the fresh target set, gain disappearing under matched cost, a material risk slice regressing, p95 latency exceeding the channel ceiling or the residual manual effort remaining unchanged. A stop condition is not pessimism. It is the mechanism that makes a learning investment bounded.
The governance forum should update beliefs from evidence rather than defend the original case. If data repair moves the task curve, shift money from model scale to context. If a new model changes residual performance, reopen routing. If evaluation saturates, fund new cases before claiming continued improvement. A forecast earns credibility by changing when the measured constraint changes.
What to ask a vendor or internal model team
Ask for a matched-quality comparison, not the largest benchmark table. Which target-like tasks improved, with what confidence interval and contamination controls? What input and output usage produced the result? How does performance change by document length, language, tool count and interaction horizon? Which behaviours came from post-training rather than pretraining? Which safety or refusal trade-offs moved?
For architecture, ask for active parameters, memory footprint, batching assumptions, cache behavior, hardware and tail latency at the relevant sequence lengths. For data, ask what categories and time periods are represented without requesting protected training details the provider cannot share. For evaluation, ask how often benchmark items were refreshed and how judge models were calibrated.
The answers will remain incomplete. The objective is not to force a provider to prove the enterprise case. It is to identify which claims can be carried from provider evidence and which must be established locally. The institution owns the last mile from model capability to customer, financial and risk outcome.
The decision i would put in front of an executive committee
State the constraint first. For example: “The workflow fails mainly because current policy clauses are not retrieved consistently; model interpretation is the second-largest error.” Then present the options against the same outcome, total cost and risk criteria. Show which evidence is observed, modelled or still unknown. Recommend the smallest investment that resolves the nearest constraint and preserves a path to scale.
Do not put parameter count or generic model rank at the centre of the page. Put cost per successfully verified case, residual material error, time returned to the business and control readiness there. Link the scaling evidence as an input to the capability assumption. This keeps an important research result in its proper place: informing one part of a system decision rather than impersonating the entire decision.
What i actually take from the curves
None of this is a scaling-denial argument. Loss keeps falling on schedule; pretraining scale remains the price of admission; the labs that stopped scaling stopped mattering. But the power law is the entry fee, not the strategy. The compounding advantages of 2026 live in the terms the law doesn't carry: what you train on, what you do after pretraining, what you pay per token served. Whether you can measure any of it well enough to steer.
My operating heuristic: treat scaling laws the way a CFO treats revenue growth: real, necessary, and never the whole model. The curve tells you the tide is rising. It tells you nothing about which boats are seaworthy, and it has never once told you where the rocks are.
From loss curve to investment decision
| Blind spot | What the loss curve omits | Evidence needed before investment |
|---|---|---|
| data quality | provenance, duplication and task relevance | ablation by source and quality tier |
| post-training | preference shaping and task adaptation | pre/post comparison on fresh slices |
| architecture | memory, latency and active-parameter economics | matched-quality serving benchmark |
| evaluation | contamination, saturation and judge bias | private set, human calibration and confidence intervals |
| Claim | What can support it | What cannot support it alone |
|---|---|---|
| training became more efficient | compute-optimal loss curve | a single downstream benchmark |
| product improved | representative task and incident measures | parameter count |
| architecture is cheaper | matched-quality total serving cost | tokens per second alone |
| capability generalises | fresh multi-domain evaluation | saturated public leaderboards |
The primary references are the original neural scaling laws, the Chinchilla compute-optimal analysis, the emergence measurement critique, Mamba and research on training with synthetic data. The curves remain useful within their measured domain. Capability, cost and control still require separate evidence.