The ruler can create the cliff

A scaling curve looks like a direct portrait of a model. It is not. It is the final trace left by a chain of transformations, and each transformation can change the curve’s shape.

Begin with the familiar regularity. In a controlled regime, average language-model cross-entropy often falls approximately as a power of model size, data or training compute. Kaplan and colleagues reported trends spanning several orders of magnitude.[1] Hoffmann and colleagues then changed the experimental allocation and found that compute-optimal training required much more data relative to parameters than an earlier prescription suggested.[2] The revision was not a failure of scaling-law reasoning. It exposed what such a law is: an empirical compression of a specified regime, not a timeless law of intelligence.

The equation describes loss, not “reasoning”, “agency” or “understanding”. A downstream benchmark adds more machinery. The model emits a distribution over candidate answers. A decoder samples or searches that distribution. A verifier may rank candidates. Finally, a metric converts the selected output into a number. The published score is therefore a composition, not a primitive measurement.

The original literature on emergent abilities defined them behaviourally: an ability is absent in smaller models and present in larger ones, so extrapolation from the smaller systems fails.[3] That definition is useful for discovery, but it does not yet explain a cause. Schaeffer, Miranda and Koyejo showed that, for fixed model outputs, discontinuous or nonlinear metrics can create apparent jumps that disappear under continuous scoring.[4] The result makes one hidden assumption visible: we often assume that the geometry of a score curve mirrors the geometry of the underlying computation.

It need not. Suppose a model’s probability of producing each required step correctly is p, and a task counts as solved only when all m steps are correct. Under a simplifying independence assumption, whole-task success is p^m. The function is continuous, but it becomes steep when m is large. Remove the independence assumption and the exact numbers change. The causal lesson survives: conjunctive success compounds local error.

flowchart TD
  A["The last carry"] --> B["The ruler can create the cliff"]
  B --> C["Where steepness actually enters"]
  C --> D["Test-time compute moves the<br/>boundary"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Where steepness actually enters

“Smooth” and “emergent” are not rival labels for one curve. They answer different questions unless the resource axis, task distribution, inference policy and metric are all fixed.

A metric can erase the path

Imagine two evaluators receive the same probability distribution for a four-option question. Model A assigns the correct answer 0.24, then 0.31, 0.42 and 0.49 as scale rises. The wrong answer remains barely higher until the final checkpoint, where the correct option reaches 0.51. Exact accuracy reports four failures followed by one success. Log score records improvement at every checkpoint. Calibration error may reveal something else again. Each metric answers a legitimate question, but only one has a built-in cliff.

The remedy is not to replace exact success with one supposedly neutral continuous metric. Different rulers answer different operational questions. Exact success asks whether the result may be released. A proper score asks whether probability moved towards the truth. Partial credit asks where the task failed. Calibration asks whether stated confidence can guide reliance. Utility weights false acceptance, abstention, latency and review. Report a small bundle of these measures against the same saved outputs. A transition is metric-stable when its location and item ordering survive reasonable task-valid alternatives, not when every curve has an identical shape. Operational discreteness may be necessary while explanatory discreteness remains unproven.

Task structure can multiply errors

Exact match is especially severe when a task is a conjunction. A proof must close every gap. A program must compile and pass every required test. A tool-using workflow may need the right plan, arguments, permission, execution and readback. Small improvements at every local decision can produce a large improvement in end-to-end completion. The aggregate curve becomes steep because the task asks for all links at once.

Item difficulty can create a population threshold

A benchmark is not one task. It is a distribution of items. Suppose each item has a difficulty d, and a model has latent competence q. The probability of success may be a smooth sigmoid of q − d. If most items cluster around one difficulty band, aggregate accuracy stays near chance until q reaches that band, then rises rapidly. Change the item mix and the apparent emergence point moves. Average benchmark curves are often more predictable than individual tasks precisely because aggregation can smooth idiosyncrasy; individual tasks remain harder to extrapolate.[8]

Sparse checkpoints can invent surprise

A sigmoid sampled at 1, 10, 100 and 1,000 units can look like a step. The same function sampled every tenth of a logarithmic unit looks gradual. This is not a cosmetic concern. Expensive model training encourages sparse observations exactly where the claim concerns a threshold. Training checkpoints from the same run can help, but they change optimisation time rather than parameter count. Small proxy models can help, but they may occupy a different regime. Predictability depends on the coordinate and on whether the observed systems lie on one comparable path.

With four observations, a power law, sigmoid and smoothly broken law may all fit within measurement error while forecasting very different next points. A high in-sample fit therefore cannot identify the generating mechanism. The discriminating evidence is prospective: fit only the smaller systems, publish the forecast interval, then test denser systems that were not used to choose the family. Repeat on several task slices. If the preferred curve changes whenever a new checkpoint arrives, the data support uncertainty about regime, not confidence about a phase transition.

Test-time compute moves the boundary

Scaling no longer ends when training stops. A deployed system can spend extra computation generating candidates, revising them, consulting tools, searching a tree or asking a verifier which path to trust.

Parallel sampling is the simplest case. Let r be the probability that one independent attempt is correct. If a perfect verifier can recognise a correct attempt, the chance that at least one of k attempts succeeds is:

The independence assumption is deliberately generous. If thirty-two samples repeat the same plan with cosmetic wording changes, their effective diversity may be close to one. Shared misconceptions, deterministic tool errors and prompt-induced attractors create correlated failures, so empirical pass@k can saturate far below the formula. Measure overlap in plans, intermediate states and failure causes, not only surface text. Then compare a wider decoder, a prompt that induces genuinely different strategies and a policy that branches after an observed failure. Useful inference width is the number of causally distinct opportunities, not the number of strings purchased.

Brown and colleagues found that repeated sampling increased coverage over large ranges of inference compute. In coding and formal proof, automatic verification could convert coverage into performance. In domains without such verifiers, majority voting and reward-model selection plateaued and failed to capture the full candidate-set gain.[9] This yields a decisive distinction: inference compute buys opportunities; verification determines whether those opportunities become trustworthy outcomes.

Width, depth and adaptation are different interventions

Repeated sampling spends compute on width. Sequential reasoning spends it on one longer trajectory. Search alternates between expansion and selection. Tool use changes the information and action space. These interventions can produce similar score curves through different causes. Snell and colleagues found that the most effective allocation depended strongly on prompt difficulty and that adaptive strategies could improve compute efficiency over fixed best-of-N sampling.[10] This means “test-time compute” is not one scalar treatment. It is a budget distributed through a policy.

The s1 work showed that a small curated dataset and budget forcing could produce test-time gains on selected mathematical benchmarks.[11] DeepSeek-R1 reported reinforcement learning that induced longer reasoning patterns and stronger results on verifiable tasks.[12] Yet forcing a model to continue is not equivalent to giving it useful computation. Later studies found non-monotonic performance, with excessive sequential “thinking” reducing accuracy, and questioned whether some simple scaling plots were mainly created by truncating outputs below their normal length.[13][14]

The verifier can become the bottleneck

Suppose a generator occasionally produces an invalid patch that exploits a weakness in the test suite. One sample may rarely hit the exploit. Ten thousand samples almost certainly will. A verifier that mistakes this exploit for success turns greater search into greater failure. The problem resembles optimisation against a misspecified objective: the search policy finds what the verifier rewards, not what the operator intended.

The economic optimum becomes end to end

Training laws traditionally ask how to allocate a fixed pretraining budget. Deployment adds inference volume, latency and per-query reasoning budgets. Sardana and colleagues showed that high inference demand can favour smaller models trained on more tokens than a Chinchilla-only calculation would suggest.[15] A more recent preprint jointly models model size, training tokens and test-time samples, reporting optima deeper in the heavily trained regime across its studied tasks.[16] The latter result is preliminary, but its framing is important: the relevant resource curve belongs to the complete service, not the training run in isolation.

A stronger test for emergence

The practical question is not whether the word “emergence” is allowed. It is what evidence would distinguish a ruler-created cliff, a search-created cliff and a candidate internal transition.

The metric-sensitivity protocol

The protocol below turns the cliff decomposition into an experiment. It starts from raw candidate evidence and adds transformations one at a time. The order matters. If the benchmark retains only a final exact score, most causal information has already been destroyed.

Before running the sweep, write a predicted signature for each possible address. A metric-created cliff should move when rescored while raw candidate probabilities remain fixed. A task-mixture cliff should move when the item distribution is reweighted. An inference-created cliff should move with sample budget, stopping policy or verifier while one-sample candidate statistics remain stable. A candidate mechanism transition should remain tied to a narrow training interval under those downstream interventions and should change several independently measured behaviours. Record the null for every stage. Otherwise a flexible analysis can relocate the cliff after seeing the data.

Curve fitting comes after experimental control, not before it. Compare a simple power law, a sigmoid and a smoothly broken form. Fit on early scales and evaluate on held-out larger scales. Report predictive intervals, not only the best in-sample line. If several families fit the observed range but diverge beyond it, the correct forecast is a range of decisions. A single smooth fit should not be used to erase a real break, and a flexible broken fit should not be used to discover one in noise.

from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path
from typing import Iterable
import csv
import math

import matplotlib.pyplot as plt
import numpy as np


@dataclass(frozen=True)
class LabConfig:
    required_steps: int = 8
    samples: int = 16
    slope: float = 1.40
    midpoint: float = 2.50
    floor: float = 0.45
    ceiling: float = 0.99
    points: int = 201

    def validate(self) -> None:
        if self.required_steps < 1:
            raise ValueError("required_steps must be at least 1")
        if self.samples < 1:
            raise ValueError("samples must be at least 1")
        if not 0 <= self.floor < self.ceiling <= 1:
            raise ValueError("require 0 <= floor < ceiling <= 1")
        if self.points < 20:
            raise ValueError("points must be at least 20")


def latent_step_success(log_compute: np.ndarray, cfg: LabConfig) -> np.ndarray:
    """Smooth local competence. This is illustrative, not a fitted LM law."""
    span = cfg.ceiling - cfg.floor
    return cfg.floor + span / (1.0 + np.exp(-cfg.slope * (log_compute - cfg.midpoint)))


def exact_task_success(per_step: np.ndarray, required_steps: int) -> np.ndarray:
    """All steps must be correct; conditional independence is assumed."""
    return np.power(per_step, required_steps)


def pass_at_k(single_attempt: np.ndarray, samples: int) -> np.ndarray:
    """At least one correct candidate; assumes independent samples and perfect selection."""
    return 1.0 - np.power(1.0 - single_attempt, samples)


def first_crossing(x: np.ndarray, y: np.ndarray, threshold: float = 0.5) -> float | None:
    hits = np.flatnonzero(y >= threshold)
    return None if len(hits) == 0 else float(x[hits[0]])


def write_csv(path: Path, rows: Iterable[tuple[float, float, float, float]]) -> None:
    with path.open("w", newline="", encoding="utf-8") as handle:
        writer = csv.writer(handle)
        writer.writerow(["log10_compute", "per_step", "exact_one", "pass_at_k"])
        writer.writerows(rows)


def main() -> None:
    cfg = LabConfig()
    cfg.validate()
    x = np.linspace(0.0, 5.0, cfg.points)
    per_step = latent_step_success(x, cfg)
    exact = exact_task_success(per_step, cfg.required_steps)
    coverage = pass_at_k(exact, cfg.samples)

    write_csv(Path("metric_sensitivity.csv"), zip(x, per_step, exact, coverage))

    exact_cross = first_crossing(x, exact)
    pass_cross = first_crossing(x, coverage)
    print(f"50% exact crossing: {exact_cross}")
    print(f"50% pass@{cfg.samples} crossing: {pass_cross}")

    fig, ax = plt.subplots(figsize=(9, 5.2))
    ax.plot(x, per_step, label="per-step success")
    ax.plot(x, exact, label="exact task, one attempt")
    ax.plot(x, coverage, label=f"pass@{cfg.samples}, perfect verifier")
    ax.axhline(0.5, linewidth=1, linestyle="--", label="50% operating threshold")
    ax.set(xlabel="log10 training compute (illustrative)", ylabel="probability", ylim=(0, 1))
    ax.legend()
    fig.tight_layout()
    fig.savefig("metric_sensitivity.png", dpi=180)


if __name__ == "__main__":
    main()

Classify the cliff before naming it

The final instrument combines metric stability with causal localisation. Metric stability asks whether the shape persists under continuous, exact and utility-weighted scores. Localisation asks whether an intervention identifies a change before the metric and selection stages. This yields four different research actions.

A positive audit does not require the curve to become smooth. It requires the source of steepness to become identifiable. A metric-created cliff can still matter in production, because a proof must verify and a payment must either execute or not. A search-created cliff can be a valuable engineering result, because configured systems are what users encounter. A mechanism-level cliff can motivate interpretability work. The mistake is not using thresholds. The mistake is allowing a threshold at one level to impersonate an explanation at another.

stateDiagram-v2
  [*] --> Claim
  state "A metric can erase the path" as Claim
  state "Task structure can multiply errors" as Test
  state "Item difficulty can create a<br/>population threshold" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]