The pass mark and the pupil: which emergent abilities survive continuous-metric reanalysis?: Rajesh Ranjan Mahapatra

The scene

A typing institute and a pass mark

Mrs. Kamala Hegde runs a small typewriting institute above a sari shop in Hubballi. Her pupils prepare for a government typing test. The rule of the test is strict: the candidate types a passage of ten words, and passes only if every word is correct. One wrong word, and the attempt counts as a fail. Mrs. Hegde keeps a register of every practice test, sorted by how many months each pupil has trained.

Her register shows something that puzzles new parents. Pupils in their first four months almost never pass. Then, somewhere around the fifth and sixth months, the pass rate climbs steeply, from almost nothing to more than half. Parents ask her what happens in month five. Mrs. Hegde has a ready answer, because she also keeps a second column. In that column she records how many of the ten words each pupil typed correctly. That number rises steadily from the first week. Nothing special happens in month five. The pass mark simply starts to notice what was already improving.

One pupil is different. Imran typed with two fingers for four months, and his word accuracy barely moved. Then he learned to touch-type over a single fortnight, and both columns jumped together. His case is the one the parents expected for everybody.

Mrs. Hegde, Imran and the institute are invented, and I will flag them whenever they return. The arithmetic of her register is not invented. It is the arithmetic behind one of the liveliest arguments in recent machine learning, about whether large language models acquire abilities suddenly as they grow.

Figure 1. Two columns in one register. Mrs. Kamala Hegde, her institute in Hubballi and her pupils are invented. The pass column stands for exact-match scores; the second column for a continuous score such as per-token accuracy. illustrative

The topic question for this paper is: which emergent abilities survive continuous-metric reanalysis? An ability is called emergent when small models lack it and larger models have it, with a sharp change somewhere in between. A continuous-metric reanalysis rescores the same model outputs with a measure that gives partial credit, such as the share of correct digits, instead of a pass-or-fail mark. If the sharp change disappears under that rescoring, the emergence was a property of the pass mark. If it remains, something in the model changed abruptly.

My answer, from the literature and from three small instruments I built, is that most of the famous jumps soften under continuous rescoring, and some of them vanish. But not all of them. Some jumps survive because the model really does acquire a component in one step, as Imran did. Others survive only at a finer grain: the average climbs in a staircase, gentler than the pass mark, while each part of the skill switches on sharply at its own scale. Rescoring tells you which kind of jump you have; it does not abolish jumps.

Part 1

What the emergence claim says

Since about 2020, the most reliable fact about language models has been that they get better as they get bigger. Kaplan and colleagues at OpenAI showed that the error of a language model on its training objective falls as a smooth power law in model size, data and compute (Kaplan et al., 2020). Plotted on logarithmic axes, the curves are close to straight lines over several orders of magnitude. Hoffmann and colleagues at DeepMind later refined how to balance model size against data, and the smoothness survived the refinement (Hoffmann et al., 2022).

The smoothness of that one number, the training loss, made a second observation all the more striking. On particular tasks, performance did not improve smoothly at all. A team at Anthropic described the combination plainly: these models have predictable loss on a broad distribution and unpredictable specific capabilities (Ganguli et al., 2022). The loss fell gently, but the ability to do three-digit arithmetic, or to unscramble a word, seemed to arrive all at once.

The definition by Wei and colleagues

In 2022 Jason Wei and fifteen co-authors gave this second observation a name and a definition. An ability is emergent, they wrote, if it is not present in smaller models but is present in larger models (Wei et al., 2022). They drew a consequence from the definition: emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. They then collected many examples from published evaluations, where a plot of task score against model scale lay flat near chance and then turned sharply upwards.

The definition has two parts that are easy to run together. The first is sharpness: the change from absent to present happens over a narrow range of scale. The second is unpredictability: nothing in the smaller models announces the change in advance. A curve can be sharp and still predictable, if one knows its formula. A curve can be gentle and still surprising, if one has not measured the right thing. The argument that followed turned on how these two parts relate to the choice of score.

The evidence came largely from the Beyond the Imitation Game benchmark, known as BIG-bench. It is one of the most remarkable collaborative efforts in the field: 204 tasks contributed by 450 authors across 132 institutions, covering linguistics, mathematics, common-sense reasoning, biology, social bias and more (Srivastava et al., 2022). Its authors evaluated model families spanning millions to hundreds of billions of parameters. On some tasks the families improved gradually. On others they found what they called breakthrough behaviour, a sudden rise past a certain size.

Why the claim mattered

The emergence claim was not only an academic curiosity. If new abilities appear suddenly and unpredictably, then a laboratory cannot know what its next, larger model will be able to do until it has trained it. That is uncomfortable for planning, for safety testing and for regulation. The Anthropic paper drew exactly this conclusion: the unpredictable qualities make it difficult to anticipate the consequences of deployment (Ganguli et al., 2022).

The claim also carried a philosophical charge. "Emergence" is an old word in science, used for properties of a whole that its parts do not have, such as the wetness of water or the flocking of birds. To say that a language model's ability to do arithmetic is emergent sounds like saying that something new has come into being. Whether anything new has come into being is a question that Indian philosophers argued about for more than a thousand years, in a debate I will come to in Part 6.

Figure 2. From smooth laws to sudden skills, and back to the question of what emergence means. Dates are first public versions. Sources are listed at the end of the paper. sourced

For a practitioner, the stakes are smaller but more immediate. I build systems on top of language models, for clients who want to know whether a cheaper, smaller model will do. If abilities emerge suddenly, a test on a small model tells me little about a large one. If they grow smoothly, a careful test on the small model is a useful forecast. [AUTHOR NOTE: optionally add a sentence on a client decision where small-model tests did or did not predict the larger model]

Four positions

The literature now holds at least four positions, placed in Figure 3. The first, from Wei and colleagues, is that emergence is real and unpredictable at the level of task scores. The second, from Schaeffer and colleagues, is that most of it is a mirage produced by the choice of metric. The third, from Du and colleagues, accepts that metrics matter but finds jumps that survive continuous metrics once performance is plotted against pre-training loss. The fourth, from complexity science, says that a jump in a score is not what the word emergence should mean at all.

Figure 3. The positions differ on where the sharpness lives, and on what level the explanation works. Placements are my judgement from the papers' abstracts and main results. illustrative, author judgement

These positions disagree on two separate questions. One is whether the sharpness lives in the model or in the measurement. The other is whether the explanation should be given at the level of the score or at the level of the mechanism inside the model. The rest of this paper takes the questions in that order: first the metric, then the mechanism, then what the word should mean.

Part 2

The metric argument

In April 2023, Rylan Schaeffer, Brando Miranda and Sanmi Koyejo of Stanford posted a paper with a provocative title: "Are emergent abilities of large language models a mirage?" Their alternative explanation was simple. For a fixed task and model family, emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behaviour with scale (Schaeffer et al., 2023). Nonlinear or discontinuous metrics produce apparent emergent abilities. Linear or continuous metrics produce smooth, continuous and predictable changes.

The paper became one of the most discussed of that year. In December 2023 it received an Outstanding Main Track Paper award at NeurIPS, the largest machine learning conference, alongside a paper on privacy auditing (NeurIPS 2023 awards). The award citation repeats the paper's conclusion: alleged emergent abilities evaporate with different metrics or with better statistics. It was an unusual honour for a paper whose main contribution was to say that a celebrated phenomenon might not exist.

The arithmetic of a pass mark

The core of the argument fits in one line of school algebra. Suppose a model produces an answer of ten tokens, and each token is correct with probability p, independently of the others. Then the chance that the whole answer is correct is p multiplied by itself ten times:

exact match = p^L, L = 10.

This compounding is what makes pass-or-fail scores sharp. If the per-token accuracy rises steadily from 50 to 90 per cent, the exact-match score rises from 0.1 per cent to 35 per cent. If it rises from 90 to 99 per cent, exact match rises from 35 to 90 per cent. Most of the visible change in the exact-match score happens in a narrow band where per-token accuracy is already high. Below that band, the exact-match score looks like zero, whatever the model is doing.

Figure 4. The longer the answer, the later and sharper the pass mark rises. Exact match equals per-token accuracy raised to the answer length, if errors are independent. At 90 per cent per token: 59 per cent for 5 tokens, 35 per cent for 10, 0.5 per cent for 50. worked case, mathematics

Thought experiment 1

The pupil who got better by half

Two of Mrs. Hegde's pupils, both invented, take the test in March. Lakshmi types each word correctly 90 per cent of the time. By June she has improved to 95 per cent, so her error rate has halved. Her brother Suresh is training for a harder test, with a passage of fifty words instead of ten.

Pause. Before you read on: by roughly how much does halving the error rate raise Lakshmi's chance of passing the ten-word test? And Suresh's chance on the fifty-word test, if he makes the same improvement?

Most people guess a modest rise, perhaps a doubling. For Lakshmi the pass chance goes from 35 to 60 per cent, which is close to that guess. For Suresh it goes from 0.5 per cent to 7.7 per cent, a fifteenfold increase from the same halving of errors. On a long enough passage, every steady improvement looks like a sudden arrival, and the longer the passage, the later and sharper the arrival.

The lesson is that the sharpness of a pass-or-fail score is partly a property of the test's length, not of the pupil. The limit is the assumption of independence. Real errors cluster: a pupil who misreads one word often misreads the next. Clustering can make the compound curve gentler or steeper than the simple formula predicts, and I will meet both cases in my own networks.

Three tests of the mirage account

Schaeffer and colleagues did not rest on the algebra. They tested the account in three ways. First, they collected outputs from the InstructGPT and GPT-3 family on two arithmetic tasks: multiplying two-digit numbers, and adding four-digit numbers. Under exact-match accuracy, the family showed emergent abilities when the target had four or five digits. When they rescored the same outputs by token edit distance, a measure of how many characters must change to reach the right answer, the improvement became smooth and predictable (Schaeffer et al., 2023).

Second, they analysed the BIG-bench results. Using hand-annotated task, metric and model-family combinations, they found that claimed emergent abilities appeared under only 4 of 39 metrics. Two metrics accounted for more than 92 per cent of the claims: Multiple Choice Grade, which is discontinuous, and Exact String Match, which is nonlinear (Schaeffer et al., 2023). When they rescored one model family's multiple-choice answers by the Brier score, which rewards the probability placed on the right option, the emergent abilities disappeared.

Third, they went the other way. Taking vision networks that nobody had ever called emergent, they designed discontinuous metrics and produced emergent-looking curves on demand. If a metric can manufacture emergence in a system that has none, then a sharp curve under that metric is weak evidence on its own.

Thought experiment 2

The institute that advertised emergent typists

A rival institute across the road, also invented, puts up a banner: "Our pupils develop the emergent ability to pass in only five months." Mrs. Hegde, irritated, rewrites the rules of her own practice test. She now demands a passage of fifty words. Her register shows that her pupils develop the same "emergent ability" at month eight. The rival institute then shortens its practice test to a single word, and announces emergence in the second week.

Pause. Before you read on: in which month do the pupils of either institute really acquire the ability to type?

There is no such month, because nothing about the pupils changed between the banners. In my smooth synthetic family, the scale at which exact match crosses 50 per cent moves from 2.5 to 5.4 to 7.2 units of log compute as the answer grows from 1 to 10 to 50 tokens. The underlying per-token curve is identical in all three cases. Only the pass mark moved, and the "emergent scale" moved with it.

The lesson is that a reported emergence point is a joint property of the model and the test, and should be reported with the test's length and scoring rule. The limit is that real tasks do not come with a dial for answer length. The fifty-word passage and the ten-word passage are different tasks for a model as well as for a pupil, since longer passages raise new difficulties of their own.

Static view: in the smooth synthetic family, per-token accuracy crosses 50 per cent at 2.5 units of log compute. Exact match on a 10-token answer crosses 50 per cent at about 5.4 units, and on a 50-token answer at about 7.2 units. The underlying per-token curve is the same in every case.

Answer length: 10 tokens

Figure 5. Drag the answer length and watch the "emergence point" move. Teal: per-token accuracy in the smooth synthetic family, which rises along an S-curve in log compute. Indigo: exact match for the chosen answer length. Nothing about the family changes as you drag. interactive, worked case

Resolution: zero is not a measurement

The second half of the mirage account concerns what Schaeffer and colleagues called resolution. They borrowed the dictionary definition: the smallest interval measurable by a scientific instrument, its resolving power. A benchmark with 200 test items cannot measure an accuracy smaller than half a per cent. If a small model's true exact-match rate is one in a thousand, a 200-item benchmark will usually record it as zero.

When they generated more test data for the arithmetic tasks, they found that every model in the InstructGPT and GPT-3 family achieved above-chance accuracy (Schaeffer et al., 2023). The flat stretch at zero, which makes the later rise look sudden, was partly an artefact of having too few questions. The small models were not unable to do the task. They did it rarely, and the benchmark was too short to catch them.

My synthetic smooth family shows the same effect. At a scale where the per-token accuracy is 39 per cent, the true exact-match rate for a ten-token answer is about 0.008 per cent. With 50, 200, 1,000 or 10,000 test items I measured exactly zero. Only with 100,000 items did any successes appear: 12 of them, a rate of 0.012 per cent, close to the predicted value. Figure 6 shows where the zeros turn into numbers as the test set grows.

Figure 6. Zero means "too few questions to see it". Smooth synthetic family, 10-token answers. Expected rates at the last two scales are 8 and 98 per 100,000. With 50 to 1,000 items every cell reads zero. Run in this paper's instrument. measured, synthetic data

The resolution argument matters because the flat stretch is half of what makes a curve look emergent. A jump from a measured 0.1 per cent to 30 per cent, plotted on a logarithmic scale, is a steady rise. A jump from a recorded zero to 30 per cent, plotted on any scale, is a cliff. A zero on a small benchmark means "below what this test can see", and it should be plotted as an upper bound, not as a point.

Part 3

An instrument for telling the two apart

The metric argument gives a recipe, but a recipe is not yet an instrument. To use it on a claimed emergent ability, one needs a procedure that takes a family of models and their outputs, and returns a verdict: this jump is the pass mark, or this jump is in the model. I built a small procedure of that kind, and before trusting it on anything real I tested it on two synthetic families whose answers I knew in advance.

The audit procedure

The procedure has four steps. First, rescore every model's outputs with a continuous measure, here the share of answer tokens that are correct. Second, transform that measure so that a smooth improvement looks like a straight line; I use the log-odds, which turns an S-shaped rise into a line. Third, fit a straight line against log compute using only the smaller models. Fourth, predict the larger models from the line, and flag a break if any of them misses the prediction by more than three times the fitting error plus half a unit.

The threshold is a judgement call, and I will come back to its weaknesses. Its job is to separate two cases. In the first, the continuous measure was on a steady course all along, and the exact-match jump is the compound arithmetic of Part 2. In the second, the continuous measure itself jumps, and no rescoring will smooth it away. The procedure also predicts the exact-match score from the fitted line, by raising the predicted token accuracy to the tenth power, so the two scores can be compared directly.

Figure 7. The audit: rescore, straighten, fit small, predict large. The threshold of three times the fitting error plus half a log-odds unit is my choice. The audit also predicts exact match from the fitted line, for comparison. mechanism diagram

Two calibration families

The first synthetic family is smooth by construction. Each model's per-token accuracy rises along an S-curve in log compute, from 10 per cent to 97 per cent across thirteen sizes. I sampled 200 test answers of ten tokens each, and scored them both ways. The exact-match score sits at zero for the first five sizes and then climbs to 72 per cent. Seen through exact match alone, it is a textbook emergent ability.

The second family has a real threshold built in. Its per-token accuracy creeps from about 55 to 57 per cent across the first eight sizes, and then jumps to 97 per cent at the ninth. This is Imran's register: a new method arrives at a particular scale, and every measure moves at once. Its exact-match score jumps from 1 per cent to 72 per cent in one step.

Figure 8. Both families look emergent under exact match; only one jumps per token. Thirteen sizes each, 200 answers of 10 tokens per size. The smooth family is the negative control and the threshold family the positive control for the audit. Run in this paper's instrument. measured, synthetic data

The audit separated the two families cleanly. For the smooth family, the line fitted to the eight smaller models predicted the larger ones with a largest miss of 0.33 log-odds units, inside the threshold of 0.61, so no break was flagged. Its predicted exact-match score for the largest model was 62 per cent, against 72 per cent observed. For the threshold family, the line was almost flat, with a slope of 0.013, and the largest model missed it by 3.3 units. A break was flagged, and the predicted exact-match score of 0.4 per cent was wrong by a factor of nearly two hundred.

Figure 9. The audit stays silent on the smooth family and flags the others. Largest miss of the line fitted to smaller models, against the break threshold. Synthetic families: line fitted to the 8 smallest of 13 sizes. Adders: fitted to the 5 smallest of 11 widths. Run in this paper's instrument. measured

These two families are my positive and negative controls. The smooth family is the negative control, a case with no real threshold where the audit must stay silent. The threshold family is the positive control, a case with a real threshold where the audit must speak. An audit that fails either control is not worth applying to anything else. This one passed both, which tells me it works on cases as clean as these. It does not tell me it works on the messier families of real models.

Thought experiment 3

Two registers with three entries

An inspector visits two institutes, both invented, each of which has trained only three batches of pupils. She reads the registers. At the smallest scale, both pass rates are zero. At the middle scale, both are 1 per cent or less. At the largest, both are 72 per cent. She also reads the second column, the share of correct words, and finds 10 and 55 per cent at the bottom, 60 and 57 per cent in the middle, and 97 per cent at the top for both.

Pause. Before you read on: from these three rows alone, can she tell which institute teaches a smooth skill and which has a sudden trick?

She cannot, at least not reliably. These are the real values of my smooth and threshold families at log compute 0, 3 and 6. The bottom rows differ, but that says only where each institute started. The middle rows of the two registers are almost identical on both columns, and so are the top rows. The difference between the families lives between the middle and top rows: one rises steadily through the gap, the other waits and then leaps. With three models per family, no metric, continuous or not, can see inside that gap.

The lesson is that continuous rescoring needs dense sampling of scale to say anything. A reanalysis on a family of three or four models, a common situation for published model families, can confirm smoothness only where it has data. The limit is cost. Training more intermediate models is expensive, which is exactly why public families are sparse, and why the observational approach in Part 5 is attractive.

What the instrument cannot do

The audit has three blind spots that matter for real cases. First, it assumes the continuous measure is the right one. A model can improve smoothly on token accuracy while doing something abrupt that token accuracy does not register, such as learning to carry a digit. Second, it depends on the transformation. A rise that is a straight line in log-odds may be a curve on another scale, and a different choice could flag different breaks. Third, it treats scale as one number, when real families differ in data, architecture and training recipe as well as size.

None of these blind spots is fatal, but each one is a place where a verdict of "smooth" or "break" needs a second look. The real families in the next two parts were chosen to probe the first blind spot directly. They are small networks I trained myself, so I can open them up and see what the continuous measure is averaging over.

Part 4

A family of adders

The first real family is a set of small networks that add two five-digit numbers. Each network reads the ten digits as separate symbols and writes out the six digits of the sum, one choice of ten for each position. The family varies one thing, the width of the network: two hidden layers of 8, 16, 32, 64, 96, 128, 192, 256, 384, 512 or 1,024 units. That gives eleven sizes, from 1,420 to 1.2 million weights. Every size was trained three times with different random seeds, for the same 4,000 steps of 256 examples each, and tested on 20,000 fresh sums.

Five-digit addition is a deliberate choice. Schaeffer and colleagues' most famous example was four- and five-digit arithmetic in GPT-3, and the task has natural parts, the digit positions, that can be scored on their own. It is also a task where the right answer needs a specific piece of machinery, the carry, that a network either has or lacks for each position.

The cliff, and its rescoring

Under exact match, the family shows a classic emergent ability. The four smallest sizes score zero or 0.1 per cent. At 96 and 128 units the score is still only 0.5 and 1.1 per cent. Then, at 192 units, it jumps to 40 per cent, and at 256 units it reaches 70 per cent. Between 128 and 192 units, a 50 per cent increase in width, the exact-match score rises 37 times. Plotted with the usual axes, it is a cliff.

Rescored by the share of correct digits, the same outputs tell a gentler story. Digit accuracy rises from 38 per cent at the smallest size to 66 per cent at 128 units, then to 87 per cent at 192 and 94 per cent at 256. The rise is real and it steepens, but there is no flat stretch at zero and no cliff. The log-probability of the correct answer, a fully continuous score, climbs from minus 9.9 to minus 5.0 across the same range before the exact-match score has moved at all.

Figure 10. The pass mark leaps at 192 units; the continuous scores were already climbing. Means of three seeds at each of 11 widths; 20,000 test sums each. Mean probability per digit is the geometric mean, from the log-probability of the correct answer. Exact match at 192 units ranged from 0.07 to 0.64 across seeds. Run in this paper's instrument. measured

The compounding arithmetic of Part 2 explains the exact-match cliff almost perfectly. If I multiply the six per-digit accuracies together at each size, the product matches the observed exact-match score to within 0.2 percentage points at every width. At 192 units, the product is 40.3 per cent and the observed score is 40.4. The errors in different digits are close to independent, so the pass mark is doing exactly what the formula says it does. Given the per-digit scores, the exact-match cliff contains no information of its own.

Figure 11. Multiply the digit scores and you get the cliff. At every width the product of the six per-digit accuracies is within 0.2 percentage points of the observed exact-match score: 40.3 against 40.4 per cent at 192 units. Errors in different digits are close to independent here. Run in this paper's instrument. measured

The staircase inside the average

That would be the end of the story if the per-digit scores were smooth. They are not. Figure 12 shows each digit position separately, and the six positions behave in strikingly different ways. The leading digit of the sum, which is only ever 0 or 1, is perfect even in the narrowest network. The ten-thousands digit is right 85 per cent of the time at 8 units. The units digit improves steadily, from 10 per cent at 8 units to 83 per cent at 128. The tens digit sits at exactly chance, 10 per cent, in every network up to 128 units, and then jumps to 59 per cent at 192.

Figure 12. Inside the average: each digit switches on at its own width. Means of three seeds. The tens digit is at chance, 0.10, at every width up to 128 and 0.59 at 192. The units digit rises steadily from 0.10 to 0.83 across 8 to 128 units. Run in this paper's instrument. measured

So the average digit accuracy is a staircase of overlapping steps. Some steps are gentle ramps, like the units digit. Others are sharp switches, like the tens digit, which goes from chance to nearly 60 per cent across one increase in width. The audit of Part 3 flags a break even in the continuous digit accuracy: the line fitted to the five smallest sizes misses the larger ones by up to 5.4 log-odds units, against a threshold of 1.0. In this family, the continuous rescoring removes the cliff in the pass mark and leaves real breaks in the parts.

Static view: exact match is 0.011 at 128 units and 0.40 at 192. The share of correct digits is 0.66 at 128 and 0.87 at 192. Digit by digit, the tens digit is 0.10 at 128 units and 0.59 at 192; the units digit is 0.83 at 128.

Figure 13. Choose a score and see how sudden the adders look. Same outputs, three ways of scoring them. "Each digit" draws the six positions, from the leading digit (lightest) to the tens digit (darkest). Run in this paper's instrument. interactive, measured

Why the digits arrive in that order

The order is a clue to the mechanism. The digits that arrive first are the ones at the top of the sum, where an approximate total is enough. Suppose a network forms a rough estimate of the sum as a single quantity. It can then read off the ten-thousands digit without any digit-by-digit addition, and it will go wrong only when the true sum lies close to a multiple of 10,000. I call this the magnitude route, and it is cheap in width.

I tested that reading directly. For a 16-unit network, I sorted the test sums by how close they fell to a multiple of 10,000. Where the sum was more than 30 per cent of the way from the nearest boundary, the ten-thousands digit was right 99.5 per cent of the time. Within 5 per cent of a boundary, it was right only 66 per cent of the time. A second seed gave 97 and 64 per cent. The narrow network is estimating, not adding, and its errors sit exactly where an estimate is fragile.

Figure 14. Narrow networks estimate the total: their errors sit at the boundaries. Distance as a share of 10,000. The 16-unit network is right 99.5 per cent of the time far from a boundary and 66 per cent near one. Bars start at 0.5. Run in this paper's instrument. measured

The magnitude route cannot reach the lower digits, because an estimate precise enough to read off the tens digit would need resolution to within ten parts in two hundred thousand. Those digits need a second route, a local one, that combines the two input digits in the same position and the carry from the position below. The units digit needs no carry, so it improves gradually as the network gains room for a 100-entry table. The tens digit needs both the local table and the carry from the units, so it waits until both are in place, and then arrives together. This is my reading of the order, supported by the boundary test for the top digits. I have not checked the local route by ablation.

Thought experiment 4

The adder that got the right answer for the wrong reason

Take the 16-unit network, and ask it for the ten-thousands digit of 41,200 plus 22,300. It answers 6, correctly. Now ask it for the same digit of 41,200 plus 18,750, a sum of 59,950, just below a boundary. It has a fair chance of answering 6, when the right answer is 5.

Pause. Before you read on: does the 16-unit network "know" the ten-thousands digit of a sum?

It knows something that agrees with the digit nine times out of ten, and a benchmark scoring that digit would credit it with 93 per cent accuracy. But it does not know the digit in the sense of computing it, and its errors are not random. They cluster where the method it uses, estimation, gives way. A wider network that has the local route gets the same digits right for a different reason, and its errors, when it makes them, fall somewhere else.

The lesson is that the same score on the same item can rest on different methods at different scales. A jump in a sub-skill can be the arrival of a new method, not the improvement of an old one. The limit is that I could see this only because the task has a known structure. For most benchmark items, no one knows which methods a model could be using.

The verdict on the adders

The adder family gives a mixed verdict, and the mixture is the point. The exact-match cliff is a mirage in the precise sense of Schaeffer and colleagues: it is fully predicted by the per-digit scores. The per-digit scores are not a mirage. They contain at least one switch, the tens digit, that looks like a threshold under any metric I tried, and a mechanism, the second route, that plausibly explains it. Rescoring moved the emergence from the whole task to one of its parts, where it became easier to understand and harder to dismiss.

Part 5

Where the jumps survive

The adders show emergence moving from a whole task into one of its parts. A sharper test is a task with no parts at all, where no partial credit is possible because there is nothing partial to give credit for. If a jump survives there under every continuous score, the mirage account has no purchase on it. The classic task of this kind is sparse parity.

Sparse parity: nothing to give partial credit for

In sparse parity, each example is a string of 40 random bits, and the label is whether the first three bits contain an odd number of ones. The network is not told which three bits matter. The task has a peculiar property. Any one or two of the three bits, taken alone, is completely uncorrelated with the label. A network that has found two of the three relevant bits does no better than one that has found none. There is no smooth path of partial solutions on the output, so every output score, discrete or continuous, should sit at chance until the network has found all three.

Barak and colleagues studied this task in depth. They found that a variety of networks learn sparse parities with discontinuous phase transitions in their training curves. On small instances, learning occurred abruptly after a number of steps that grows as a power of the input size (Barak et al., 2022). Their title names the twist: hidden progress. The output scores do not move, but the network is making progress underneath, as gradient descent gradually amplifies the relevant bits.

I trained one-hidden-layer networks of 32, 64, 128 and 256 units on the task, four seeds each, for 6,000 steps of plain gradient descent. Every run that learned the task did so in the same way. The test loss, a fully continuous score, sat at 0.69, the value for pure guessing, for between 1,600 and 5,900 steps. It then fell from 0.6 to below 0.1 within 400 to 700 steps. No rescoring of the outputs could smooth this, because the outputs were at chance under every score until the moment they were not.

Figure 15. Parity: flat at guessing, then a fall in a few hundred steps. Four seeds each of 32 and 256 hidden units. The loss is a fully continuous score. Every run that learned fell from 0.6 to below 0.1 within 400 to 700 steps. Run in this paper's instrument. measured

The width of the network changed when the jump came, not what it looked like. By step 3,000, none of the 32- or 64-unit runs had learned the task, two of four 128-unit runs had, and three of four 256-unit runs had. By step 6,000, half of the narrow runs had learned it and all of the wide ones. Read as a scaling plot at a fixed training budget, that is an emergent ability in width. Read run by run, it is a jump in training time whose waiting time shrinks as the network grows.

Static view: by step 3,000, none of four runs had learned parity at 32 or 64 units, two of four at 128 units and three of four at 256. By step 6,000, two of four had learned at 32 and 64 units and all four at 128 and 256.

Training budget: 3,000 steps

Figure 16. Pick a width and a training budget: is parity "emergent"? Four seeds per width. The shaded region is the training budget; a run counts as learned if its loss fell below 0.1 inside it. Run in this paper's instrument. interactive, measured

How hidden was the progress?

I also tracked a simple internal measure: how much larger, on average, the first-layer weights on the three relevant bits were than the weights on the other 37. At the start this ratio is about 1. Between the start of training and 500 steps before each jump, it crept up by only 1 to 8 per cent. After the jump it rose to between 1.5 and 4.6. My crude measure, then, shows little hidden progress. Barak and colleagues used sharper measures and found more. What my measure does show is that the jump is not confined to the output: the network's weights reorganise at the same moment its loss falls.

Sparse parity is an emergent ability that survives continuous-metric reanalysis, in my networks and in the literature. The reason is structural. The mirage account works by finding a continuous quantity, like per-token accuracy, of which the pass mark is a sharp function. Parity has no such quantity at the output, so the sharpness has nowhere to hide except in the network itself.

Jumps against pre-training loss

Parity is a toy. The most important evidence that some jumps survive in real language models comes from Du and colleagues. They proposed studying emergent abilities against pre-training loss instead of model size or compute. Models with the same pre-training loss, but different sizes and amounts of data, gave the same performance on downstream tasks. And on certain tasks, they found, a model shows an emergent ability regardless of the continuity of the metric, once its pre-training loss falls below a specific threshold. Above that threshold, performance stays at the level of random guessing (Du et al., 2024).

The phrase "regardless of the continuity of metrics" is the direct answer to the mirage account. Du and colleagues accept that metrics matter. Their claim is that, for some tasks, the jump survives a continuous metric once the right variable is put on the horizontal axis. Pre-training loss absorbs differences in model size, data and recipe that make size alone a noisy axis. The proposal leaves the emergence real but relocates it, from "at a certain size" to "at a certain level of competence on the training objective".

Quanta: a theory of the staircase

The staircase inside my adders has a theory. Michaud, Liu, Girit and Tegmark proposed that what a network knows comes in discrete chunks, which they called quanta (Michaud et al., 2023). If quanta are learned in order of how often they are used, and those frequencies follow a power law, then the average loss falls as a smooth power law while each quantum switches on sharply. The smooth curve of the scaling laws and the sudden arrival of particular skills become two views of one process.

My digit positions behave like quanta in this sense. Each one is a separable skill, each switches on over a narrow range of width, and the average across them is smoother than any one of them. The match is not exact. My digits are not learned in order of frequency, since every digit appears in every example; they are learned in order of how cheaply the network can get them. But the shape of the explanation, smooth averages over sharp parts, is the one that fits.

Smooth enough to forecast

If many jumps are sigmoids in disguise, they should be predictable from smaller models. Ruan, Maddison and Hashimoto tested this without training any models of their own. They built scaling laws from about 100 publicly available models, and showed that the differences between model families are consistent with a low-dimensional space of capabilities. Using it, they found that several emergent phenomena follow a smooth, sigmoidal behaviour and are predictable from small models (Ruan et al., 2024).

This is the strongest support for the mirage account at the scale of real language models. It does not say that the curves are straight. It says they are smooth S-curves whose position can be forecast before the steep part arrives. A sigmoid can still be steep, and a capability can still rise from 10 to 90 per cent across a small range of compute. What disappears is the claim that nothing in smaller models announced it.

Trends that turn round

There is one more kind of surprise that rescoring does not touch: a trend that reverses. McKenzie and colleagues ran a public contest, the Inverse Scaling Prize, inviting tasks on which larger models do worse. The winning tasks showed inverse scaling on 11 datasets, which they traced to four causes, among them a preference for repeating memorised sequences over following instructions, and easy distractor tasks (McKenzie et al., 2023). Their tasks helped reveal U-shaped and inverted-U trends, where an initial trend reverses.

A U-shaped curve is a different challenge to forecasting from an emergent jump, but it lands in the same place. If a trend can turn round at a scale nobody has yet trained, then extrapolation from small models is fragile even when every metric is continuous. The mirage account removes one source of surprise. It does not remove surprise.

Where reanalysis cannot decide

The cases so far have a clear verdict. Some do not, and they mark the boundary of the method. The first is open-ended output, such as an essay or a proof, where there is no agreed continuous score. Any partial-credit rubric there is a theory of the task, and a jump under one rubric may vanish under another. The second is the sparse family, the problem of the inspector in Thought experiment 3. The third is correlated error: if a model's mistakes cluster, the compound formula fails, and the exact-match curve can be sharper or softer than per-token scores predict.

In my adders the errors happened to be close to independent, so the formula held to within 0.2 points. That is a fact about this family, not a law. A reanalysis that assumes independence without checking it can manufacture smoothness as easily as an exact-match score can manufacture emergence.

Figure 17. Three places where rescoring cannot give a verdict. Each is a limit of the method, not evidence for or against emergence. failure boundary

Part 6

Was the ability already there?

Strip the debate of its plots and it becomes an old question: when a new ability appears, was it already present in what came before, or is it something new? Physicists, complexity scientists and Indian philosophers have all argued about this, with different words. Each tradition sharpens one part of what the machine learning debate leaves vague.

More is different

In 1972 the physicist Philip Anderson published a short essay in Science called "More is different". His target was the belief that, once the fundamental laws are known, everything else is just application. Anderson argued that the ability to reduce everything to simple laws does not imply the ability to start from those laws and reconstruct the universe. At each level of complexity, he wrote, entirely new properties appear, and understanding them requires research as fundamental as any other (Anderson, 1972).

Anderson's emergence is not about sharp curves. A crystal is emergent because it breaks a symmetry that its atoms' laws respect, not because it appears suddenly. Superconductivity is emergent because it needs a new description, in terms of collective states, that no list of individual electrons supplies. Sharpness often comes with this kind of emergence, as in a phase transition, but it is a symptom, not the definition.

Krakauer, Krakauer and Mitchell made this point directly about language models. In complexity science, they wrote, emergence describes how many-body systems manifest higher-level properties that can be described by replacing high-dimensional mechanisms with lower-dimensional effective variables (Krakauer et al., 2025). On that definition, a jump in a benchmark score is neither necessary nor sufficient for emergence. What matters is whether a new, simpler description of the model's behaviour becomes available and useful.

Figure 18. The register explains the pass mark, not the pupil. The analogy captures the compounding and the effect of test length. It strains on new methods and correlated errors, and breaks on self-knowledge. analogy check

Thought experiment 5

The physicist who says we are all using the wrong word

A complexity scientist, invented for this paper, reads the whole debate and is unimpressed. "You are arguing about whether a curve is steep," she says. "That is a fact about your axes. Emergence is about whether the system acquires a new level of description. A model that learns to carry digits has acquired a variable, the carry, that summarises a great deal of lower-level detail. That is emergence whether the score rises in one step or a hundred." She adds that recognition, self and prakāśa are not needed to describe any of it.

Pause. Before you read on: is she right that steepness is beside the point, and right that the contemplative vocabulary is surplus?

On the first point she is largely right, and this paper has been partly arguing about the wrong property. The parity networks of Part 5 grew an internal variable, a focus on three particular bits, and that variable is the emergent thing on her definition, whatever the shape of the loss curve. On the second point she is also right about the networks. Nothing in my instruments needs RTM to describe it. Where she goes too far is in assuming that a new effective variable is all that "a new ability" could mean, for a system that also experiences having it.

The lesson is that the cleanest test of emergence asks for a new variable inside, not a steep curve outside. The limit is practical. Finding new effective variables in a large model is the hardest problem in interpretability, while rescoring a benchmark takes an afternoon. The steepness debate persists because it is cheap to have, not because it is the deep one.

Satkāryavāda and asatkāryavāda

Classical Indian philosophy organised a long debate around exactly this question, under the name of causation. The Sāṃkhya school held satkāryavāda, the doctrine that the effect already exists in its cause before it is produced. The Sāṃkhyakārikā gives five reasons. What does not exist cannot be produced. A cause must be taken up to produce its effect. Not everything comes from everything. A capable cause produces only what it is capable of. And the effect shares the nature of its cause. The stock example is oil, which can be pressed from sesame seeds but never from sand, because it was in the sesame all along.

The Nyāya and Vaiśeṣika schools held the opposite, asatkāryavāda: the effect is a new thing that did not exist before. A pot is a new whole, an avayavin, distinct from the two halves of clay that compose it. It can hold water, which the lumps of clay could not. On this view, production is the beginning of something real, called ārambha, and the parts do not contain the whole in miniature.

The mirage account is a kind of satkāryavāda. The ability to add five-digit numbers was already present in the small model, as a per-token competence, and the larger model only shows it more fully. The exact-match score is the oil press, revealing what was in the seed. The emergence account, in its strong form, is asatkāryavāda: the large model has a new whole, an ability that the small one simply did not have. Each side has a model of production, and each had a sharp critic in classical India.

My own instruments suggest a position that both classical schools would find uncomfortable. At the level of the average, the satkārya view fits: the mean digit accuracy rose in a staircase of modest steps, and the exact-match cliff was mostly compounding. At the level of each digit, the asatkārya view fits: a given digit position went from chance to above 80 per cent within a doubling of width, as a new sub-skill came into being. Whether the effect pre-exists in the cause depends on how finely you cut the effect.

Figure 19. Whether the effect pre-exists depends on how finely you cut it. Paraphrased from the Sāṃkhyakārikā, Nyāya-Vaiśeṣika sources and Utpaladeva. The mapping to the adder family is this paper's reading. conceptual diagram

Trika philosophy offers a third shape. Utpaladeva and Abhinavagupta held that every effect exists first within consciousness and is manifested outwardly by its will, in a doctrine called ābhāsavāda, the theory of appearances. In the section of the Īśvarapratyabhijñākārikā on causation, Utpaladeva explains the power to produce as the capacity of awareness to show externally what it holds inwardly. This is a satkāryavāda without Sāṃkhya's dualism. For a machine it supplies a careful phrase: the network's later ability was latent in its training data and objective, and is shown as its capacity grows.

The lover who did not recognise

The Pratyabhijñā school took its name from recognition, and it explained recognition with a story that is directly relevant here. A woman has fallen in love with a man by hearing of his qualities. One day he stands in front of her, but she does not know who he is, and she feels nothing special. Then someone tells her that this is the very man, and her heart is filled. Nothing in the light reaching her eyes has changed. What changed was the recognition that the present person is the one already known.

Thought experiment 6

The beloved at the door

Sit with the story for a moment, in the way it was meant to be read. The woman's perception improved gradually as the man walked closer: first a figure, then a face, then the details of his clothes. Her recognition did not improve gradually. It arrived in a moment, and it changed everything about how she held the same perception.

Pause. Before you read on: in your own experience of understanding something, did the understanding grow like the perception, or arrive like the recognition?

The Trika answer is that both happen, at different layers. The perception is a manifestation that grows; the recognition is a reflexive act that is either made or not. In RTM terms, L2 grows smoothly while L4 switches. A continuous metric of perception would show no jump at all, and it would miss the most important event in the story. Reading this, I am reminded that a smooth measure can be blind to a change of state in what it measures. [AUTHOR NOTE: optionally add a line on a moment of recognition in your own practice, if you wish to share one]

The lesson is that some changes are discrete at one layer and continuous at the layer below, and the choice of metric decides which one we see. The limit is severe for machines. The woman's recognition involves her own awareness of the man and of her love, what Trika calls vimarśa. My networks show discrete changes in internal structure, but nothing in them knows that it has changed.

Reading the evidence through RTM

The Recognition Theory of Mind is my working hypothesis, not an established theory, and nothing in this paper suggests that a model is conscious. Applied carefully, it separates the layers at which the emergence question can be asked. At L1, spanda or responsiveness, every model in every family responds to its input, and there is no emergence to speak of. At L2, prakāśa or manifestation, the debate lives: what the model can show, and how sharply that changes with scale.

At L4, pratyabhijñā or recognition, there is a structural parallel. The digit networks of Part 4 came to treat a new configuration of inputs as an instance of a sub-skill already learned, such as carrying. That has the shape of recognition, the present taken as the already known, but there is no one who recognises. L3, vimarśa or reflexive awareness, is absent at every scale I tested. A model that scores perfectly on five-digit addition has no access to the fact that it can add. L5, the luminous ground, is outside what any of these instruments could address.

Figure 20. The debate lives at L2; recognition has a shape, and no one recognises. Recognition Theory of Mind layers, a working hypothesis. The dotted L4 rung marks a structural parallel, not a capacity. author hypothesis

One prediction follows from this reading, and I hold it loosely. If anything like reflexivity were ever to appear in a machine, it would not show up first as a jump in a task score. It would show up as a new internal variable that tracks the model's own states and is used by it. That is Krakauer's kind of emergence applied to L3, and it is the kind no benchmark rescoring can find.

Part 7

What to do with an emergence claim

For anyone who builds with language models, or evaluates them, the debate reduces to a set of habits. None of them is expensive. Together they turn a sharp curve from a surprise into a measurement with known limits.

Five habits

First, always record a continuous score next to the pass-or-fail one. For answers with a single correct string, record the per-token log-probability of the right answer, or the token edit distance. For multiple-choice items, record the probability placed on the correct option. These cost nothing extra to compute and they change what can be learned from the same run.

Second, report resolution. A score of zero on 200 items should be written as "less than 0.5 per cent", and plotted as an upper bound. If the small models matter, add items until they score above zero, as Schaeffer and colleagues did. Third, report the answer length and scoring rule with any emergence point, since the point moves with both.

Fourth, decompose. If a task has natural parts, such as digit positions, reasoning steps or sub-questions, score them separately. An aggregate that rises smoothly can hide parts that each jump, and it is the parts that predict where the next failure will be. Fifth, sample scale densely where the curve bends. Three models per family cannot distinguish a steep smooth rise from a true break, as the inspector found in Thought experiment 3.

Figure 21. Four questions before you call an ability emergent. A claim that answers all four says which kind of jump it has; a single curve does not. decision instrument

In my own work this shows up when a client asks whether a smaller model will do. A small model that scores zero on a hard extraction task under exact match may have a high per-field accuracy and fail only because one field in ten is wrong. Rescoring tells me whether the gap to a larger model is a smooth climb that fine-tuning might close, or a missing component that it will not. [AUTHOR NOTE: optionally name the kind of task where this decision came up]

Thought experiment 7

The safety test that kept passing

A laboratory, invented for this paper, evaluates each new model for a dangerous capability, say the ability to complete a multi-step attack on a computer network. The test has 50 scenarios, each scored pass or fail. Three generations of models score zero, and each time the laboratory reports "no capability detected". The fourth generation passes 40 per cent of scenarios. Afterwards, an analyst rescores the old outputs step by step, and finds that the share of correct steps had risen from 20 to 45 to 70 per cent across the three earlier generations.

Pause. Before you read on: was the fourth model's capability emergent, and was the laboratory's testing adequate?

On the mirage account the capability was not emergent at all. It was growing steadily, in plain view of a continuous metric that nobody recorded. That is uncomfortable in two directions. For the laboratory, it means three reports of "no capability" were true only in the narrowest sense, and an honest forecast was available. For those who welcome the mirage account as reassurance, it means the predictability it promises exists only if someone measures the continuous score, which pass-or-fail evaluations are designed not to do.

The lesson is that a pass-or-fail safety test reports the past and hides the trend. The limit is that a continuous score is not always available. For a multi-step task, defining a "correct step" requires judgement, and a model can find a route that no step-wise rubric anticipated. Continuous rescoring then needs a rubric that is itself a hypothesis about how the task is done.

How to write up a jump

When I see a sharp curve in an evaluation, I now write it up in a fixed form. The form names the score, the answer length and the number of items. It gives the smallest score the test can resolve. It shows the continuous score next to the pass mark, on the same axis of scale. It lists which parts of the task were scored separately, and what each part did. Finally, it says which of three kinds of jump the evidence supports: a metric jump, a part-level jump or a model-level jump.

The form is short, but it changes conversations. A client who hears "the larger model suddenly handles invoices" hears a story about a mysterious leap. A client who hears "the larger model gets the tax field right, which the smaller one misses one time in five, and the other fields were already fine" hears a story about one component. The second story suggests a remedy, such as a targeted check on that field, that the first story hides.

Sizing models for a client

The practical consequence for model selection is a little sobering. Where an ability is a smooth climb, a careful continuous measurement on a few small models will forecast a larger one, and the forecast can be checked. Where an ability depends on a component that switches on, the forecast is only as good as one's understanding of that component. In the addition family, knowing that the tens and hundreds digits had not yet switched on was the most useful single fact for predicting the next model's score.

Figure 22. Six kinds of jump, three kinds of verdict. Teal: largely a property of the metric. Saffron: survives at the level of parts. Coral: survives continuous scores. Lavender: a trend reversal that rescoring does not address. summary of results

Objections

Against this paper

A strong critic would make four objections. The first is that my instruments are toys. A two-layer perceptron adding five-digit numbers and a one-layer network learning parity are far from a language model with billions of parameters trained on text. That is true. The toys were chosen because I could open them and check the mechanism behind each curve, which no one can yet do for a large model. Their job is to show that each kind of curve can arise, not how common each kind is.

The second objection is that my families vary only width, at a fixed training budget, while real families vary size, data and compute together. A wider network trained for the same number of steps is not simply a "larger" model in the sense of the scaling literature. That is also true, and it means my emergence points are points in width at 4,000 or 2,000 steps, not in compute. Du and colleagues' use of pre-training loss as the axis is a partial answer to this, since it absorbs the different ways of making a model better.

The third objection is that my audit's threshold, three times the fitting error plus half a unit, is arbitrary, and that a different threshold would flag different breaks. It would, at the margins. Both calibration families sit far from the threshold, so the verdicts on them would not change. For any real family close to the threshold, the verdict should be reported as uncertain, not as smooth or broken.

The fourth objection is philosophical. Bringing in satkāryavāda, Anderson and a story about a lover may seem to dress up a statistical point. I accept that the statistical point stands without them. What the older debates add is a warning that the question "was it already there?" has no answer until one fixes the level of description. The emergence debate in machine learning went round in circles for a year partly because the two sides were answering at different levels.

Falsifiers

What would change my mind

I would give up the claim that most famous emergent jumps are softened by rescoring under one condition. A careful reanalysis of a large public model family, with dense sampling and high resolution, would have to find that most claimed emergent tasks still show breaks in their continuous scores. The observational scaling work of Ruan and colleagues points the other way, but it is one study.

I would give up the claim that some jumps survive if the parity jump in my networks turned out to be smooth in training compute as well as in width, once the step budget was adjusted for each size. The hidden-progress results of Barak and colleagues make me think it would not, since their transitions were sharp in training time. I would also be moved by evidence that Du and colleagues' loss thresholds disappear under a different choice of loss axis.

For the adder family itself, two results would change my reading. If the tens digit rose gradually once the networks were trained for four times as long, the switch would be an artefact of my fixed training budget, and I would describe it as a speed effect, not a threshold. If a transformer trained on the same task learned its digits in a different order, my account of the magnitude route would be a fact about perceptrons, not about adders in general. Prediction P20-1 in the ledger puts the first of these to a test that anyone can run.

On the RTM side, I would take the recognition framing of Part 6 more seriously if a model were found with an internal variable that marks its own acquisition of a skill. The variable would also have to be used, for instance to decide when to attempt a task. That would be a step towards something like L3. I know of no such finding.

Figure 23. What I do not know, placed by how soon it could be settled. Placement is my judgement. author judgement

Figure 24. How sure I am of each claim, at a glance. Bar length is my subjective confidence; pattern and colour give the ledger category, so the chart reads in greyscale. author judgement

Ledger

Claim Ledger

Each main claim of this paper, with how sure I am, what kind of evidence carries it, and what would knock it down.

ID	Claim	Confidence	Evidence type	What would refute it
C1	Many exact-match emergence curves are produced largely by compounding per-token accuracy.	established	Schaeffer et al. (2023); this paper: product of per-digit scores within 0.2 points of exact match at all 11 widths	A family where exact match jumps while every per-token score and their product stay flat
C2	Small test sets record rare successes as zero, which makes later rises look sudden.	established	Schaeffer et al. (2023); this paper's resolution grid	Flat zeros that persist with 100 times more test items
C3	The scale at which an ability "emerges" moves with answer length and scoring rule.	established	Mathematics of compounding; this paper: 2.5, 5.4 and 7.2 units for 1, 10 and 50 tokens	An emergence point that stays fixed as answer length changes, with per-token accuracy fixed
C4	In my adder family, continuous scores also break because separate digits switch on sharply at different widths.	likely	This paper: audit misses of 5.4 log-odds on digit share; tens digit 0.10 at 128 units, 0.59 at 192; three seeds	Longer training or denser widths turning each digit curve into a gradual ramp
C5	Sparse parity shows jumps that survive every output score.	established	Barak et al. (2022); this paper: loss at 0.69 for up to 5,900 steps, then below 0.1 within 700	A continuous output score that rises steadily before the jump
C6	Some abilities of real language models jump against pre-training loss even under continuous metrics.	likely	Du et al. (2024)	Reanalysis showing those thresholds vanish under a different loss axis or denser sampling
C7	Many emergent abilities of real models are smooth sigmoids that can be forecast from smaller models.	likely	Ruan et al. (2024)	Forecasts from small models failing on newly released families
C8	Steepness is a poor test of emergence; a new internal variable is a better one.	likely	Anderson (1972); Krakauer et al. (2025); this paper's parity and digit results	A useful notion of emergence that depends only on curve shape
C9	Under RTM, these jumps are discrete changes in what is shown (L2) with a recognition-like shape (L4) and no reflexivity (L3).	speculative	Conceptual analysis under RTM	A model component that tracks its own acquisition of a skill and uses it

Dated predictions

ID	Prediction	Judged by	How it will be judged
P20-1	Retraining this paper's adder family for 16,000 steps instead of 4,000 will leave the order in which digits reach 50 per cent accuracy unchanged in at least two of three seeds: leading and ten-thousands first, tens last.	This paper's scripts, rerun by me or anyone	Run e20b_family.py with 16,000 steps and compare the width at which each digit first exceeds 0.5
P20-2	By the end of 2028, a published study of a public language-model family with at least six sizes will report a task where per-token log-probability of the correct answer stays within 10 per cent of its smallest-model value across at least three sizes, then improves by more than half within one further size.	Peer-reviewed or widely cited work	Literature search on 31 December 2028

Words

Glossary

Emergent ability: An ability absent in smaller models and present in larger ones, with a sharp change between.
Exact match: A score that gives credit only if the whole answer is exactly right.
Per-token accuracy: The share of the pieces of an answer (tokens, digits, words) that are correct.
Token edit distance: How many tokens must be changed to turn an answer into the right one.
Brier score: A score for probabilistic answers: the squared gap between the probabilities given and the truth.
Multiple Choice Grade: A score that gives credit if the option with the highest probability is the right one.
Resolution: The smallest difference a measurement can detect; for a benchmark, roughly one over the number of items.
Log-odds: The logarithm of p divided by 1 minus p; it turns an S-shaped rise into a straight line.
Scaling law: A formula for how a model's loss changes with size, data or compute.
Pre-training loss: A model's error on its main training objective, usually predicting the next token of text.
Sparse parity: A task whose label is whether a hidden few of many bits contain an odd number of ones.
Quanta: In Michaud et al., discrete chunks of skill that a network learns one at a time.
Inverse scaling: Getting worse at a task as models get larger.
satkāryavāda सत्कार्यवाद: Sanskrit: the doctrine that the effect exists in its cause before it is produced; held by Sāṃkhya.
asatkāryavāda असत्कार्यवाद: Sanskrit: the doctrine that the effect does not exist before it is produced; held by Nyāya and Vaiśeṣika.
ārambhavāda आरम्भवाद: Sanskrit: the theory that production is a real beginning of a new thing.
avayavin अवयविन्: Sanskrit: a whole made of parts, held by Nyāya to be distinct from them.
ābhāsavāda आभासवाद: Sanskrit: the Trika theory that all things are appearances shown by consciousness.
Sāṃkhya साङ्ख्य: One of the six classical schools; dualist, with a theory of causation as transformation.
Nyāya न्याय: The classical school of logic and epistemology, allied with Vaiśeṣika.
spanda स्पन्द: Sanskrit: vibration, responsiveness; RTM layer L1.
prakāśa प्रकाश: Sanskrit: light, manifestation; RTM layer L2.
vimarśa विमर्श: Sanskrit: reflective awareness; RTM layer L3.
pratyabhijñā प्रत्यभिज्ञा: Sanskrit: recognition; knowing a present thing as one known before. RTM layer L4.

Conclusion

What survives rescoring

This paper asked which emergent abilities survive continuous-metric reanalysis. In brief, the sharpness of many celebrated jumps belongs to the pass mark. Exact match compounds per-token accuracy, small benchmarks round rare successes down to zero, and a family of three or four models cannot show what happens between them. Rescoring with partial credit, adding test items and sampling scale more densely remove a large share of the surprise.

What survives falls into two kinds. The first is the genuine threshold, where a continuous score itself sits at chance and then leaves it: my parity networks, Imran's touch-typing, and the loss thresholds reported by Du and colleagues. The second is emergence at a finer grain. The average improves smoothly while each component switches on at its own scale, as the digit positions did in my addition networks. Michaud and colleagues' quanta are a theory of exactly this kind.

Figure 25. The paper on one page. Pin it next to the next claim that a model "suddenly" acquired a skill. summary

Neither kind is a mystery. Both become intelligible once the right quantity is tracked: the per-digit accuracy in the addition networks, and in the parity networks the waiting time before the jump, which shrinks as the network widens. The practical message is to record continuous scores, report resolution and decompose tasks into parts. These habits cost almost nothing, and they turn an emergence claim into a measurement that can be checked.

The older debates leave a wider lesson. The question "was the ability already in the smaller model?" has the answer yes at one level and no at another, as the Sāṃkhya and Nyāya philosophers might have agreed if they had been shown the same register. The Pratyabhijñā story adds that a change can be discrete at one layer and continuous at the layer below. In my networks, the discrete changes were in structure. Nothing recognised them from inside.

This paper closes the group of six numbered 15 to 20 in the series, all on how machines think. Across them, a pattern has recurred. What looks like sudden understanding from outside, in attention, the residual stream, induction heads, in-context learning, grokking and now scale, turns out inside to be a structure assembled step by step, with an abrupt change in what it can show. The harder question, what it would mean for a machine to know that it understands, remains open, and later papers in the series take it up.

[AUTHOR NOTE: optionally close with a sentence on what this batch of papers has changed in how you evaluate models]

Evidence

Sources

Primary research

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R. and others (2020). Scaling laws for neural language models. arXiv:2001.08361. Verified; abstract checked.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E. and others (2022). Training compute-optimal large language models. arXiv:2203.15556. Verified.
Ganguli, D., Hernandez, D., Lovitt, L., DasSarma, N., Henighan, T., Jones, A. and others (2022). Predictability and surprise in large generative models. arXiv:2202.07785. Verified; abstract quoted.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A. and others (2022). Beyond the imitation game: quantifying and extrapolating the capabilities of language models. arXiv:2206.04615. Verified; abstract quoted.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S. and others (2022). Emergent abilities of large language models. arXiv:2206.07682. Verified; abstract quoted.
Barak, B., Edelman, B. L., Goel, S., Kakade, S., Malach, E. and Zhang, C. (2022). Hidden progress in deep learning: SGD learns parities near the computational limit. arXiv:2207.08799. Verified; abstract quoted.
Michaud, E. J., Liu, Z., Girit, U. and Tegmark, M. (2023). The quantization model of neural scaling. arXiv:2303.13506. Verified; abstract quoted.
Schaeffer, R., Miranda, B. and Koyejo, S. (2023). Are emergent abilities of large language models a mirage? arXiv:2304.15004. Verified; abstract and main text quoted.
McKenzie, I. R., Lyzhov, A., Pieler, M., Parrish, A., Mueller, A., Prabhu, A. and others (2023). Inverse scaling: when bigger isn't better. arXiv:2306.09479. Verified; abstract quoted.
Du, Z., Zeng, A., Dong, Y. and Tang, J. (2024). Understanding emergent abilities of language models from the loss perspective. arXiv:2403.15796. Verified; abstract quoted.
Ruan, Y., Maddison, C. J. and Hashimoto, T. (2024). Observational scaling laws and the predictability of language model performance. arXiv:2405.10938. Verified; abstract quoted.
Krakauer, D. C., Krakauer, J. W. and Mitchell, M. (2025). Large language models and emergence: a complex systems perspective. arXiv:2506.11135. Verified; abstract quoted.
Anderson, P. W. (1972). More is different. Science 177 (4047), 393 to 396. doi:10.1126/science.177.4047.393. Paraphrased.

Announcements

NeurIPS Communications Chairs (2023). Announcing the NeurIPS 2023 paper awards. 11 December 2023. blog.neurips.cc. Source of the award story.

Primary texts and background

Īśvarakṛṣṇa, Sāṃkhyakārikā, verse 9, the five reasons for satkāryavāda. Standard editions and translations; paraphrased.
Nyāya and Vaiśeṣika sources on asatkāryavāda and the whole (avayavin), for example the Nyāyasūtra with Vātsyāyana's commentary, book 4. Standard editions; paraphrased.
Utpaladeva, Īśvarapratyabhijñākārikā with his own commentary (vṛtti): the section on causation in book 2, and the example of the lover who does not recognise her beloved, near the end of the work. Standard editions, for example R. Torella (1994), IsMEO; paraphrased.
Abhinavagupta, Īśvarapratyabhijñāvimarśinī, on ābhāsavāda. Standard editions; paraphrased.

Reference

Cite this work

Mahapatra, R. R. (2026). The pass mark and the pupil: which emergent abilities survive continuous-metric reanalysis?. Mind Frontier paper U0370. rajeshrmahapatra.com. https://rajeshrmahapatra.com/papers/mind-frontier/U0370-emergent-abilities-continuous-metrics.html. DOI: to be assigned.

@misc{mahapatra2026emergent,
  author = {Mahapatra, Rajesh Ranjan},
  title = {The pass mark and the pupil: which emergent abilities survive continuous-metric reanalysis?},
  year = {2026},
  howpublished = {Mind Frontier paper U0370, rajeshrmahapatra.com},
  url = {https://rajeshrmahapatra.com/papers/mind-frontier/U0370-emergent-abilities-continuous-metrics.html},
  doi = {10.xxxx/placeholder}
}