The fading slate: does weight decay make a network grok by pruning its lookup table?: Rajesh Ranjan Mahapatra

The scene

Two pupils, one answer sheet and a slate that fades

Mrs. Sunita Rao teaches arithmetic to a class of twelve-year-olds at a school in Mysuru. Each term she sets one puzzle that runs for weeks. This term it is clock sums on a clock with 59 hours. Three o'clock plus fifty-seven hours is one o'clock, because the hand passes 59 and starts again. She gives the class a sheet of 1,392 solved sums, about two in five of all the pairs, and keeps the rest back for a test at the end.

Kiran, who is quick, memorises the sheet. Within two days he can answer every sum on it, and he is proud of it. On the hidden sums he scores nothing at all, because he has learned answers, not clocks. Asha is slower. She also starts by memorising, but Mrs. Rao has a rule in her class: work is done on slates, and the slates are wiped a little every evening. Whatever is not used again fades. Kiran rewrites his whole table each morning. Asha, tired of rewriting, starts to notice that some sums share a pattern. After three weeks of no visible progress, she gets the hidden sums right almost overnight.

Mrs. Rao, Kiran and Asha are invented, though the clock is real and so are the numbers in this paper. They come from small networks I trained on exactly Mrs. Rao's puzzle. A network trained with the fading slate behaved like Asha. A network trained without it behaved like Kiran for 20,000 steps and never changed.

Figure 1. Same sheet, same marks, different minds. Mrs. Rao, Kiran, Asha and the fading slates are invented. They stand for memorisation, weight decay and the generalising rule in a small trained network. illustrative

The pattern is called grokking: a network fits its training data perfectly, appears to stop learning, and then, long afterwards, suddenly starts to get unseen examples right. The question this paper asks is the one in the topic list: does delayed generalisation in modular arithmetic arise from L2 weight decay pruning high-norm memorising lookup tables? Weight decay is the fading slate, a small pull on every weight towards zero at every step of training.

My answer is a qualified yes. In my networks, weight decay was necessary: without it, nothing generalised. The memorising network was large in the sense that matters, with a weight norm three times higher at the end. The generalising one had a small norm and stored the clock in a handful of waves. But the pruning picture is incomplete. A network can have a small norm and still generalise nothing, and other routes to grokking exist that use no weight decay at all. Weight decay favours the cheaper solution; it does not build it.

Part 1

What grokking is, and why clock sums

Most of what we know about training neural networks says that the important things happen early. A network's error on its training data falls; its error on held-out data falls with it for a while, then levels off or rises as the network starts to fit noise. Engineers stop training around that point. The habit is so strong that it has a name, early stopping, and a whole literature on when to apply it.

Grokking breaks the habit. The training error reaches zero early. The held-out error stays high, sometimes for a very long time. Then, with no change to the data or the settings, the held-out error drops to zero as well. The word comes from Robert Heinlein's 1961 novel Stranger in a Strange Land, where to grok something is to understand it so completely that it becomes part of you.

Why it matters beyond clock sums

Grokking matters because it touches one of the oldest puzzles in deep learning. Modern networks have far more weights than training examples. In principle, such a network can memorise any training set, including one with random labels, and many do. Yet in practice they usually generalise. Something in training steers them away from pure memorisation, towards solutions that work on new data. Nobody has a complete account of what that something is.

Grokking separates the two outcomes in time. In most training runs, memorising and generalising happen together and cannot be told apart. In a grokking run, the network memorises first and generalises much later, with a long stretch in between where it holds a table and nothing else. That stretch is a laboratory. Whatever changes during it is, almost by definition, the thing that turns memorisation into generalisation. Grokking slows down the moment we most want to see.

A puzzle with a clean answer

Clock sums are the standard test, for a reason. Pick a whole number, here 59, and imagine a clock with that many hours. The sum of two hours is the hour you land on, so 30 plus 40 is 11, because 70 is 11 more than 59. Mathematicians call this addition modulo 59. There are 59 times 59, or 3,481, possible sums, so the whole task fits in a table.

That small size is what makes the puzzle useful. A network can learn it in two very different ways. It can memorise the table, one answer per pair. Or it can learn the rule, the fact that hours go round a circle, and compute any answer from it. On the training sums, both ways get everything right. Only the held-out sums tell them apart.

Figure 2. Clock sums: a task small enough to fit in a table. Addition modulo 59. Power et al. (2022) used 97 hours and transformers; Nanda et al. (2023) used 113. My network is a two-layer MLP with about 88,000 weights, trained with AdamW at learning rate 0.001. mechanism diagram

In my setup, each of the 59 hours gets its own list of 128 numbers, called an embedding, which the network learns. The two embeddings of a sum are joined, passed through one hidden layer of 256 units, and turned into a guess over the 59 possible answers. The network has about 88,000 adjustable weights. I trained it on 40 per cent of the table and kept the other 60 per cent hidden.

A minimal worked case

The difference between the two ways shows up even on a tiny clock. Take a clock with 5 hours. There are 25 sums. Here is how each pupil stores them.

Kiran writes a table with 25 cells, one answer in each. To answer "3 plus 4", he looks up the cell and reads 2. Each cell is a separate fact. Nothing connects cell (3, 4) to cell (4, 3).
Asha draws a circle with 5 marks. To add, she turns the hand 3 marks, then 4 more, and reads where it stops. That is one fact, about turning, used 25 times.
Suppose only 10 of the 25 sums are on the sheet. Kiran can answer exactly those 10. Asha, if she has found the circle, can answer all 25.
Now suppose every fact costs something to keep, and the cost grows with how much there is to remember. Kiran's table costs 10 units. Asha's circle costs about 1, plus whatever it takes to find it.

Weight decay puts a price on storage. Each weight shrinks a little at every step, so a weight survives only if the training data keep pushing it back up. A large table of separate facts needs many large weights. A compact rule needs few. The pruning hypothesis says that weight decay slowly makes the table too expensive to keep, and the rule wins by default.

Figure 3. A table stores answers; a circle stores the reason for them. The worked case from Part 1, on a 5-hour clock. Only the cells on the training sheet can be filled in the table; the circle answers the empty cells too. worked example

Thought experiment 1

The pupil who already has full marks

Think of Kiran and Asha on the third day of term. Both answer every sum on the sheet correctly. Neither answers any hidden sum. Mrs. Rao could stop the exercise there and mark them both as having learned the sheet.

Pause. Before you read on: if training continues on the same sheet, with the same scores already perfect, should anything change?

Most people say no, and the textbooks agree. If the training error is zero, there is no error left to drive learning, so the network should sit still. In my networks without weight decay, that is roughly what happened, and the hidden sums stayed at zero for 20,000 steps. With weight decay, it is wrong. The training error was zero from step 100, but the hidden sums went from 0 to 50 per cent by step 5,500 and to 95 per cent by step 8,200.

The lesson is that perfect training accuracy is not the end of learning when something other than the error is also being optimised. Weight decay keeps adding a second goal, small weights, and the network keeps moving to serve it. The limit is that the second goal must exist. A network with nothing but the error to reduce has no reason to change once the error is gone, and mine did not.

Part 2

A forgotten training run, and the explanations that followed

The discovery was an accident, and it is the first true story in this paper. A team at OpenAI led by Alethea Power was training small transformers on clock sums with a clock of 97 hours. As expected, the networks memorised their training sums and failed the held-out ones. Then, as Power told a conference audience in 2022, "we got lucky", adding, "and by lucky, I mean forgetful". According to Quanta Magazine, the team member running the training went on vacation and forgot to stop it. When the team looked again, the network had become accurate on data it had never seen.

The paper that followed, by Power and colleagues (2022), showed that this "improvement in generalization can happen well past the point of overfitting". It also found that "smaller datasets require increasing amounts of optimization for generalization". The authors tried several kinds of regularisation and found weight decay the most effective at bringing generalisation forward. That observation is the seed of the hypothesis this paper tests.

Figure 4. From a forgotten run to a family of explanations in two years. Dates are first public versions. Saffron: norm and efficiency accounts. Teal: mechanism studies. Coral: routes that need no weight decay, or wider settings. sourced

What the network had found

The second true story is the one that opened the network up. Nanda and colleagues (2023) took a small transformer that had grokked clock sums with 113 hours and worked out exactly what it computed. The answer surprised almost everyone. The network had learned to place each hour on a few circles, each turning at a different speed, using sines and cosines. To add two hours, it combined their positions using the trigonometric identities taught in school, which turn addition into rotation. They called it an algorithm that uses "discrete Fourier transforms and trigonometric identities to convert addition to rotation about a circle".

A Fourier transform breaks a pattern into waves of different speeds. If an hour's embedding is made of only a few waves, the network has found the circle. If it is spread evenly over all of them, it has not. That gives a measure anyone can compute, and I use it in Part 3: the share of an embedding's power held by its five strongest waves.

Nanda and colleagues went further. Knowing the mechanism, they could track it throughout training, and found that "grokking, rather than being a sudden shift, arises from the gradual amplification of structured mechanisms encoded in the weights, followed by the later removal of memorizing components". They named three phases: memorisation, circuit formation and cleanup. The sudden jump in held-out accuracy comes at cleanup, when the memorised parts are finally removed and stop interfering.

Five accounts of why it waits

Several explanations followed, and they overlap. Liu and colleagues (2022) mapped the settings in which grokking happens and found it in a "Goldilocks zone" between pure memorisation and confusion. In a later paper called Omnigrok, Liu, Michaud and Tegmark (2022) plotted training and test loss against the size of the weights. Training loss looks like an "L" and test loss like a "U". A network that starts with large weights fits its training data at once but sits on the high side of the U, and weight decay slowly walks it down to the bottom. They called this the "LU mechanism", and used it both to induce grokking on images and to eliminate it on clock sums.

Varma and colleagues (2023) gave the most direct version of the pruning hypothesis. A generalising circuit, they argued, is "slower to learn but more efficient, producing larger logits with the same parameter norm". Memorising circuits get less efficient as the training set grows, so there is a critical dataset size at which the two break even. They confirmed striking predictions, including "ungrokking", in which a network that has generalised goes back to memorising when retrained on a smaller dataset.

Two accounts need no weight decay at all. Thilak and colleagues (2022) found that adaptive optimisers such as Adam go through cycles of instability late in training, which they called the slingshot mechanism. "Without explicit regularization", grokking "almost exclusively happens at the onset of Slingshots". Kumar and colleagues (2023) showed grokking in plain gradient descent with no regularisation. On their view, the network first fits with its initial features, in a "lazy" regime, and only later learns new features, in a "rich" one.

Finally, the solution is not always the same. Zhong and colleagues (2023) found that networks on clock sums sometimes learn Nanda's "Clock" algorithm and sometimes a different one they called "Pizza", depending on small changes to settings. And Humayun, Balestriero and Baraniuk (2024) argued that grokking is "much more widespread" than toy tasks suggest, appearing in image classifiers trained for long enough.

One phrase from this literature is worth keeping in mind. Liu and colleagues described the zone in which networks learn real structure as reminiscent of "intelligence from starvation" in evolution, "where resource limitations drive discovery of more efficient solutions". Weight decay is one such limitation. The question is whether starvation alone is enough, or whether something else must happen while the network goes hungry.

Figure 5. The accounts differ on two questions: is decay needed, and who builds the rule? Placements are my judgement. Sources: Liu et al. (2022, Omnigrok); Varma et al. (2023); Kumar et al. (2023); Thilak et al. (2022). illustrative, author judgement

The Clock and Pizza result is a useful warning for this paper. I did not check which of the two algorithms my network learned, or whether it learned a mix. My wave measure would rise for either, since both rest on Fourier features. The claims I make below are about the competition between a table and some compact rule. They do not depend on which compact rule it was.

So the field holds a cluster of related answers. On one, weight decay prunes the memorised table and the rule is what remains. On others, the rule is built by a process that weight decay merely speeds up, or by an optimiser quirk, or by a delayed switch to learning features. The experiments in Parts 3 and 4 try to tell these apart in one small network.

Part 3

Three networks, one sheet of sums

To test the pruning hypothesis directly, I trained the same small network three times on the same 1,392 sums, changing only the strength of weight decay. With a setting of 0 there was no decay. With 1, each weight shrank by about one part in a thousand at every step, before the data pushed back. With 3, the pull was three times stronger. Everything else, including the starting weights, was identical. I measured four things every 100 steps: accuracy on the training sums, accuracy on the hidden sums, the overall size of the weights, and how much of each embedding was held by its five strongest waves.

The size of the weights is measured by their norm, the square root of the sum of every weight squared. It is a single number that grows when weights grow. At the start, my networks had a norm of about 88. The wave measure starts at about 0.20, close to what random embeddings give, since five waves out of 29 would hold about 0.17 of the power by chance.

How the measurements were made

Two of the four measurements need a word of explanation. The wave measure takes each of the 128 columns of the embedding table, treats it as a pattern over the 59 hours, and splits it into 29 waves of different speeds, plus a constant. It then adds up the power of each wave across all columns, and reports the share held by the five strongest. A network that has found the circle puts most of its power into a few speeds. A network that has not spreads it evenly.

The ablation, used in Thought Experiment 2 and Part 4, does the reverse. It rebuilds each embedding from only some of its waves and runs the network as before. If the network still works on the reduced embeddings, the waves that were kept are sufficient. If it fails when they are removed, they are necessary. Both measurements follow the methods Nanda and colleagues used on transformers, simplified for a smaller network.

The network with no decay

Without decay, the network behaved exactly like Kiran. It got every training sum right by step 100. It got every hidden sum wrong, and kept getting them wrong for 20,000 steps. Its accuracy on hidden sums was actually 0.0, below the 1.7 per cent a random guess would score, because it confidently gave the answer from a memorised neighbour. Meanwhile its norm kept growing, from 88 to 110 by step 1,000 and to 257 by step 20,000. The wave measure barely moved, from 0.20 to 0.21.

Figure 6. Without decay: perfect on the sheet, zero on the test, and swelling. No weight decay, 40 per cent of sums for training. The dashed line is the weight norm divided by 300, so it fits the same axis: 88 at the start, 257 at the end. Run in this paper's instrument. measured, synthetic data

The growing norm needs explaining, because the network had nothing left to learn. The reason is in how training errors are scored. The usual score, cross-entropy, rewards a network for being more confident in a right answer even when it already ranks that answer first. The cheapest way to be more confident is to make every weight a little larger. A memorising network with no brake therefore inflates itself forever. Kiran rewrites his table each morning in thicker and thicker chalk.

This run is the negative control for the whole paper. If anything in the setup other than weight decay could produce generalisation, this run would have had 20,000 steps to show it. It did not.

The network with decay

With a decay of 1, the network also got every training sum right by step 100, and also failed every hidden sum at first. Then the two networks parted. The norm of the decayed network fell, from 88 to 72 by step 1,000 and to 60 by step 6,000. At step 3,000 it got 3.5 per cent of hidden sums right. At step 4,000, 16 per cent. At 5,000, 38 per cent. It crossed 50 per cent at step 5,500 and 95 per cent at step 8,200, and by step 12,000 it was perfect.

Figure 7. With decay: the same memorising start, then the test sums arrive. Weight decay 1.0, same data and starting weights. Hidden accuracy: 0.035 at step 3,000, 0.63 at 6,000, 0.94 at 8,000, 1.0 by 12,000. Dashed: norm divided by 100. Run in this paper's instrument. measured, synthetic data

Over the same period the wave measure rose steadily, from 0.20 at the start to 0.29 at step 6,000, 0.42 at step 12,000 and 0.52 at step 20,000. The embeddings were becoming made of a few waves. That is the signature of the circle that Nanda and colleagues found in transformers, appearing here in a much simpler network.

Seen at the scale of the whole run, the jump in hidden accuracy looks sudden: near zero for a quarter of the run, then near perfect. Seen at the scale of a few hundred steps, it is not sudden at all. The climb from 4 per cent to 94 per cent took about 5,000 steps, and the wave measure was rising the whole time. The suddenness is partly a matter of where you stand.

Figure 8. Without a brake, a memoriser inflates; with one, the norm settles low. Square root of the sum of all squared weights. Decay 3 dips to 54 at step 500, rises to 66 by step 2,000 as the rule forms, then settles at 57. Run in this paper's instrument. measured, synthetic data

The network with a decay of 3 told the same story with a twist. It also generalised, crossing 50 per cent at step 4,600, earlier than the network with a decay of 1. It took longer to reach 95 per cent, at step 10,100, and ended with a lower wave measure, 0.38 against 0.52. Stronger decay is not simply faster grokking. It shrinks everything, including the parts of the rule that need to be large.

Figure 9. With decay, the embeddings become a few waves: the circle forms. For each run, the share of embedding power in the five strongest of 29 Fourier frequencies. The dotted line marks an even spread, 0.17. Run in this paper's instrument. measured, synthetic data

The interactive below lets you put the three runs side by side. Choose a decay setting and the chart shows accuracy on training and hidden sums, with the norm on the same time axis.

Static view: without decay, training accuracy is perfect from step 100 and hidden accuracy stays at zero for 20,000 steps while the norm rises to 257. With decay 1, hidden accuracy passes 95 per cent at step 8,200 and the norm falls to 53. With decay 3, hidden accuracy passes 50 per cent at step 4,600 and 95 per cent at step 10,100.

Figure 10. Choose a decay setting and compare the three runs. Teal: training accuracy. Indigo: hidden accuracy. Coral dashed: weight norm, scaled so 300 sits at the top. All runs share data and starting weights. Run in this paper's instrument. interactive, measured

Watching one run step by step

The second interactive follows the decay-1 run alone. Drag the slider through training and read off the four measurements at each point. The most instructive stretch is between steps 2,000 and 5,000. Training accuracy is perfect and does not change. The hidden accuracy is still small. But the norm is falling and the wave measure is rising. A person watching only the training score would see nothing happening. A person watching the weights would see the rule taking shape.

Static view of the decay-1 run: at step 2,000, training accuracy 1.0, hidden 0.002, norm 66.5, top-five wave share 0.22. At step 5,000: 1.0, 0.375, 61.3, 0.27. At step 8,000: 1.0, 0.94, 58.2, 0.34. At step 20,000: 1.0, 1.0, 52.7, 0.52.

Training step: 0

Figure 11. Drag through the decay-1 run and watch the inside change before the score does. Four bars at the chosen step: training accuracy, hidden accuracy, norm (as a share of the starting 88) and top-five wave share. Between steps 2,000 and 5,000 the training score is fixed while the other bars move. Run in this paper's instrument. interactive, measured

Thought experiment 2

Two pupils with the same marks

On the third day of term, Kiran and Asha hand in the same answer sheet with every sum correct. A school inspector looking only at the marks cannot tell them apart. Mrs. Rao can, because she has watched them work. Two of my networks were in the same position: at step 12,000 the no-decay network and the decay-1 network both scored 100 per cent on the training sums.

Pause. Before you read on: if you could not see the hidden sums, is there any way to tell from the training sums alone which network has found the rule?

From the training answers, no. Both are perfect, and both would score the same on any test drawn from the sheet. From the inside, yes, and easily. I took each network's embeddings, broke them into their 29 waves, kept only the strongest few, and threw away the rest. The decay-1 network, keeping only 8 of its 29 waves, still got 100 per cent of training sums and 98 per cent of hidden ones right. The no-decay network, keeping its 8 strongest, fell to 11 per cent on its own training sums.

The lesson is that the same behaviour on the data you have can rest on opposite mechanisms, and only the mechanism predicts behaviour on data you do not have. The limit is that I could run this test because I knew what the rule should look like. For most real tasks there is no known Fourier basis to check against, and the inside of the network is much harder to read.

Part 4

Is it pruning? Four tests

The pruning hypothesis makes four claims that can be tested one at a time. First, that the memorising solution has a high norm. Second, that weight decay removes it and leaves the rule. Third, that reducing the norm is enough to produce generalisation. Fourth, that weight decay is needed at all. My networks gave a clear answer to each, and they were not all the same answer.

Figure 12. Four tests, three different verdicts. Verdicts from Part 4. Each rests on one or more runs in this paper's instrument. summary of results

Test 1: the cost of the table

In one sense, plainly yes. The no-decay network ended with a norm of 257, against 53 for the decay-1 network, which by then was perfect on both training and hidden sums. The memorised table carried five times the norm of the rule at the end of training.

But a further experiment showed that the high norm was not something the table needed. I trained the network with no decay, but after every step I rescaled all its weights so the norm stayed fixed at 55, the level the decay-1 network ended near. That network still memorised every training sum within 1,000 steps, while getting the hidden sums wrong. So a lookup table can be stored at low norm. Its high norm in the no-decay run came from inflation, the endless push for confidence, not from the table itself. The table is not big; it is greedy.

Test 2: what decay leaves behind

Here the answer is yes. The wave ablation from Thought Experiment 2 is the clearest evidence. In the decay-1 network at step 12,000, the strongest wave held 10 per cent of the embedding's power and the top 8 held 60 per cent. Keeping only those 8 left it perfect on training sums and 98 per cent right on hidden ones. Removing those 8 and keeping the other 21 dropped hidden accuracy to 19 per cent.

The no-decay network was the mirror image. Its strongest wave held 4 per cent of the power, hardly more than an even share. Keeping its top 8 waves wrecked it, dropping training accuracy to 11 per cent, and removing them dropped it to 30 per cent. Everything mattered a little, so no small part was enough. That is what a table looks like through a Fourier lens.

Figure 13. The rule lives in a few waves; the table lives everywhere. Embeddings rebuilt from only the chosen frequencies (plus the constant). Decay-1 network at step 12,000, scored on hidden sums. No-decay network at step 6,000, scored on its own training sums, since it has no hidden accuracy to lose. Run in this paper's instrument. measured, synthetic data

The third interactive lets you run the ablation yourself on the two networks. Choose how many waves to keep, and see what each network can still do.

Static view: keeping only the 8 strongest waves, the decay-1 network scores 1.00 on training and 0.98 on hidden sums; the no-decay network scores 0.11 on training. Dropping those 8 waves, the decay-1 network falls to 0.19 on hidden sums.

Figure 14. Remove waves yourself and see which network survives. "Keep" rebuilds each embedding from only its strongest waves; "drop" removes them and keeps the rest. Run in this paper's instrument. interactive, measured

Test 3: whether a small norm is enough

This is where the simple version of the hypothesis fails, and it fails twice. The first counterexample came from the network with a decay of 3. Its norm fell very fast, to 54 by step 500, about where the decay-1 network would be at step 15,000. If small norm were enough, it should have generalised then. At step 500 it got 0 per cent of hidden sums right. Its norm then rose again, to 66 by step 2,000, as the rule began to form, and only after that did hidden accuracy climb.

The second counterexample is the fixed-norm network from Test 1. Its norm was 55 from the very first step. Yet hidden accuracy stayed under 1 per cent until step 2,000, reached 11 per cent at step 4,000 and 42 per cent at step 8,000. The delay was still there, even though there was nothing large to prune. A copy of the same network held at its starting norm of 88 got 0.2 per cent of hidden sums right after 8,000 steps.

Figure 15. A small norm from the start still has to wait: the rule takes time to build. After every step, all weights rescaled to a fixed total norm; no weight decay. Both fixed-norm runs memorised the training sums within 1,000 steps. Hidden accuracy at step 8,000: 0.42 at norm 55, 0.002 at norm 88, 0.94 with decay 1. Run in this paper's instrument. measured, synthetic data

Taken together, these results separate two things the hypothesis runs together. A small norm is a condition that lets the rule win. The rule itself has to be built, and building it takes thousands of steps whatever the norm. Nanda and colleagues called this phase "circuit formation", and in my networks it was the slow part. The delay is mostly construction time, not demolition time.

Test 4: whether decay is needed at all

In my setup, some limit on the norm was needed. The no-decay network never generalised in 20,000 steps, and the network held at a norm of 88 barely began in 8,000. Either weight decay or a fixed small norm produced generalisation. That fits the Omnigrok picture of Liu, Michaud and Tegmark, in which what matters is the size of the weights, however it is controlled.

In general, though, decay is not needed. Thilak and colleagues found grokking without any explicit regularisation, driven by the late instability of the Adam optimiser. Kumar and colleagues found it in plain gradient descent with no regularisation, from a slow move away from the starting features. My networks used AdamW with standard settings and did not show either effect within 20,000 steps, but neither result is in doubt for the settings those authors used. Weight decay is one route to grokking, not the only one.

The failure boundary: too few sums

The last experiment varied the size of the sheet. With decay of 1, I trained on 20, 30, 50 and 70 per cent of all sums, for 6,000 steps each. With 70 per cent, the network reached 95 per cent on hidden sums by step 1,000. With 50 per cent, by step 3,200. With 40 per cent, the main setting, by step 8,200 in the longer run. With 30 and 20 per cent, it never generalised at all within 6,000 steps, ending at 0 per cent.

Figure 16. Below a third of the sums, the table never loses. Weight decay 1, 6,000 steps, one seed each. Step at which hidden accuracy first reached 95 per cent: 1,000 at 70 per cent, 3,200 at 50 per cent, 8,200 at 40 per cent (longer run), never at 20 or 30 per cent. Run in this paper's instrument. measured, synthetic data

This is the pattern Power and colleagues reported, and it fits the efficiency account of Varma and colleagues neatly. A table grows with the number of training sums, while the rule stays the same size. With many sums, the table is expensive and the rule wins quickly. With few, the table is cheap enough that decay cannot make it lose, at least within the time I allowed. Somewhere between 30 and 40 per cent lies the point where, in my network, the two break even.

Thought experiment 3

The pupil who forgot on purpose

Mrs. Rao's fading slates have an unexpected side effect. A boy in the back row, Pradeep, discovers that if he does nothing at all, his slate is wiped clean every evening and he never has to remember anything. He is very pleased with this. By the end of term he has a perfectly clean slate and knows no sums at all.

Pause. Before you read on: if fading is what makes Asha find the rule, why does it not make Pradeep find it too?

Because fading only removes. It does not add. Asha had a table to begin with, and the training sums kept pushing her to answer correctly. Fading made the table costly, so the cheaper rule, which also answered correctly, came out ahead. Pradeep had no pressure to answer anything. In network terms, weight decay with no data behind it simply drives every weight to zero, and a network of zeros knows nothing.

The lesson is that weight decay is a referee, not a player. It decides between solutions that fit the data, favouring the cheaper one. It cannot produce a solution on its own. The limit of the joke is that real decay is never alone: in training it always acts alongside the data, which is why "decay causes generalisation" is a tempting but loose summary.

How much to trust these numbers

Most runs in this paper used a single random start, so I repeated the main setting, decay of 1 on 40 per cent of the sums, with two more seeds. The first new seed reached 95 per cent on hidden sums at step 8,700 and the second at step 7,400, against 8,200 for the original. All three were perfect by step 12,000, with norms between 53 and 56. So the timing moved by about a thousand steps either way, and the shape of the story did not move at all.

One more setting is worth reporting because it shows a limit on the other side. With a weak decay of 0.3, the network's norm rose to 93 by step 1,000 and then fell only to 70 by step 6,000, still well above the decay-1 network's 60. After 6,000 steps it got 0.2 per cent of hidden sums right. Too little decay is, within a fixed budget of training, much the same as none. The fading slate has to fade fast enough to matter.

Part 5

Why the rule is cheaper than the table

The experiments show that a limit on the norm lets the rule win. They do not yet say why the rule is cheaper. The answer that best fits my data is the efficiency account of Varma and colleagues: for the same total size of weights, the rule produces more confident correct answers than the table does. Under a fixed budget, confidence is what the training score pays for, so the rule earns more per unit of weight.

Counting the cost

A rough count shows the difference. The table needs to store, somewhere in its weights, one association for each of the 1,392 training sums. Each association needs its own small push in the weights, and the pushes add up. Spread a fixed budget over 1,392 separate facts and each gets very little, so each answer is given with low confidence.

The rule needs only a few waves. In my decay-1 network, about 8 waves carried the computation. Each wave is shared by every sum, so a single increase in its size raises the confidence of all 3,481 answers at once, including the hidden ones. Spread the same budget over 8 shared waves and each gets a great deal. Under weight decay, every unit of norm must justify itself against the pull towards zero, and shared waves justify themselves far more easily than private facts.

Figure 17. Weight decay is a tax; shared structure pays it more easily. The efficiency account follows Varma et al. (2023). The count of 8 waves comes from the ablation in Part 4. mechanism diagram

This also explains the failure boundary. With 20 per cent of the sums, the table has only 696 facts to store, so each gets a larger share of the budget, and the table's confidence is much closer to the rule's. The advantage of the rule shrinks, and within the time I allowed, it never won. With 70 per cent, the table has 2,436 facts and is very inefficient, and the rule won within 1,000 steps. Varma and colleagues predicted exactly this dependence on dataset size and confirmed it, including the strange case of "ungrokking" when a grokked network is retrained on a smaller set.

A worked case of the budget

Here is the arithmetic in its simplest form. Suppose a network has a budget of 10 units of weight to spend. Confidence in an answer grows with the weight behind it. The table must spend separately for each of 10 training facts, so each fact gets 1 unit. The rule spends on 2 shared waves, so each gets 5 units, and every fact draws on both.

Table: each training answer is backed by 1 unit. Hidden answers are backed by nothing.
Rule: each training answer is backed by 5 + 5 = 10 units. Hidden answers are backed by the same 10.
Now let decay remove 10 per cent of every weight each round. The table loses 0.1 per fact, which the data must restore 10 separate times. The rule loses 0.5 per wave, which the data restore through every fact at once.

The numbers are invented and the real network is not this tidy. But the shape is the point. Shared structure makes each unit of weight do more work, and decay is a tax on units. Over enough rounds, the tax falls hardest on the solution that uses its units least well.

Why building the rule is slow

None of this explains the long delay, and Part 4 showed the delay is mostly construction. The best available answer is that the rule is hard to find by small steps. At the start, the embeddings are random. A single wave in the embeddings does little good on its own, because the hidden layer has to learn to combine waves for the same hours before any sum comes out right. The gradient towards the rule is weak while the rule is half-built, and the table, which helps right away, takes the easy credit first.

So training follows the path of least resistance. It builds the table in a hundred steps because the table pays off at once. It builds the rule over thousands, because each small piece pays off only a little until the pieces fit together. Weight decay keeps taxing the table throughout, so that once the rule is good enough to compete, the table loses. Nanda and colleagues saw this in their transformer as gradual circuit formation, then cleanup. The ablations in Part 4 show the same split in my network: by step 12,000 the table was gone and 8 waves carried everything.

Figure 18. Memorise in a hundred steps, build for thousands, then clean up. Phase names follow Nanda et al. (2023). Boundaries are my reading of the decay-1 run: hidden accuracy passes 50 per cent at step 5,500, and the top-8 ablation at step 12,000 shows the table gone. measured, with author reading

An analogy, and where it breaks

Mrs. Rao's fading slates are the analogy I have used throughout, and it holds in the main. Fading makes every stored mark cost effort to keep, and a pupil who must rewrite a long table every morning has a strong reason to find something shorter. A pupil with a permanent notebook has none.

The analogy breaks in two places. First, a pupil notices patterns by looking for them; Asha chose to stop rewriting. A network does not choose. The gradient simply moves each weight a little in whichever direction lowers the score, and the rule emerges only because it lowers the score more per unit of weight. Second, the slate fades the table and nothing else. Weight decay fades the rule too, which is why too much decay, in the decay-3 run, left a weaker rule with a lower wave measure. A real slate does not erase the method along with the facts.

Figure 19. The slate explains the pressure, not the search. The analogy captures why the table loses. It breaks where it gives the network a choice, and where it spares the rule from the same fading. analogy check

A second analogy: the grammarian's half-vowel

Sanskrit grammar offers a sharper picture of the pressure towards compact rules. A student can learn Sanskrit nouns by memorising tables of forms, eight cases in three numbers for every type of stem. Or the student can learn Pāṇini's grammar, about four thousand short rules from which every form can be generated. The tables are quicker to start with. The rules take years, but once learned they cover words the student has never met.

The later grammarians were famous for valuing brevity above almost everything. A traditional saying, recorded in Nāgeśa's Paribhāṣenduśekhara, holds that grammarians rejoice at saving half a short vowel as much as at the birth of a son. That is weight decay as a cultural value: every unit of length in a rule must justify itself. The analogy breaks in a telling place. Pāṇini searched for brevity on purpose, knowing what he wanted. The network has no aim of brevity. It only feels the tax, and brevity is what survives it.

Thought experiment 4

The rival who says nothing sudden happened

A sceptic takes my curves and says the word "grokking" should be retired. Nothing sudden happens, on this view. The network improves smoothly in its weights the whole time; the jump appears only because accuracy is a harsh measure that stays near zero until the rule is nearly complete. To speak of a moment of understanding, let alone a recognition, is to read a story into a smooth curve.

Pause. Before you read on: is there anything sudden in my data, or is the sceptic right that it is all a smooth climb measured with a cliff-edged ruler?

The sceptic is mostly right, and the evidence is in Part 3. The wave measure rose steadily from the start of the decay-1 run. Hidden accuracy climbed over about 5,000 steps, not in an instant. Nanda and colleagues made the same point with their progress measures, which change smoothly long before the accuracy jumps. This is the same argument that Paper 20 of this series takes up for emergent abilities in large models: a sharp change in a score need not mean a sharp change underneath.

There is one thing the smooth story leaves out, though. The network's behaviour on hidden sums changes from systematically wrong to systematically right, and the two regimes rest on opposite internal structures, as the ablation showed. The path between them is smooth; the endpoints are not two points on a scale. The lesson is that "sudden" belongs to the score, but "different" belongs to the mechanism. The limit is that I measured one simple network, where the endpoints are unusually clean.

Part 6

Sudden and gradual awakening

The argument about whether grokking is sudden has an ancient parallel, and it is worth taking slowly. In the Chan Buddhist tradition of China, a famous story tells of two monks at the monastery of the Fifth Patriarch, Hongren, in the seventh century. The account, preserved in the Platform Sūtra, is legendary in its details, but its argument is clear and has shaped East Asian thought for more than a thousand years.

Hongren asked his students to write a verse showing their understanding. The senior monk, Shenxiu, wrote that the body is the bodhi tree and the mind is like the stand of a bright mirror. His verse ends with the instruction to polish the mirror constantly and let no dust collect on it. Huineng, an illiterate kitchen worker, had a verse written beside it that said bodhi has no tree and the mirror no stand. In the line most often quoted, there is "originally not a single thing", so where could dust collect? Hongren, the story says, recognised Huineng's understanding and passed the robe to him in secret.

The two verses came to stand for two schools. The Northern school, linked with Shenxiu, was later described as teaching gradual awakening, through steady practice that wipes the mirror clean. The Southern school, linked with Huineng, taught sudden awakening, a direct seeing of what was always there. Historians now think the contrast was sharpened by Huineng's later followers. But the question it poses is real. Is understanding the end of a slow process of cleaning, or an event that the process merely makes room for?

Figure 20. Gradual wiping, sudden seeing, and a network that does both without a mirror. Paraphrased from the Platform Sūtra, attributed to the seventh century and later edited by Huineng's followers. The bottom panel is this paper's reading. conceptual diagram

Thought experiment 5

The mirror that is wiped every step

Weight decay looks remarkably like Shenxiu's practice. At every step, without exception, it wipes a little from every weight, and whatever the data do not keep renewing fades away. Over thousands of steps, the dust of the memorised table is cleared and what remains is the rule. Then, to an observer watching only the score, understanding appears all at once, as Huineng said it would.

Pause. Before you read on: in the network, which verse is closer to the truth, the gradual wiping or the sudden seeing?

In the network, both, and that is what makes the parallel instructive. The process is wholly gradual, as Shenxiu taught. The wave measure rises step by step, and nothing happens at any one step that did not happen a little at the step before. But the result is a different kind of thing from what came before, as Huineng insisted. A table is not a partial rule; it is something else, and the ablations show it. A gradual process arrived at a discontinuous result.

The lesson is that the old dispute may have been less of a contradiction than its partisans believed. A process can be gradual in its steps and sudden in what it reveals. The limit is severe, however. Huineng's point was that the mind's nature is "originally not a single thing", never covered and never needing to be uncovered. Nothing in a network is originally anything. The rule was not there waiting under the dust; it was built, piece by piece, by the very process that cleared the table. A network has dust and a rule, but no original face.

[AUTHOR NOTE: optionally add a sentence on whether the sudden and gradual framing matches anything in your own practice]

Removal as a method

Indian traditions have their own version of understanding by removal. The Upaniṣadic phrase neti neti, "not this, not this", describes the self by stripping away everything it is not. In Advaita Vedānta, the method is to remove what has been wrongly superimposed, which Śaṅkara called adhyāsa, the classic image being the rope mistaken for a snake. When the error is removed, nothing new is added; the rope was there all along.

Weight decay is a method of removal too, and the parallel holds for half its length. What decay removes is what the data do not renew, which is close to removing what is not needed. But the parallel breaks at the same place Huineng's verse did. In Vedānta, what remains after removal was always present and is the most basic thing there is. In the network, what remains after decay was constructed during training. Remove the table from a freshly initialised network and you get random noise, not a rule. Removal clears the ground; it does not supply what grows there.

The means that take time, and the means that do not

The Trika tradition of Kashmir offers a finer version of the sudden and gradual debate. Abhinavagupta, in his Tantrāloka, set out a ladder of means, or upāya, for realising one's nature. The individual means, āṇavopāya, works through effort, breath, ritual and concentration, and it is gradual. The means of power, śāktopāya, works through the refinement of thought. The divine means, śāmbhavopāya, is a direct, almost thoughtless entry. Beyond them all lies anupāya, "no means", for those in whom recognition is simply present.

What is striking is that the tradition does not set these against each other. They are suited to different capacities, and the gradual means are understood to ripen into the direct ones. A practitioner's long effort is not wasted when sudden recognition comes; it is what made the ground ready. The network behaves something like the lower rungs of this ladder. Thousands of steps of effort, then a change that looks immediate. The break, again, is at the top. Anupāya presupposes a nature that was always there. A network has only what its effort built.

Where grokking sits in the RTM layers

In the Recognition Theory of Mind (RTM), the hypothesis that frames this series, grokking touches the lower layers clearly and the upper ones not at all. L1, spanda or responsiveness, is present throughout: the weights change with every step. L2, prakāśa or manifestation, changes in an interesting way. Before grokking, the network's internal structure makes only training sums available. After, it makes the structure of the clock available, and every sum follows from it. That is a real change in what the system can present.

L3, vimarśa or reflexive awareness, is absent. The network does not notice that it has found the rule; nothing in it represents its own state of understanding. L4, pratyabhijñā or recognition, has at most a shape. Every sum is now handled as an instance of one pattern, which is formally like recognising each sum as the same clock. But no one recognises anything. The word "grok" itself, in Heinlein's sense, implies an understanding so complete that the knower merges with the known. My network merged with nothing, because there was no knower to merge.

Figure 21. Structure without anyone grasping it. Recognition Theory of Mind layers applied to grokking. The dotted L4 rung marks a structural parallel, not a capacity. author hypothesis

I would put it this way. Grokking shows that a system can move from storing particulars to holding a structure, by gradual pressure, with no awareness at any stage. That is a useful result for anyone who thinks understanding must involve awareness. It suggests that one thing we call understanding, the grasp of structure behind particulars, can occur without the other thing we often mean by the word.

Part 7

What grokking means for building real systems

Few production models are trained on clock sums, and few engineers will ever see a grokking curve in their own work. But the lesson about measurement carries over directly. It is also worth saying what does not carry over. Large models trained once on vast and varied data rarely sit for thousands of steps with a perfect training score, because their training data are never fitted perfectly. The late, clean jump of a toy task is unusual. A model can look finished on the data you have while its internal structure is still changing, and the change may matter a great deal for data you do not have.

Early stopping can stop too early

Early stopping ends training when held-out performance stops improving. In my no-decay network, held-out accuracy was 0 for 20,000 steps, so early stopping would have ended training at once, correctly. In my decay-1 network, held-out accuracy was 0 for the first 2,000 steps and under 4 per cent until step 3,000. An early-stopping rule with a patience of a few hundred steps would have stopped training there, and called the network a failure, about 5,000 steps before it became perfect.

For large models trained on varied data, the effect is usually much weaker, because their held-out performance rarely stays flat for long. But Humayun and colleagues found delayed improvements in robustness in image classifiers long after accuracy had settled, which suggests the habit of stopping at the first plateau can leave something on the table.

Memorisation that does not show

The second lesson is about what a model stores. My no-decay network was perfect on its training data and held no rule at all. In a production model, the analogue is a model that has memorised parts of its training data, such as personal records or confidential text, while also learning general structure. Accuracy on the training set cannot tell you which you have. The wave ablation could, for my network, because the rule had a known shape. For real models the tools are cruder: extraction tests, membership-inference tests, and checks of how outputs change when training examples are removed.

What to watch in a real training run

Most training dashboards show two curves, training loss and validation loss. The grokking results suggest adding at least two more. The first is the weight norm, which is cheap to compute and often revealing. A norm that keeps climbing while training loss is already near zero is a sign that the network is inflating a memorised solution. A norm that falls while validation loss is flat may be a sign that something is being rebuilt underneath.

The second is some measure of internal structure suited to the task. For clock sums, it was the wave share. For other tasks, candidates include the effective rank of a layer's weights, the sparsity of its activations, or the local complexity measure that Humayun and colleagues proposed. None of these is universal. But a measure that moves while the scores sit still is exactly what an engineer needs to decide whether to keep training.

A third habit is cheap and often skipped: rerun the training with two or three regularisation strengths. In my runs, decay of 0.3 did nothing useful in 6,000 steps, decay of 1 worked best, and decay of 3 generalised sooner but less cleanly. One setting tells you very little about the others. The choice is often inherited from a template and never revisited.

Thought experiment 6

The model we shipped on the third day

Suppose a team trains a classifier on a small, carefully labelled dataset of loan applications. It reaches 100 per cent on its training set within a day and scores reasonably on a small validation set. The team ships it. Months later, an audit finds that it has memorised individual applicants and applies their outcomes to new applicants who resemble them in irrelevant ways.

Pause. Before you read on: was the team's mistake in shipping too early, or in something else?

The uncomfortable answer is that the team may have done nothing that standard practice forbids. They stopped when the scores stopped improving. What they lacked was any measure of whether the model held a rule or a table, and with a small dataset, a table is cheap. My fraction sweep showed that with 20 or 30 per cent of the sums, the table never lost within 6,000 steps. Small datasets are exactly where memorisation is cheapest and hardest to see.

The lesson is that a perfect training score is evidence about the data, not about the model, and that regularisation choices made casually can decide whether a model memorises. The limit is that the analogy to loan data is loose. Real data are noisy and the "rule" may not exist in a clean form. What carries over is the warning that the training score cannot tell a rule from a table.

[AUTHOR NOTE: optionally add a sentence on whether you have seen a model in client work that scored perfectly in training and turned out to be memorising]

Four questions for any delayed-generalisation claim

When someone reports grokking, or a late improvement of any kind, four questions sort the explanations. First, was there weight decay or another limit on the norm, and what happened without it? Second, did a measure of internal structure change before the accuracy jumped? Third, did the effect depend on the size of the training set in the way the efficiency account predicts? Fourth, was the optimiser adaptive, and were there signs of the late instabilities Thilak and colleagues described? A claim that answers all four is far more useful than one that reports the curve alone.

Figure 22. Four questions before you explain a late jump. A claim that answers all four separates the main accounts; a curve alone does not. decision instrument

Objections

The best case that I have this wrong

The strongest objection is that my network is not the network in which grokking was found. Power and colleagues used transformers; Nanda and colleagues used a one-layer transformer with attention. Mine is a two-layer network on joined embeddings, with no attention at all. It may grok for different reasons. I accept the gap. In reply, the wave measure and the ablation show that my network found the same kind of solution, a few Fourier waves, as theirs did. That makes it likely, though not certain, that the same pressures were at work.

A second objection concerns my "construction time" claim. The fixed-norm network was held at 55 from the start, but perhaps 55 is still too large for the rule to win quickly. A smaller fixed norm might remove the delay, as Liu and colleagues reported for some settings in Omnigrok. If so, my conclusion that the delay is mostly construction would weaken. I ran only two fixed norms, and this is the experiment I would most like to extend.

A third objection is that weight decay in AdamW is not the same as an L2 penalty added to the loss. With adaptive optimisers the two behave differently. The topic question asks about L2 weight decay, and I tested decoupled weight decay. In practice the two have the same aim, small weights, and both were used in the grokking literature. But a careful test with a true L2 penalty might give different timings.

A fourth objection is that one seed per setting tells us little. The timings in Part 3 could shift by thousands of steps with a different random start. That is true, and it is why I ran two further seeds of the main setting, reported in Part 4. The qualitative results, no generalisation without a norm limit and steady generalisation with one, are the kind that tend to survive reseeding. The exact step numbers are not.

A fifth objection is philosophical. Comparing weight decay to Shenxiu's mirror-wiping trivialises a spiritual practice by turning it into an optimisation trick. I take this seriously. My use of the verses is to clarify a structural question about gradual and sudden change, not to claim that a network does anything like practice. The break I drew in Thought Experiment 5, that a network has no original face, is meant to keep the two apart.

A sixth objection is about scale. A clock of 59 hours and a network of 88,000 weights are tiny, and the clean separation between table and rule may be a luxury of tiny problems. In a large model trained on text, there are millions of overlapping patterns, each partly memorised and partly generalised. I agree that the clean story will not transfer intact. What may transfer is the logic: a limit on weight size favours shared structure, and shared structure takes time to build.

Tests

What would change my mind

Several results would make me revise this paper. If a network held at a fixed small norm generalised almost at once, I would drop the claim that the delay is mostly construction, and accept that removing the table is the slow part after all. If networks trained with no decay and no norm limit routinely grokked clock sums in settings like mine, given more steps, I would weaken the claim that a norm limit is needed here.

On the mechanism, I would revise the efficiency account if a grokked network were found whose rule had a higher norm than the table it replaced, at matched training accuracy. On the boundary, I would revise C7 if the 20 and 30 per cent runs generalised when trained much longer, since that would make the boundary a matter of time, not of a break-even point.

On the practical side, I would revise my advice about early stopping if studies of large models found that late improvements on held-out data, of the kind seen here, essentially never occur in realistic training. Prediction P19-2 is a bet that they do, at least sometimes. If 2028 arrives with no such report, the practical lesson of this paper shrinks to small datasets and toy tasks.

Figure 23. What I do not know, placed by how soon it could be settled. Placement is my judgement. author judgement

I would also revise C4 if the fixed-norm result failed to repeat. My claim that a table can live at low norm rests on one run held at 55. If other seeds, or slightly smaller norms, failed to memorise at all, the table would after all need a large norm, and the simple pruning story would regain some of the ground this paper takes from it. That run is cheap, and it is the second one I would do.

On the RTM side, I would take the recognition framing more seriously if a grokked network were found to contain a component that signals, in some measurable way, that it has switched from table to rule, and uses that signal. That would be a functional analogue of noticing one's own understanding, a sign of something like L3. I know of no such finding.

Figure 24. How sure I am of each claim, at a glance. Bar length is my subjective confidence; pattern and colour give the ledger category, so the chart reads in greyscale. author judgement

Ledger

Claim Ledger

Each main claim of this paper, with how sure I am, what kind of evidence carries it, and what would knock it down.

ID	Claim	Confidence	Evidence type	What would refute it
C1	In my setting, a network with no limit on its weight norm memorises clock sums and does not generalise within 20,000 steps.	established	This paper: hidden accuracy 0.0 at step 20,000, norm 257; norm held at 88 gives 0.002 at step 8,000	The same network generalising without decay or a norm limit given more steps
C2	With weight decay, generalisation arrives as the norm falls and the embeddings concentrate into a few Fourier waves.	established	Nanda et al. (2023); this paper: 95% at step 8,200, norm 53, top-5 share 0.52 at step 20,000	Grokked networks whose embeddings stay spread across all frequencies
C3	The grokked network runs on a few waves; the memorising one has no small sufficient part.	established	This paper's ablation: top 8 waves keep 0.98 hidden accuracy; no-decay network falls to 0.11 on training sums	A grokked network that fails when reduced to its strongest waves
C4	A memorising table does not need a high norm; the inflation without decay comes from the push for confidence.	likely	This paper: norm held at 55 still memorises within 1,000 steps	Memorisation failing at a fixed small norm across settings
C5	Reducing the norm is not sufficient; most of the delay is the time needed to build the rule.	likely	This paper: decay-3 norm 54 at step 500 with zero hidden accuracy; fixed norm 55 reaches only 0.42 by step 8,000	A smaller fixed norm removing the delay almost entirely
C6	Weight decay is not necessary for grokking in general.	established	Thilak et al. (2022); Kumar et al. (2023)	Re-analysis showing those cases depend on hidden regularisation
C7	In my network, the break-even training fraction lies between 30 and 40 per cent of all sums.	likely	This paper's fraction sweep, one seed, 6,000 steps	The 20 and 30 per cent runs generalising with longer training
C8	The rule wins because it gives more confident answers per unit of weight than the table.	likely	Varma et al. (2023); consistent with this paper's fraction sweep	A grokked rule with higher norm than the table it replaced, at matched accuracy
C9	Grokking shows a move from particulars to structure with no awareness at any stage: L2 changes, L4 has a shape, L3 is absent.	speculative	Conceptual analysis under RTM	A component that marks the switch from table to rule and is used by the network

Dated predictions

ID	Prediction	Judged by	How it will be judged
P19-1	Holding this paper's network at a fixed norm of 35 without weight decay will still leave hidden accuracy below 50 per cent at step 2,000, though training accuracy reaches 100 per cent by then.	This paper's scripts, rerun by me or anyone	Run e19c with fixnorm 35 and report hidden accuracy at step 2,000
P19-2	By the end of 2028, a published study will report delayed generalisation of at least 20 percentage points on a held-out set, at least 1,000 steps after training accuracy passed 99 per cent, in a language model of over 100 million parameters trained on natural text.	Peer-reviewed or widely cited work	Literature search on 31 December 2028

Words

Glossary

Grokking: Generalisation that arrives long after a network has fitted its training data perfectly.
Weight decay: A small pull on every weight towards zero at every training step.
L2 penalty: A cost added to the training score equal to the sum of squared weights; close to weight decay, but not identical with adaptive optimisers.
Weight norm: The square root of the sum of all squared weights; one number for the overall size of the weights.
Modular arithmetic: Arithmetic on a clock: numbers wrap round after reaching a fixed value.
Embedding: The list of numbers a network learns for each input symbol.
Fourier wave: A regular up-and-down pattern of a given speed; any pattern over the 59 hours can be split into such waves.
Ablation: Removing part of a network to see what it was doing.
Cross-entropy: The usual training score for classification; it keeps rewarding more confidence in correct answers.
Early stopping: Ending training when held-out performance stops improving.
ardhamātrālāghava अर्धमात्रालाघव: Sanskrit: economy of half a mora; the grammarians' ideal of brevity in rules.
bodhi बोधि: Sanskrit: awakening; also the name of the tree under which the Buddha awoke.
neti neti नेति नेति: Sanskrit: "not this, not this"; the Upaniṣadic method of describing the self by negation.
adhyāsa अध्यास: Sanskrit: superimposition; in Śaṅkara, the error of taking one thing for another, as a rope for a snake.
dùnwù 頓悟 and jiànwù 漸悟: Chinese: sudden awakening and gradual awakening, the two positions associated with Huineng and Shenxiu.
upāya उपाय: Sanskrit: means; in Trika, the graded methods for realising one's nature.
āṇavopāya आणवोपाय: Sanskrit: the individual, gradual means, through effort and concentration.
śāktopāya शाक्तोपाय: Sanskrit: the means of power, through the refinement of thought.
śāmbhavopāya शाम्भवोपाय: Sanskrit: the divine, direct means.
anupāya अनुपाय: Sanskrit: "no means"; recognition without method.
spanda स्पन्द: Sanskrit: vibration, responsiveness; RTM layer L1.
prakāśa प्रकाश: Sanskrit: light, manifestation; RTM layer L2.
vimarśa विमर्श: Sanskrit: reflective awareness; RTM layer L3.
pratyabhijñā प्रत्यभिज्ञा: Sanskrit: recognition; knowing a present thing as one known before. RTM layer L4.

Conclusion

Wiping the slate, and building the clock

This paper asked whether delayed generalisation in modular arithmetic arises from L2 weight decay pruning high-norm memorising lookup tables. In my networks the answer is partly yes. Without a limit on the norm, the network memorised and never generalised, and its norm inflated without end. With weight decay, the norm fell, the embeddings came to be made of a few waves, and the network became perfect on sums it had never seen. The memorised table was gone by the end, and 8 waves carried the whole computation.

But the pruning picture leaves out the slow part. A network held at a small norm from the start still took thousands of steps to generalise, and a network whose norm fell fast generalised only later. Weight decay sets the terms of the competition between table and rule. It makes shared structure pay and private facts cost. It does not build the rule, and building is what takes time.

Figure 25. The paper on one page. Pin it next to the next claim that a model "suddenly understood". summary

For anyone training models, the practical message is that a perfect training score says nothing about which of two very different solutions a model has found. On small datasets, where memorisation is cheap, that question deserves its own measurement. Regularisation is not a detail to be tuned last. It can decide whether a model learns the clock or the answer sheet.

The experiments also left me with a sharper sense of what "sudden" means. In every run, the inside of the network changed smoothly, step by step. What changed suddenly was the score, because a test that asks for exactly the right answer gives no credit to a rule that is nearly right. The next paper takes up that same gap between smooth change and sharp scores at the scale of large language models.

Shenxiu wiped the mirror and Huineng said there was no mirror. In my networks, the wiping was real and steady, and what appeared when it was done was a different kind of thing from what had been there before. Neither monk quite describes it. The mirror did not hold a face waiting to be revealed. The face was assembled while the dust was being cleared, by the same slow hand.

The next paper in this series turns from training time to scale. It asks whether the abilities said to "emerge" suddenly in large language models are real jumps or artefacts of how they are scored.

[AUTHOR NOTE: optionally close with a sentence on what grokking suggests to you about learning by repetition]

Evidence

Sources

Primary research

Power, A., Burda, Y., Edwards, H., Babuschkin, I. and Misra, V. (2022). Grokking: generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177. Verified; abstract quoted.
Liu, Z., Kitouni, O., Nolte, N., Michaud, E. J., Tegmark, M. and Williams, M. (2022). Towards understanding grokking: an effective theory of representation learning. arXiv:2205.10343. Verified; abstract quoted.
Liu, Z., Michaud, E. J. and Tegmark, M. (2022). Omnigrok: grokking beyond algorithmic data. arXiv:2210.01117. Verified; abstract quoted.
Thilak, V., Littwin, E., Zhai, S., Saremi, O., Paiss, R. and Susskind, J. (2022). The slingshot mechanism: an empirical study of adaptive optimizers and the grokking phenomenon. arXiv:2206.04817. Verified; abstract quoted.
Nanda, N., Chan, L., Lieberum, T., Smith, J. and Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. arXiv:2301.05217. Verified; abstract quoted.
Varma, V., Shah, R., Kenton, Z., Kramár, J. and Kumar, R. (2023). Explaining grokking through circuit efficiency. arXiv:2309.02390. Verified; abstract quoted.
Kumar, T., Bordelon, B., Gershman, S. J. and Pehlevan, C. (2023). Grokking as the transition from lazy to rich training dynamics. arXiv:2310.06110. Verified; abstract quoted.
Zhong, Z., Liu, Z., Tegmark, M. and Andreas, J. (2023). The clock and the pizza: two stories in mechanistic explanation of neural networks. arXiv:2306.17844. Verified.
Humayun, A. I., Balestriero, R. and Baraniuk, R. (2024). Deep networks always grok and here is why. arXiv:2402.15555. Verified; abstract quoted.

Journalism

Ananthaswamy, A. (2024). How do machines 'grok' data? Quanta Magazine, 12 April 2024. quantamagazine.org. Source of the vacation story and Power's quotation.

Primary texts and background

Heinlein, R. A. (1961). Stranger in a Strange Land. G. P. Putnam's Sons. Origin of the word "grok".
The Platform Sūtra of the Sixth Patriarch, attributed to Huineng; the verses of Shenxiu and Huineng. Standard translations, for example P. B. Yampolsky (1967), Columbia University Press. Paraphrased; one short phrase quoted in its common rendering.
Nāgeśa Bhaṭṭa, Paribhāṣenduśekhara, the closing maxim on the grammarians' delight in saving half a short vowel (ardhamātrālāghava). Standard editions; paraphrased.
Pāṇini, Aṣṭādhyāyī, with about 4,000 sūtras. Standard editions.
Abhinavagupta, Tantrāloka, chapters 1 to 5, on the upāyas (anupāya, śāmbhava, śākta, āṇava). Standard editions; paraphrased.
Bṛhadāraṇyaka Upaniṣad 2.3.6 and elsewhere, for neti neti; Śaṅkara's introduction to the Brahmasūtrabhāṣya, on adhyāsa. Standard editions; paraphrased.

Reference

Cite this work

Mahapatra, R. R. (2026). The fading slate: does weight decay make a network grok by pruning its lookup table?. Mind Frontier paper U0535. rajeshrmahapatra.com. https://rajeshrmahapatra.com/papers/mind-frontier/U0535-grokking-weight-decay-lookup-tables.html. DOI: to be assigned.

@misc{mahapatra2026grokking,
  author = {Mahapatra, Rajesh Ranjan},
  title = {The fading slate: does weight decay make a network grok by pruning its lookup table?},
  year = {2026},
  howpublished = {Mind Frontier paper U0535, rajeshrmahapatra.com},
  url = {https://rajeshrmahapatra.com/papers/mind-frontier/U0535-grokking-weight-decay-lookup-tables.html},
  doi = {10.xxxx/placeholder}
}