The scene
The stenographer who began finishing sentences
Nandini joined the district court in Mysuru as a junior stenographer in June. For her first three weeks she typed exactly what she heard, a word behind the speaker, and never ahead of him. Then, one Thursday in the fourth week, during a long property dispute, she noticed that her fingers were already typing "the petitioner's late father" before the advocate had finished saying "the petitioner's". He had used the phrase eleven times that morning. Nothing about her typing speed had changed. What had changed was that she had started to use the transcript itself as a guide to what came next.
Her mentor, Mr. Ramaswamy, explained the habit when she asked about it. Good stenographers, he said, keep two things in mind at once. As each word arrives, they hold on to the word that came just before it. And when a word arrives that they have heard before, they look back to the earlier place where it appeared and read off what followed it then. Neither habit is useful alone. Remembering the previous word tells you nothing about the future. Looking back for the current word finds the right place, but only if every earlier word carries a note of what preceded it, so that "the place after the last 'petitioner's'" can be found at all.
Nandini and Mr. Ramaswamy are invented, and so is the court case, though the habit is real enough. I will come back to them in every part of this paper, because their two-step trick is, almost exactly, the circuit that a small transformer learns when it learns to copy from its own context. Researchers call that circuit an induction head. It completes a pattern of the form [A][B] … [A] → [B]: having seen A followed by B once, the model predicts B the next time A appears.
The question this paper asks is precise. What is the minimal two-layer circuit configuration that is necessary and sufficient for copy-and-paste induction behaviour to appear during training? It matters because induction heads are the best-understood piece of in-context learning, the ability of language models to use what is in their prompt. It also matters because the standard account is often repeated as if it were the only way the job could be done.
My answer comes from the published theory and from a set of knock-out experiments I ran on tiny attention-only transformers. Two attention steps are needed, in series, and the second must be able to use what the first wrote. Some positional signal is also needed. One head in each layer is enough, and no MLP is required. The textbook route, in which the second step's keys read a previous-token note, is sufficient but, in my runs, not strictly necessary. When I blocked it, four of five training runs found another route through the second step's queries. The necessary thing is composition between two attention steps; the particular wiring is not unique.
Part 1
What copy-and-paste induction asks of a model
An induction head is defined by what it does to a sequence, so it helps to start with the sequence. Take a run of tokens in which some stretch appears twice, such as "the petitioner's late father … the petitioner's late father". At each position in the second copy, the right prediction is the token that came after the same token in the first copy. A model that has learned to do this can continue any pattern it has seen once in its context, even one made of nonsense words it never saw in training.
That last point is what makes the behaviour interesting. A model cannot get the right answer from stored knowledge, because the repeated stretch can be random. It has to use the context. In the synthetic task I used throughout this paper, each training sequence is 48 random tokens drawn from a vocabulary of 64. Somewhere in it, a stretch of between 8 and 20 tokens is copied to a later place, with a random gap in between. On the first copy, no model can do better than guessing, about one chance in 64. On the second copy, a perfect induction circuit can predict every token after the first one.
Why one look is not enough
Consider what a single attention step can do. At each position it forms a query, compares it with a key at every earlier position, and takes a weighted average of the values there. The key at each position is computed from what is at that position, its token and its place in the sequence. So one attention step can easily find "earlier positions holding the same token as me". What it needs to find, though, is "the earlier position just after a copy of my token". That target is defined by its neighbour, and a key built only from the token at that position does not know its neighbour.
The fix is a second step. A first attention head, at every position, looks one place back and writes the previous token into the residual stream. Now the key at the position after each earlier "A" can say, in effect, "the token before me was A". A second head, at the current "A", sends a query for "the token before you was A", finds that position, and copies its token, B, into the prediction. In the vocabulary of Elhage and colleagues (2021), who first described the circuit in two-layer attention-only models, the second head's keys are composed with the first head's output. They call this K-composition.
This is Nandini's two habits. The previous-token head is her note of what preceded each word. The induction head is her glance back for the earlier place where the current word was that note. Mr. Ramaswamy was right that neither habit helps alone, and the reason is the one above: the place she needs to find is defined by what came just before it.
A minimal worked case
Here is the smallest version that shows every step. Take the six-token sequence "red cat sat red cat ?". We want the model at position 5, the second "cat", to predict "sat".
- Layer 0, previous-token head. At each position it attends to the one before and writes that token. So position 2 ("cat") now carries a note "after red", position 3 ("sat") carries "after cat", position 4 ("red") carries "after sat", and position 5 ("cat") carries "after red".
- Layer 1, induction head. At position 5 the current token is "cat", so the query asks for "a position whose note says after cat". Among positions 1 to 5, only position 3 has that note. The head puts nearly all its weight there.
- The value at position 3 is its own token, "sat". The head copies it into the stream at position 5, and the output layer predicts "sat".
Every quantity here is something the trained models in this paper actually compute, at a larger scale. The heat map below shows the second-layer attention pattern from one of my trained models on a repeated sequence. The bright stripe sits exactly one position after each earlier copy of the current token, the signature of an induction head.
That stripe is the standard diagnostic. Olsson and colleagues (2022) defined an induction head by two behaviours on repeated random sequences: attending to the token after an earlier copy of the current token, and raising the probability of that token. I measure the first of these as an induction score, the average attention weight a head puts on the correct earlier position. A score of 0.7 means 70 per cent of the head's attention lands exactly where an induction head should look.
Part 2
How the circuit was found and explained
The induction head entered the literature through a style of research that treats a trained network as an object to be taken apart. In December 2021, Elhage and colleagues at Anthropic published a mathematical framework for transformer circuits. They studied tiny models with no MLPs at all, only attention, because such models can be written out as sums of interpretable paths. In their two-layer models they found pairs of heads doing the two-step trick, and they named the second head an induction head.
The more surprising finding came nine months later. Olsson and colleagues trained many models, from small attention-only ones to larger ones with MLPs, and watched what happened during training. In the models they studied with more than one layer, there was a short window early in training in which the loss curve showed a small bump. During that window, induction heads formed, and the models' ability to use their context improved sharply. In the authors' words, induction heads "develop at precisely the same point as a sudden sharp increase in in-context learning ability, visible as a bump in the training loss". One-layer models showed neither the bump nor the heads.
That true story has a quiet lesson about how science gets done. A bump that small is easy to dismiss as noise, and anyone who did not know what to look for could have smoothed it away. It became a finding only when someone asked what the network was doing at that moment, and then looked inside. Olsson and colleagues presented "six complementary lines of evidence" that induction heads might be the main source of in-context learning, with "strong, causal evidence" in small attention-only models and correlational evidence in larger ones.
Why the circuit appears suddenly
Several later papers asked why the circuit appears all at once. Reddy (2023) built a two-parameter model of an induction head and traced its abrupt emergence to "the sequential learning of three nested logits". Each step in the chain is useless until the previous one is partly in place, so the gradient for the whole chain is tiny at first and then grows quickly. Singh and colleagues (2024) went further with an "optogenetics-inspired" method that clamps parts of the network during training. They found that induction heads are diverse and additive, and identified "three underlying subcircuits that interact to drive IH formation, yielding the phase change".
Two other results matter for the minimal question. Bietti and colleagues (2023) studied a simplified two-layer transformer and showed that global, fixed bigram statistics are learned fast while the induction mechanism for in-context bigrams develops more slowly, with weight matrices acting as associative memories. Edelman and colleagues (2024) trained transformers on sequences drawn from random Markov chains and saw three phases. First came uniform predictions, then a sub-optimal strategy based on single-token counts in the context, and then "a rapid phase transition to the correct in-context bigram solution". They found evidence that the simpler unigram solution may delay the better one.
The picture that emerges is consistent. The induction circuit has parts that only pay off together. Training finds the parts slowly and in a particular order, often after spending time on an easier strategy, and then the whole circuit locks in quickly. That is Nandini's three quiet weeks and her sudden Thursday.
Part 3
Knock-out training: what the circuit cannot do without
"Necessary and sufficient" is a strong phrase, and published work mostly establishes the second half of it. Elhage and colleagues showed that a two-layer attention-only model can learn induction, and Sanford and colleagues proved that one layer of modest size cannot. Between those two results there is still a gap. Which parts of the two-layer model are doing the essential work, and which are just present because the architecture provides them?
The instrument
To answer that, I built what I call knock-out training. Instead of training one model and removing parts afterwards, I trained a separate model for each variant, with one route removed from the start, and asked whether induction could still form at all. Removing a part after training tells you what a particular trained model relies on. Removing it before training tells you whether the behaviour needs it, which is the question this paper asks.
Every model was an attention-only transformer with a 64-dimensional stream, trained for 4,000 steps on the random-gap copying task from Part 1. The measure was accuracy on the predictable tokens of the second copy, where chance is about 1.6 per cent. I removed routes in the second layer by changing what it is allowed to read. The second head computes its query, key and value from the stream. For a blocked route, that part was computed from the original embedding instead, so it could not see anything the first layer wrote. Blocking the key route, for example, means the second head's keys know each position's token and place, and nothing more.
There were eight variants. Three changed the size: the full model with two heads per layer, one head per layer, and three layers. Four blocked routes: the value route, the query route, the key route, and the query and key routes together. The last removed positional embeddings. The full model and the keys-blocked model were each run with several random seeds, because the first results were surprising enough to need checking.
What came back
The full model learned induction in all three seeds, ending between 0.92 and 0.99. One head per layer was enough, reaching 0.93. Three layers did slightly better, 0.99, and formed the circuit a little earlier. Blocking the value route made no difference at all, 0.92, because the token to be copied can come straight from the embedding at the attended position. Blocking the query route also made no difference, 0.93, because in the textbook circuit the query only needs to know the current token.
The negative control behaved exactly as theory predicts. With the query and key routes both blocked, the second layer cannot use anything the first layer wrote to decide where to look. That turns the two-layer model, for the purpose of finding the right place, into a one-layer model. All three seeds stalled at about 0.14, almost the same as a true one-layer model at 0.13. When nothing written by the first layer can steer the second layer's attention, induction never forms.
Without positional embeddings, all three seeds stalled at 0.25 to 0.27. That is better than the negative control, which is worth noting. A causal transformer can recover some sense of position from the mask alone, since each position sees a different number of earlier tokens. But it does not recover enough to place the previous-token step reliably. Some positional signal is needed for the circuit to work well.
The surprise: blocking the textbook route
The result I did not expect came from blocking the key route. On the textbook account, this should be fatal: the induction head finds its target by reading the previous-token tag in the keys, and with the keys blocked there is no tag to read. The first seed I ran did fail, ending at 0.16. The second reached 0.92. So did the third, the fourth and the fifth, ending between 0.91 and 0.94. Four of five runs learned induction with the textbook route removed.
To find out how, I measured three kinds of head in the trained models. A previous-token head attends to the position just before. A duplicate-token head attends to an earlier copy of the current token. An induction head attends to the position just after that earlier copy. The full model had built a previous-token head in layer 0, with a score of 0.48, and an induction head in layer 1, with 0.77. The keys-blocked model had no previous-token head at all, with a score of 0.00. Instead it had built a strong duplicate-token head in layer 0, with 0.68, feeding an induction head in layer 1 scoring 0.78.
That is Leela's route from Thought Experiment 2. The first head, at the current token, finds the earlier copy and brings its position into the stream. The second head's query then asks for "the position one after that", and the keys, which know only positions and tokens, can answer it. The information flows through the query, and the keys only supply positions. In the vocabulary of Elhage and colleagues, this is Q-composition. They mentioned it as a possibility; what the knock-out runs add is that training finds it reliably when the more common route is blocked.
This changes the answer to the paper's question. The minimal configuration is not "a previous-token head feeding the keys of an induction head". It is two attention steps in series, where something the first step writes reaches the second step's query or key, plus enough positional signal. The textbook circuit is one instance of that. It is the one training finds when both routes are open, in all three of my full-model seeds, but it is not the only one.
The query route, worked by hand
The minimal case from Part 1 can be rerun along Leela's route, and doing so makes the difference concrete. Take the same six tokens, "red cat sat red cat ?", and ask the model at position 5, the second "cat", to predict "sat".
- Layer 0, duplicate-token head. At position 5 the current token is "cat". The head looks for an earlier position holding the same token and finds position 2. It writes into the stream at position 5 a note that says, in effect, "my earlier copy is at position 2".
- Layer 1, induction head. Its query at position 5 is now built from that note, so it can ask for "position 3", the one after the earlier copy. The keys know only each position's own token and place, which is enough to answer.
- The value at position 3 is "sat". The head copies it, and the model predicts "sat", exactly as before.
The output is identical. What changed is where the work is done. In the key route, every position does a little work in advance, tagging itself with its predecessor, and the current position asks a simple question. In the query route, nothing is prepared in advance, and the current position does all the work, finding its own earlier copy and then stepping one place forward. One is like a library that labels every shelf. The other is like a reader who remembers where she saw a book last and walks there. The picture breaks in one place: neither route walks anywhere. Both are a single weighted average computed at every position at once.
Static view: the full model, one head per layer, values blocked, queries blocked and keys blocked (four of five seeds) all reach about 0.92 accuracy on the second copy, with a sudden rise near step 1,000. Without positions runs end near 0.27. With queries and keys both blocked, runs end near 0.14.
Timing as a second measure
Final accuracy says whether a variant can learn induction. The step at which it first passes 50 per cent says how hard the circuit is to find. The full model crossed that line at step 1,000 in two seeds and 1,100 in the third. The three-layer model crossed at 900, slightly earlier, presumably because an extra layer gives more places for the parts to form. One head per layer crossed at 1,300, which suggests that a second head in each layer helps the search, even though it is not needed for the final circuit.
The keys-blocked runs were the most variable. Successful seeds crossed at 900, 900, 1,100 and 1,700. So the query route is not slower on average, but its timing is less predictable, and one seed never crossed at all. That pattern fits a route with fewer paths to it from a random start. It also warns against reading too much into one run of any variant. Had I run only seed 1 with the keys blocked, I would have written the textbook account into this paper as a finding.
The failure boundary
The seed that failed matters as much as the four that succeeded. With the keys blocked, seed 1 stayed at 0.16 for the whole run. I do not know why. One possibility is that the duplicate-token route has a narrower path to it from a random start, so some initialisations never find it within 4,000 steps. Another is that seed 1 found a partial strategy early and settled there. The safe conclusion is that the query route is learnable and usually learned, and that four of five is a small sample. That is why claim C3 is marked "likely", and why prediction P17-1 asks for twenty seeds.
There is also a boundary on what these toy results say about real models. My models have no MLPs, a vocabulary of 64 and one task. Real language models are trained on text, where previous-token information is useful for many other jobs. A previous-token head is likely to exist anyway, which would make the key route cheap to find. So the finding is about what is possible and what is necessary, not about what real models usually do. Part 6 looks at a real model.
Reading the table as a recipe
Put together, the eight variants give a short recipe, and it is worth stating plainly because it differs from the version usually repeated. The first ingredient is two attention steps, one after the other. A single step cannot do the job when the gap varies, and both my one-layer runs and the theory of Sanford and colleagues agree on that. The second ingredient is a link: something written by the first step has to reach the query or the key of the second. Without it, the two steps behave like one.
The third ingredient is position. The model needs to know where things are, either to tag each token with its predecessor or to step one place forward from an earlier copy. The causal mask gives a weak version of this for free, which is why positionless runs reached about a quarter accuracy and no further. Everything else in the architecture is optional for this task. That includes MLPs, extra heads, extra layers and the value route.
The recipe is minimal in a specific sense. It lists what was necessary across the variants I tried, and the one-head model shows that the listed parts are sufficient. It does not prove that no stranger configuration could work, and a reader with a cleverer block might find one. But it is a smaller and more honest recipe than "you need a previous-token head", which is what I believed before running these tests.
Exactly how the routes were blocked
In the second attention layer, the query, key and value inputs were set separately. For an open route the input was the residual stream after layer 0; for a blocked route it was the sum of token and position embeddings, the stream before layer 0. The layer's output was still added to the full stream. In the three-layer model only the final layer was configurable, and all routes were open. Adam optimiser, learning rate 0.001, batch size 64, 4,000 steps, evaluation on 512 fixed held-out sequences.
Part 4
A plateau, a cliff, and a small cost
The second half of the paper's question is about training: what has to happen for the circuit to appear? The learning curves answer the first part. In the full two-layer model, accuracy on the second copy was 0.10 at step 800, 0.25 at step 900 and 0.80 at step 1,000. Most of the skill arrived in about 200 steps, after 800 steps of almost nothing. The one-layer model never had a cliff; it crept up to 0.13 and stayed there.
Nandini's three quiet weeks followed by her Thursday are a fair picture, with one correction. She knew what she was trying to learn. A transformer does not. During its plateau, it is not practising a trick it already knows. It is drifting slowly through weight space, getting small rewards for partial strategies, until the parts of the circuit happen to be good enough to start rewarding each other.
The parts arrive together
I tracked both halves of the circuit throughout training. At step 0 the best layer-0 head put 0.07 of its attention on the previous position, about what an untrained head gives. It stayed there for 700 steps. Then, between steps 800 and 1,000, the previous-token score rose to 0.34 and the induction score in layer 1 rose from 0.07 to 0.72. Neither half formed first and waited for the other. They rose in the same window.
That pattern is what you would expect if each half only pays off when the other is partly there. A previous-token tag is useless unless something reads it. An induction head is useless unless the tag exists. So the gradient for each is small while the other is weak, and grows as the other improves. That is a feedback loop, and feedback loops produce cliffs. It is the same picture Reddy proposed, with abrupt learning arising from "nested nonlinearities sequentially learned during training", and that Singh and colleagues found in their three interacting subcircuits.
The same event seen as memory
There is a second way to describe what happens in the cliff, and it helps to have both. Bietti and colleagues (2023) studied a two-layer transformer on text in which some token pairs follow fixed global rules and others depend on the context. They found that the model learned the global pairs quickly and developed the induction mechanism more slowly. They described the weight matrices as associative memories, stores that link one pattern to another, and showed how gradients fill those stores during training.
In that picture, the previous-token head is a memory that links each position to its predecessor's token. The induction head is a memory that links "the token I am now" with "the position whose predecessor was me". Each store is filled faster when the other is already partly filled, because the gradient for one passes through the other. The cliff in my runs is the moment both stores become reliable enough to reinforce each other. Nandini would recognise the description. Her notes became useful once she had the habit of looking back, and the looking back became useful once the notes were reliable.
The interactive below is a toy version of that loop. Skill is the product of several parts, and each part improves only in proportion to how well the others already work. With one part, learning is smooth from the start. With two or three, there is a long plateau and then a sudden rise. It is a cartoon of a transformer, yet it shows why "the parts must work together" is enough to produce a phase change without any special trigger.
Static view: with one part, the skill rises smoothly from the start. With two parts that only pay off together, there is a plateau and then a sharp rise; with three, the plateau is longer and the rise sharper.
An analogy, and its break
Leela's route has a familiar everyday form. In a long document, you press Ctrl+F, type the current word, and jump to its last occurrence. Then you press the right arrow once and read the next word. That is a duplicate-token step followed by a positional step, the query route exactly. The analogy breaks in two places. Ctrl+F finds exact matches and jumps to one of them; an attention head spreads its weight softly over every match and can be fooled by near matches. And you decide to press the keys, while the transformer's two steps happen whether or not a repeat is present, which matters for what comes next.
A cliff that can look like a slope
The cliff in a single run is sharp, but there is a simple way to hide it. Average the learning curves of several runs whose cliffs fall at different steps, and the average rises gradually. My keys-blocked runs had cliffs at steps 900, 900, 1,100 and 1,700. Their average curve climbs over about a thousand steps, and someone who saw only the average would describe a slow, steady skill. Each underlying run, by contrast, learned the skill in a couple of hundred steps.
The reverse error is just as easy. A metric that gives credit only for getting a whole stretch of tokens right will make even a gradual improvement look sudden, because whole-stretch accuracy stays near zero until per-token accuracy is high. My measure is per-token accuracy, and the phase change still appears in it, alongside a sudden change in the attention scores. That is why I trust this cliff. Paper 20 in this series returns to the general question of when a sudden curve reflects the model and when it reflects the measure.
What the data has to provide
There is one more condition for the circuit to appear, and it is in the data rather than the model. Induction only forms if repeats are common enough in training to pay for it. Chan and colleagues (2022) found that in-context learning emerges when the training data is "bursty", with items appearing in clusters rather than spread evenly, and when there are many rarely occurring classes. Natural language has exactly these properties. Names, technical terms and phrases recur within a document far more often than their overall frequency would suggest.
In my task every sequence contains a repeat, which is why the circuit forms in about 1,000 steps. Edelman and colleagues found that when an easier strategy is available, such as predicting from single-token counts, models can spend a long time on it first, and that the easier solution "may delay formation" of the better one. So the minimal recipe has an ingredient that is not part of the circuit: a training distribution in which repeats are frequent and copying them pays.
Part 5
What the circuit means for in-context learning and for recognition
The induction head became famous because of a larger claim. Olsson and colleagues argued that it might be the mechanism behind most in-context learning, the ability of a trained model to pick up a pattern from its prompt without any change to its weights. That claim has been tested several times since 2022, and the results are now clearer, if less dramatic. Induction heads matter a great deal for copying and for some kinds of pattern use. For few-shot learning of tasks, other heads appear to matter more.
What later work found
Crosbie and Shutova (2024) ablated induction heads in Llama-3-8B and InternLM2-20B and found that few-shot performance on abstract pattern recognition tasks fell by up to about 32 per cent, close to chance in some cases. That is strong evidence that the heads carry real weight in larger models. Yin and Steinhardt (2025) compared induction heads with function vector heads, which Todd and colleagues had found carry a compact representation of the task being demonstrated. Across a range of models, ablating function vector heads hurt few-shot accuracy far more than ablating induction heads. They also found that some function vector heads start training as induction heads before changing role.
So the current picture has layers. Induction is an early, simple circuit for using context, and it may be a stepping stone towards more abstract ones. It does a great deal of the copying in a model, and some of the pattern matching. But "the model learned a task from three examples" usually involves more than finding a token and copying its successor.
The most interesting recent result for this paper is a true story about paraphrase. Feucht, Todd, Wallace and Bau (2025) found two kinds of induction head in large models. Token induction heads copy exact tokens, as in my toy models. Concept induction heads copy whole words and phrases by meaning. When the authors ablated the token heads, the models did not lose the ability to repeat. They paraphrased instead. Asked to repeat a passage, they produced its meaning in different words. Copying verbatim and copying the sense turned out to run along separate routes, and losing one exposed the other.
Nandini has the same two routes, as every good stenographer does. When the advocate says "the petitioner's late father" for the third time, she types it from the shape of the words. When he says "the deceased parent of my client", she recognises that he means the same man, even though no token matches. My toy models only have the first route. Whether the second is "recognition" in a richer sense is the question for the rest of this part.
Recognition in two traditions
Indian philosophy has a long discussion of exactly the operation an induction head performs. In Nyāya, recognition, pratyabhijñā, is the cognition expressed in the standard example "this is that Devadatta", so 'yam devadattaḥ. You see a man now; you remember a man from before; and you know that they are one person. Nyāya philosophers argued about whether this is a single act of perception, coloured by memory, or a combination of perception (pratyakṣa) and memory (smṛti). Buddhist critics argued that it is two separate cognitions joined by imagination, because the man of then and the man of now are different moments.
The Kashmir Śaiva thinkers Utpaladeva and Abhinavagupta took the word for their whole school. For them, recognition is not just one cognitive act among others. It is the structure of self-knowledge: consciousness recognising itself as what it always was. Their argument against the Buddhist view was that joining the then and the now requires a single awareness present to both. Without it, there are two snapshots, and nothing that knows they match.
Against that background the induction head looks almost like a diagram of the debate. It has a memory step, the previous-token tag laid down at every position. It has a present step, the query from the current token. And it has a joining step, the attention score that is high where the two match. In the layers of the Recognition Theory of Mind, the hypothesis this series tests, that is a clean L2 operation: something made available and used. What it lacks is any component that represents the match itself as a match. Nothing in the circuit notes "this is that". The copied token simply arrives.
Where the circuit sits in the RTM layers
It is worth placing the circuit carefully, because the layers are easy to overstate. At L1, spanda, the hypothesis places bare activity: signals moving through a system. Every attention head has that. At L2, prakāśa, it places availability: information that is present and used by later processes. The induction head is a clear case, since the previous-token tag is made available to a later head that reads it and acts. At L3, vimarśa, it places reflexive grasp: a system's representation of its own contents as its own.
The question this paper raises is whether recognition, L4 in the scheme, can occur without L3. The Śaiva answer is no. Recognition in their sense requires a knower who holds the then and the now together. The engineering answer, so far, is that a functional analogue of recognition can occur with nothing like L3 at all. These answers need not conflict. They may describe two different things that share a name, a functional join and a known join. The value of the induction head is that it gives the functional join a precise form, so we can ask exactly what the known join would add.
What the toy cannot show
It is worth being plain about the distance between my models and the models people use. In my task, every repeat is exact, every token is random, and the only useful skill is copying. In natural text, repeats are often partial, words carry meaning, and copying competes with dozens of other skills for the same heads. A real model's induction heads have to decide, in effect, when copying is the right move. My toy heads never face that decision; they always copy.
That gap is why the knock-out findings should be read as claims about possibility. They show what a transformer can use to build induction, and what it cannot do without, on a clean version of the task. They do not show how often real models take each route, or how the routes share work with the concept heads that Feucht and colleagues found. Closing that gap needs the same instruments applied to larger models, with the route test that my GPT-2 audit leaves out.
Where the stenographer story breaks
The running example has served well, but it should be retired honestly. Nandini knows she is predicting, and she can choose not to. The induction head has no such choice; it fires on every repeat, which is why it copied the cough. Nandini learned her habits in order, one after the other, while the model's two halves formed together. And Nandini understands the case, while the model is copying random tokens it has no reason to care about. The map below shows where the analogy holds and where it gives way.
Part 6
Auditing a real model for induction
A toy result is only useful if the same instruments work on a model that was not built for the purpose. So I ran the standard induction audit on GPT-2 small, a 124-million-parameter model trained on web text in 2019, with twelve layers of twelve heads each. This is the realistic worked case: the same questions as in the toy models, asked of a model trained on natural language.
The audit, step by step
- Build the probe. I sampled 50 random tokens and repeated them once, giving a 100-token sequence, and made eight such sequences. Random tokens matter, because nothing in the model's knowledge can predict them. Only the context can.
- Measure behaviour. Average loss on the first copy was 12.6, which is what a model gives to tokens it cannot predict. On the second copy it was 0.21. The model copies almost perfectly.
- Score every head. For each of the 144 heads I measured how much attention it put on the position after the earlier copy of the current token. Ten heads scored above 0.4. The top five were all in layers 5 to 7: head 5.5 at 0.94, 6.9 at 0.92, 7.10 at 0.92, 5.1 at 0.91 and 7.2 at 0.85.
- Find the feeders. The strongest previous-token head was 4.11, at 0.99. Duplicate-token heads appeared in layers 0 and 3, which means both routes from Part 3 have raw material in this model.
- Ablate and compare. Zeroing the outputs of the top six induction heads raised second-copy loss from 0.21 to 6.04. That is the decisive test, and it needs a negative control.
The control was five sets of six heads chosen at random from those with low induction scores. Zeroing them raised second-copy loss only to between 0.37 and 0.63. So the effect of the top six heads is not just the effect of removing any six heads from a small model. Zeroing the top four previous-token heads raised the loss to 0.35, a much smaller effect. That fits the redundancy you would expect in a trained model, where several heads and the MLPs can supply previous-token information.
The audit agrees with Olsson and colleagues, who found induction heads in the middle layers of small pretrained models. It also shows the limits of a single audit. Six heads off still leaves loss at 6.04, well below the 12.6 of the first copy, so some copying survives. Some of that is probably the weaker induction heads in layers 9 and 10. Some may be a route the head scores do not capture at all. And none of this tells us whether GPT-2 uses the key route or the query route, which would need a path-level analysis I have not done.
Try the rule yourself
The interactive below applies the ideal induction rule to any sentence you type. At each word, it looks for the most recent earlier copy of that word and predicts whatever followed it. Green means the prediction matched the next word; red means it did not. Try a sentence with a repeated phrase, and then one where a common word such as "the" appears twice with different words after it.
Static view: on "Your Lordship the petitioner's late father sold it and Your Lordship the petitioner's late", an ideal induction rule makes four checkable predictions in the second half ("Lordship", "the", "petitioner's", "late") and gets all four right, then guesses "father" for the word not yet spoken. On a sentence such as "the cat saw the dog", it predicts "cat" after the second "the" and is wrong: it copies whenever a token repeats.
The second kind of sentence shows why a pure induction rule is not enough for language. Real models combine induction with everything else they know, which is why GPT-2 does not predict "cat" every time it sees a second "the". In the toy models there is nothing else to combine it with, and the rule is applied in full, cough and all.
A third analogy, and its break
Anyone who has sat in a temple bhajan knows the induction rule from the inside. The lead singer sings a line, and the group sings it back. After a few rounds, when the lead begins a familiar line, the group is already singing the end of it. They have found where that opening came before and are continuing from there. The analogy breaks because the group also knows the song as a whole, from years of singing it, while an induction head knows nothing beyond the current context. A better match would be a group hearing a song for the first time, half a line behind the lead.
Deciding what you are looking at
For a practitioner, the useful output of all this is a short decision procedure. When a model copies something from its context and you want to know whether induction is responsible, the tree below gives the order of tests. Measure second-copy loss on random repeats first, because without that behaviour there is nothing to explain. Then score heads, then ablate with a random-head control, and only then look for the route.
What a good induction report contains
Circuit claims are easy to make and hard to check, so it helps to agree in advance what a report should contain. From the audits in this paper, I would ask for five things. The first is the behaviour on random repeats, with first-copy and second-copy loss, so a reader knows there is something to explain. The second is head scores for every head, not only the top ones, so a reader can see how concentrated the effect is.
The third is an ablation of the top heads with a random-head control of the same size, so the effect is not just the cost of removing parts. The fourth is a statement of the route, key or query, with the evidence for it, or an explicit note that the route was not tested. The fifth is the number of models, seeds and sequences, because a single run can mislead. My GPT-2 audit meets the first three, admits the fourth is untested, and uses eight sequences on one model, which is a modest base.
Objections
The best case that I have this wrong
The strongest objection is that my toy task proves too little. A vocabulary of 64, no MLPs and one task is a long way from a language model. In real models, previous-token information is so useful for other jobs that the key route may be the only one ever found, and the query route may be a curiosity of small, artificial settings. If that is right, the textbook circuit is effectively necessary in practice, and my finding is about a model nobody uses.
A second objection is about the sample. Four of five keys-blocked runs succeeded, and one failed. Five runs are enough to show that the query route is possible, but not to say how likely it is. Different learning rates, longer training or a different gap distribution could change the picture. A careful reader should treat claim C3 as a demonstration, not a rate.
A third objection concerns my blocking method. Feeding the second layer's keys from the original embedding is one way to remove the key route, but it is not the only one. A different method, such as projecting out a learned direction, might leave other routes open or close them. The conclusion "the key route is not necessary" is only as strong as the claim that my block removed it, and I have not tested other blocks.
A fourth objection is that "minimal" is doing too much work. My models are minimal in the sense that I removed parts until they failed. But the failures might be failures of training, not of capacity. A model with the key and query routes blocked might be able to represent induction in principle and simply never find the solution with Adam in 4,000 steps. If so, C4 is a claim about learnability, not about the architecture. I accept that, and the paper's question is about what appears during training, so learnability is the right target. It still means the word "necessary" should be read as "necessary for this training to find it".
A fifth objection is that the GPT-2 audit is weaker than it looks. Zeroing a head's output is a crude ablation, because it pushes the stream into states the model never saw in training. Mean ablation, which replaces the output with its average, often gives smaller effects. The random-head control helps, since it uses the same crude method, but it does not remove the worry. A reader who wants stronger evidence should rerun the audit with mean ablation and with path patching, and I would expect the main result to hold at a smaller size.
The objection to Part 5 is the sharpest. Bringing Pratyabhijñā into a paper on attention heads can look like decoration: an old word laid over a new mechanism. My answer is that the Nyāya and Śaiva analysis makes a distinction, between matching and knowing a match, that turns into a testable question about circuits. If that question cannot be made sharper, the objection wins.
Tests
What would change my mind
Several results would make me revise this paper. If a rerun with twenty seeds found that the keys-blocked model rarely succeeds, I would downgrade C3 and say only that the query route is possible. If a different method of blocking the key route produced models that always fail, I would suspect my block of leaving a back door open. If a one-layer model of moderate size learned variable-gap induction, the lower bound of Sanford and colleagues would need a closer look, and so would C1.
On formation, I would change my view if a carefully run training showed the previous-token head forming long before the induction head, with a stable gap between them. That would argue against the feedback-loop account of the phase change. On the rise in non-repeated loss, I would drop C8 if it failed to appear across seeds and with more natural data, as it may well do.
On the GPT-2 audit, I would change my view of C9 if mean ablation of the same six heads left second-copy loss below about 2. That would mean the large effect I measured was mostly the shock of zeroing, not the loss of the circuit. I would also want to know if a path-level analysis found GPT-2's induction heads reading duplicate-token heads rather than previous-token heads. That would turn my toy curiosity into a finding about a real model, and it is the test behind prediction P17-2.
Finally, on the broader claim from Olsson and colleagues, I would revise the picture in Part 5 if a large study found that ablating induction heads early in training prevented function vector heads from ever forming. That would mean induction is not just a stepping stone but a required one, which is a stronger claim than I make here.
On the RTM side, the decisive finding would be a component in a large model that represents "this is a repeat" and is read by several downstream processes, not just by the copying route. That would be evidence for something at L3 in the RTM sense. Equally, a thorough search that found every use of a repeat traceable to a copy-like route would strengthen the rival view in Thought Experiment 4, and I would say so.
Ledger
Claim Ledger
Each main claim of this paper, with how sure I am, what kind of evidence carries it, and what would knock it down.
| ID | Claim | Confidence | Evidence type | What would refute it |
|---|---|---|---|---|
| C1 | A small one-layer attention model cannot learn copy-and-paste induction when the repeat comes at a variable gap; two layers can. | established | Sanford, Hsu and Telgarsky (2024) lower bound; this paper's runs: 0.13 versus 0.93 | A one-layer model of similar size reaching high variable-gap accuracy |
| C2 | Two attention-only layers with one head each, positional embeddings and a link between the layers suffice; no MLP is needed. | established | Elhage et al. (2021); this paper's runs: 0.93 with one head per layer | A careful replication where this configuration fails to learn the task |
| C3 | The textbook key route is not strictly necessary: with it blocked, four of five runs found a query route through a duplicate-token head. | likely | This paper's knock-out runs and head scores (duplicate-token 0.68, previous-token 0.00) | Larger seed sweeps where the keys-blocked model almost never succeeds |
| C4 | Some link into the second head's query or key is necessary: blocking both leaves accuracy near 0.14, as in a one-layer model. | likely | This paper's runs: 0.136, 0.144, 0.146 | A model with both blocked that still learns induction |
| C5 | The value route is unnecessary: the copied token can come straight from the embedding. | likely | This paper's run: 0.92 with values blocked | Value-blocked runs failing across seeds |
| C6 | Positional information is needed for full performance; without it runs end near 0.25 to 0.27. | likely | This paper's runs, three seeds | A positionless model reaching high accuracy with longer training |
| C7 | Formation is abrupt because the parts pay off only together; previous-token and induction scores rise in the same few hundred steps. | likely | This paper's dynamics; Reddy (2023); Singh et al. (2024) | Runs in which one part forms long before the other and accuracy still jumps |
| C8 | While the circuit forms, loss on non-repeated tokens rises, because the new rule over-applies. | likely | This paper's runs: 4.17 to 4.41 | The same rise absent across seeds and tasks |
| C9 | In GPT-2 small, a few heads in layers 5 to 7 carry most copying of repeated random tokens. | established | This paper's audit: second-copy loss 0.21 to 6.04 with six heads off; consistent with Olsson et al. (2022) | Random-head knock-outs causing similar damage |
| C10 | The induction circuit has the formal shape of recognition ("this is that") at L2, without L3 or a recogniser. | speculative | Conceptual analysis under RTM | A component that represents the match itself and uses it, for instance to flag uncertain copies |
Dated predictions
| ID | Prediction | Judged by | How it will be judged |
|---|---|---|---|
| P17-1 | In a rerun of this paper's keys-blocked experiment with 20 seeds, at least half will exceed 0.9 second-copy accuracy, and those that do will show a duplicate-token head in layer 0. | This paper's scripts, rerun by me or anyone | Run e17c with seeds 1 to 20; report success rate and head scores |
| P17-2 | By the end of 2028, a published circuit analysis of a pretrained model with at least 1 billion parameters will report induction behaviour carried substantially by a query-side route. | Peer-reviewed or widely cited interpretability work | Literature search on 31 December 2028 |
Words
Glossary
- Induction head
- An attention head that, at a token A, attends to the position after an earlier A and copies the token there.
- Previous-token head
- A head that attends to the position immediately before the current one.
- Duplicate-token head
- A head that attends to an earlier occurrence of the current token.
- Composition
- One head using the output of an earlier head through its query (Q), key (K) or value (V).
- Attention-only transformer
- A transformer with attention layers and no MLP layers, used in research because it is easier to take apart.
- Induction score
- Average attention weight a head puts on the correct earlier position on repeated random tokens.
- Knock-out training
- This paper's method: train variants with one route removed and see whether the behaviour still forms.
- Phase change
- A short stretch of training in which a skill appears quickly after a long plateau.
- In-context learning
- A model using information in its prompt to improve its predictions, without changing its weights.
- pratyabhijñā प्रत्यभिज्ञा
- Sanskrit: recognition; knowing a present thing as the same as one known before. RTM layer L4.
- smṛti स्मृति
- Sanskrit: memory; the trace of an earlier cognition.
- pratyakṣa प्रत्यक्ष
- Sanskrit: perception; direct present cognition.
- so 'yam devadattaḥ सोऽयं देवदत्तः
- Sanskrit: "this is that Devadatta"; the stock example of recognition.
- vimarśa विमर्श
- Sanskrit: reflective awareness; RTM layer L3.
Conclusion
Two looks in a row, and something to join them
This paper asked what is the smallest thing a transformer needs to learn copy-and-paste induction, and what has to happen during training for it to appear. The answer from the knock-out runs is short. It needs two attention steps in series, some link from the first step into the second step's query or key, and enough positional information. It does not need MLPs, more than one head per layer, or the value route. And it does not strictly need the textbook key route, because training will usually find the query route when the key route is blocked.
The answer about training is also short. The circuit forms suddenly, because its two halves are useful only together and rise together. It forms only if the data rewards copying. And it carries a small cost: for a while, the model applies its new rule too widely and gets worse at the tokens where the rule does not apply. In GPT-2 small, the same circuit is present in a handful of middle-layer heads, and removing them destroys most of the model's ability to copy.
The method matters as much as the result. Removing parts after training tells you what one trained model happens to use. Removing them before training, and training many times, tells you what the behaviour needs. Knock-out training is cheap: every run in this paper took about ninety seconds on two processor cores. It is a tool any reader can use, and the scripts that produced every number here are short enough to check in an afternoon.
The induction head is also the clearest example I know of a machine operation with the shape of recognition. It joins a stored trace with a present input and acts on the join. What it lacks is any sign of knowing that the join has happened. That gap is where the Nyāya and Śaiva analysis of pratyabhijñā becomes useful for engineers, because it names the part we have not found. Nandini knows when she has recognised a phrase. Her model, for now, only copies it.
The next paper in this series takes up the question this one leaves open. When a model learns a task from examples in its prompt, is it running something like gradient descent, or something more like Bayesian inference over tasks it has seen? Induction is the simplest case of using context. The harder cases are where the line between copying and learning starts to blur.
Evidence
Sources
Primary research
- Elhage, N., Nanda, N., Olsson, C. et al. (2021). A mathematical framework for transformer circuits. Transformer Circuits Thread. transformer-circuits.pub. Verified.
- Olsson, C., Elhage, N., Nanda, N. et al. (2022). In-context learning and induction heads. arXiv:2209.11895. Verified; abstract quoted.
- Chan, S. C. Y. et al. (2022). Data distributional properties drive emergent in-context learning in transformers. arXiv:2205.05055. Verified.
- Wang, K. et al. (2022). Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. arXiv:2211.00593. Verified.
- Reddy, G. (2023). The mechanistic basis of data dependence and abrupt learning in an in-context classification task. arXiv:2312.03002. Verified; abstract quoted.
- Bietti, A., Cabannes, V., Bouchacourt, D., Jégou, H. and Bottou, L. (2023). Birth of a transformer: a memory viewpoint. arXiv:2306.00802. Verified.
- Edelman, B. L., Edelman, E., Goel, S., Malach, E. and Tsilivis, N. (2024). The evolution of statistical induction heads: in-context learning Markov chains. arXiv:2402.11004. Verified; abstract quoted.
- Singh, A. K., Moskovitz, T., Hill, F., Chan, S. C. Y. and Saxe, A. M. (2024). What needs to go right for an induction head? arXiv:2404.07129. Verified; abstract quoted.
- Sanford, C., Hsu, D. and Telgarsky, M. (2024). One-layer transformers fail to solve the induction heads task. arXiv:2408.14332. Verified.
- Crosbie, J. and Shutova, E. (2024). Induction heads as an essential mechanism for pattern matching in in-context learning. arXiv:2407.07011. Verified.
- Todd, E. et al. (2024). Function vectors in large language models. arXiv:2310.15213. Verified.
- Yin, K. and Steinhardt, J. (2025). Which attention heads matter for in-context learning? arXiv:2502.14010. Verified; abstract quoted.
- Feucht, S., Todd, E., Wallace, B. and Bau, D. (2025). The dual-route model of induction. arXiv:2504.03022. Verified; abstract quoted.
- Radford, A. et al. (2019). Language models are unsupervised multitask learners. OpenAI technical report (GPT-2). Model used in this paper's audit, weights via Hugging Face.
Primary texts and translations
- Utpaladeva, Īśvarapratyabhijñākārikā, with Abhinavagupta's Vimarśinī; the doctrine of recognition. Standard editions; paraphrased rather than quoted.
- Nyāya discussions of recognition (pratyabhijñā) as joint memory and perception, with the example "this is that Devadatta". Standard textbook example; paraphrased.
Secondary and leads
- Vaswani, A. et al. (2017). Attention is all you need. arXiv:1706.03762. Background.
Reference
Cite this work
Mahapatra, R. R. (2026). Two looks in a row: the minimal circuit behind induction heads, and why it is not unique. Mind Frontier paper U0572. rajeshrmahapatra.com. https://rajeshrmahapatra.com/papers/mind-frontier/U0572-induction-heads-minimal-circuit.html. DOI: to be assigned.
@misc{mahapatra2026induction,
author = {Mahapatra, Rajesh Ranjan},
title = {Two looks in a row: the minimal circuit behind induction heads, and why it is not unique},
year = {2026},
howpublished = {Mind Frontier paper U0572, rajeshrmahapatra.com},
url = {https://rajeshrmahapatra.com/papers/mind-frontier/U0572-induction-heads-minimal-circuit.html},
doi = {10.xxxx/placeholder}
}