Learning or recognising? What a transformer does with the examples in its prompt: Rajesh Ranjan Mahapatra

The scene

Two tasters, five cups and a buyer nobody knew

Meera has worked in the tasting room of a tea broking house in Fort Kochi for eleven years. Every week, before the auction, buyers send in a few sample cups with their own scores and ask the brokers to score the rest of a lot the way they would. Meera has eight regular buyers. A hotel chain likes body and colour, a packer in Kolkata likes briskness, and an exporter cares about almost nothing except the leaf. When five scored cups arrive on a Monday, she usually knows which buyer sent them by the second cup. From there she simply scores the lot as that buyer always does.

Arjun joined as a trainee in the same month. He has no regular buyers in his head. When the five cups arrive, he looks at the scores, works out roughly how much the buyer seems to reward each quality, and applies that rule to the rest. On the Monday batch, Meera is faster and more accurate. On Wednesday, a new buyer from Dubai sends his first five cups. Meera decides he must be another hotel chain and scores the lot that way. Arjun fits his rule as usual. The buyer's reply makes it clear that Arjun was right.

The head taster, Mr. Thomas Kurian, listened to the two of them argue about it over lunch. "Both of you are guessing what this buyer wants from five cups," he said. "You only disagree about which buyers exist." Meera, Arjun, Mr. Kurian and the Dubai buyer are invented, though the Kochi tea auction is real. They will return in every part of this paper, because their disagreement is a fair picture of one of the liveliest arguments about large language models.

Figure 1. Recognise the buyer, or fit the buyer's rule. The tasting room, the tasters and the buyers are invented. They stand for task retrieval and algorithm learning in a trained transformer. illustrative

The argument is about in-context learning: a model's ability to perform a task from a few examples in its prompt, with no change to its weights. One camp says the model runs a learning algorithm inside its activations, something like gradient descent on the examples. Another says it performs Bayesian inference, working out which of the tasks it already knows the examples come from. The question this paper asks is the one in the topic list: does in-context learning implement an implicit learning algorithm, or retrieve task priors?

My answer comes from the published work and from experiments I ran on small transformers and on GPT-2 small. It depends on what the model was trained on. A model trained on a few tasks behaves like Meera. A model trained on endless tasks behaves like Arjun. And Mr. Kurian is right that this is not a contest between two kinds of mind. Both behaviours are Bayesian guesses with different priors, and gradient descent is one way of computing a guess, not a rival to Bayes. GPT-2 small, tested with flipped and arbitrary labels, behaves mostly like Meera.

Part 1

What in-context learning is, and two stories about it

The phrase entered common use with a true story from 2020. Brown and colleagues trained GPT-3, a language model with 175 billion parameters, on text from the web, and found that it could do tasks it had never been trained to do if a few examples were placed in the prompt. Show it three English words with their French translations, then a fourth English word, and it often supplies the French. The paper's title, "Language models are few-shot learners", named the surprise. Nothing in the model's weights changed between tasks. Everything happened in a single forward pass over the prompt.

That raised an obvious question. When a model "learns" from three examples, what is it doing? There are two natural answers, and they correspond to the two tasters. Either the model works something out from the examples, as Arjun does, or it uses the examples to find something it already knows, as Meera does. The first answer treats the prompt as training data. The second treats it as a clue.

A task small enough to answer the question

Language is too messy to settle this, because nobody knows exactly which tasks a language model met in training. So researchers built a cleaner version. Garg and colleagues (2022) trained transformers from scratch on prompts that each contain a tiny dataset. Every prompt has its own hidden rule, a linear function that turns a list of numbers into one number. The prompt shows a few input-output pairs and then a new input, and the model must predict the output. Because each prompt has a fresh rule, the model cannot memorise answers. It must use the examples.

Figure 2. Each prompt is a tiny dataset with its own rule. The setup follows Garg et al. (2022): train a transformer on many such prompts, then ask it to predict the query. Inputs have 4 numbers; up to 15 examples per prompt. mechanism diagram

In my version, every input is a list of 4 numbers drawn at random, the rule multiplies each by a weight and adds them, and a prompt contains up to 15 examples. Guessing zero gives an error of about 4, and a perfect prediction gives 0. With four unknown weights, a careful learner needs four examples to pin the rule down exactly. Before that, it has to guess, and how it guesses depends on what it expects rules to look like.

That last point is where the two stories part. A learner that knows only eight possible rules can guess very well after one or two examples, because it only has to tell eight candidates apart. A learner that expects any rule at all must wait for more data. The examples in the prompt are the same in both cases. What differs is the learner's prior, the spread of tasks it expects before it sees anything.

Figure 3. Two stories about the same prompt. The learning-algorithm account follows von Oswald et al. (2023) and Akyürek et al. (2023); the retrieval account follows Xie et al. (2022). The bottom panel is this paper's thesis. mechanism diagram

Two questions that look like one

The neuroscientist David Marr argued in 1982 that any information-processing system can be described at three levels. The first says what problem is being solved and what the best answer would be. The second says what procedure produces the answer. The third says what physical machinery runs the procedure. A cash register adds numbers, the procedure is column addition with carries, and the machinery is a circuit board. All three descriptions are true at once.

"Bayesian inference" is a first-level description. It says the model's answer is the best guess given what it expects and what it has seen. "Gradient descent" is a second-level description. It names a procedure, small steps downhill on an error measure. These are not competing answers to one question. A procedure can compute a Bayesian answer, and often does. Much of the argument dissolves once the levels are kept apart, though, as later parts show, not all of it.

Figure 4. Bayes and gradient descent answer different questions. The three levels follow David Marr's 1982 scheme for describing information-processing systems. "Gradient descent or Bayes?" mixes the first two rungs. conceptual diagram

A minimal worked case

Here is the smallest version that shows the difference, using a single number per cup rather than four. Meera knows four buyers. Each scores a cup by multiplying one quality, say briskness, by a fixed weight: 0.5, 1, 2 or 3. Arjun knows no buyers, and simply estimates the weight from the cups he sees.

One cup arrives. Its briskness is 1.2, and the buyer's score is 2.5.
Meera asks which of her four buyers best explains that score. Buyer 2 would have given 2.4, which is close. Buyer 3 would have given 3.6, and buyer 1 would have given 1.2, both far off. With a small allowance for tasting noise, nearly all her belief goes to buyer 2. She predicts the next cup with weight 2.
Arjun divides the score by the briskness and gets about 2.1. He predicts the next cup with weight 2.1.
If the buyer really is buyer 2, Meera is exactly right and Arjun is slightly off. If the buyer is a stranger with weight 1.5, Meera's belief still lands on buyer 2, because 2 is the nearest weight she knows, and she is wrong by 0.5. Arjun's estimate moves towards 1.5 as cups arrive.

Every step here is a Bayesian guess. Meera's prior puts all its weight on four values. Arjun's prior, in effect, spreads its weight over every value. The interactive below runs exactly this calculation. Press the button to add cups one at a time, and switch between a known buyer and a new one.

Static view: with four known buyers who weight one quality by 0.5, 1, 2 and 3, a known buyer with weight 2 is identified after one cup, with nearly all belief on that buyer. A new buyer with weight 1.5 sends nearly all belief to the buyer at 2, so the recogniser's guess is off by about 0.5, while the learner's estimate lands near 1.6 after one cup.

Figure 5. Watch the recogniser's belief move cup by cup. A one-number version of the regression task. Bars: belief over four known buyers given the cups so far, with small tasting noise. Line: the learner's least-squares estimate. Cup values are fixed so the demonstration repeats exactly. interactive, exact rule

Thought experiment 1

The specialist and the generalist on Monday morning

Return to the Monday batch, where the buyer is one of Meera's regulars. Suppose you could see only the first two cups and had to bet on which taster would score the rest of the lot more accurately.

Pause. Before you read on: with only two cups, does the taster who knows eight buyers do better or worse than the one who fits a rule from scratch?

Most people back Arjun, because "learning from the data" sounds more rigorous than "recognising the buyer". With two cups and four qualities to weigh, Arjun cannot pin the rule down; the data underdetermine it. Meera only has to tell eight buyers apart, and two cups are usually plenty. In my experiments, the model trained on 8 tasks had an error of 0.17 after three examples, against 1.06 for ridge regression, the best general-purpose learner for this task. With few examples, recognition wins whenever the task is familiar.

The lesson is that a strong prior is not a weakness. It lets a learner extract far more from very little data, which is the whole point of few-shot learning. The limit is the obvious one, and Wednesday supplies it. A strong prior is only an advantage when the world matches it, and the model has no way to tell, from inside, when it does not.

Part 2

How two camps formed, and how the data began to split them

The Bayesian story came first. Xie and colleagues (2022) proposed that in-context learning arises because training documents have long-range coherence: to predict the next word, a model must infer the latent "concept" a document is about. At test time, they argued, "in-context learning occurs when the LM also infers a shared latent concept between examples in a prompt". On this view, a prompt is a clue to which of the model's known concepts is in play. That is Meera's method.

A striking result soon seemed to support it. Min and colleagues (2022) replaced the labels in few-shot demonstrations with random ones and found that this "barely hurts performance" across 12 models, including GPT-3. What mattered was the label space, the kind of input text and the format of the prompt. If a model ignores whether the example answers are right, it is hard to say it is learning the mapping from them. It looks much more like the examples tell the model which task to perform, and the model supplies the answers from what it already knows.

Figure 6. Two camps formed in two years, then the data began to split them. Dates are first public versions. Saffron: learning-algorithm account. Lavender: Bayesian and task-vector accounts. Coral: tests on pretrained models. sourced

The learning-algorithm story came from the regression experiments. Akyürek and colleagues (2023) proved that transformers can implement gradient descent and closed-form ridge regression, and found that trained models "closely match the predictors computed by gradient descent, ridge regression, and exact least-squares regression". Von Oswald and colleagues (2023) went further. They gave a weight construction that makes one layer of linear self-attention equivalent to one step of gradient descent. Then they trained such layers and found, in their word, "remarkably", that the learned weights matched the construction.

That last finding is the second true story in this paper, and it is worth pausing on. Constructions showing that a network could compute something are common, and most trained networks ignore them. Here the optimiser, left to itself, arrived at the very weights a person had written down by hand. It is rare in deep learning for a theory of what a network does to be confirmed so directly.

The two camps were then brought together by an experiment that is the closest ancestor of mine. Raventós, Paul, Chen and Ganguli (2023) varied the number of distinct tasks in training. Below a "task diversity threshold", the model "behaves like a Bayesian estimator with the non-diverse pretraining task distribution as the prior". Above it, the model's behaviour "aligns with that of ridge regression", which corresponds to a Gaussian prior over all tasks. Meera and Arjun, in other words, are the same kind of learner trained on different worlds.

Meanwhile, tests on real language models found that both stories hold in part. Wei and colleagues (2023) showed that small models ignore flipped labels and rely on what they already know, while large models "can override semantic priors" when the examples contradict them. Pan and colleagues (2023) separated task recognition, which works "even without ground-truth labels", from task learning, the ability to pick up mappings not seen in training. They found that recognition does not improve much with scale, while learning does. And Shen, Mishra and Khashabi (2023) looked for the gradient-descent signature in pretrained models and found differences, including that in-context learning and gradient descent respond differently to the order of examples.

A third line of work looked inside pretrained models for the summary of a task. Hendel, Geva and Globerson (2023) found that the functions learned in context "often have a very simple structure". The model compresses the examples into a single "task vector", which then steers its processing of the query. Todd and colleagues (2024) found similar vectors, carried by a small number of attention heads. These can be lifted out of one prompt and added to another to trigger the same task. This is neither pure retrieval nor a learning algorithm in the regression sense. It is a compact code for "which task is this", built from the examples, that the rest of the network then uses.

Finally, Panwar, Ahuja and Goyal (2024) took the Bayesian view head on. A model with unlimited training data and compute is a Bayesian predictor for its training distribution, and they asked how far that view goes in explaining in-context learning. In several function classes, including mixtures of them, trained transformers behaved much as the Bayesian predictor would. The cases where they did not were informative about what the networks were missing. That framing is the one I find most useful, and Part 3 tests it directly.

The result of all this is a field in which nobody now holds either pure view. The question has shifted from "which one?" to "when, and how much of each?". This paper's experiments are an attempt to see that shift in models small enough to take apart.

Part 3

Training a Meera and an Arjun

To see the two tasters in a transformer, I trained two copies of the same small model on two different worlds. Both were four-layer transformers with a 64-dimensional stream and four heads per layer, the ordinary softmax kind used in language models. Each was trained for 6,000 steps on prompts of 15 examples from the regression task in Part 1. The only difference was the source of the rules. The first model saw rules drawn from a fixed pool of 8. The second saw a fresh rule in every prompt, drawn from a Gaussian, so it never met the same rule twice.

I then tested each model on 2,000 new prompts of two kinds. In the seen-task test, the rules came from the pool of 8. In the new-task test, the rules were drawn fresh. For every prompt and every number of examples, I compared the model's prediction with three reference estimators. Ridge regression is the best general learner for this task. The pool estimator is the Bayesian answer for a learner who knows the rule is one of the 8. One gradient step is the simplest version of the gradient-descent story. I also recorded the error of always guessing zero, as a floor any model must beat.

The model trained on eight tasks

On seen tasks, the narrowly trained model behaved like Meera. Its error fell from 3.6 with no examples to 1.28 with one and 0.17 with three. Ridge regression, which cannot know that only 8 rules exist, was at 2.83 and 1.06 for the same counts. The model was closer still to the pool estimator, which is what a perfect Meera would do. By 15 examples, the model's error was 0.02, close to perfect.

Figure 7. On familiar tasks, recognising beats learning. Four-layer transformer trained on 8 fixed tasks, 6,000 steps. With 3 examples: model 0.17, ridge 1.06, Bayes over the pool 0.004. 2,000 test prompts. Run in this paper's instrument. measured, synthetic data

On new tasks, the same model fell apart. With 15 examples, ridge regression had an error of 0.00, because 15 examples pin down 4 weights exactly. The narrow model's error was 2.24, about half the error of guessing zero. It improved a little with more examples, but never came close. The pool estimator did even worse, at 2.81, because it insists that the rule must be one of the 8. The model, in other words, was not a perfect Meera. It had picked up a little of Arjun's method too, enough to do slightly better than pure lookup on strangers.

Figure 8. Given a stranger, the recogniser keeps looking for an old face. Same model, new tasks drawn fresh. At 15 examples: model 2.24, ridge 0.00, Bayes over the pool 2.81. Run in this paper's instrument. measured, synthetic data

Thought experiment 3

The buyer who was not there

On Wednesday the Dubai buyer's five cups arrive, and Meera, with great confidence, scores the lot as the hotel chain would. When the buyer complains, she rechecks. She is now sure he is the packer from Kolkata, and scores the lot again, just as badly. After the third attempt she suggests to Mr. Kurian that perhaps the buyer does not know his own taste.

Pause. Before you read on: given 15 examples from a rule it has never seen, will the narrowly trained model do better or worse than simply picking the nearest of its 8 known rules?

Slightly better, which surprised me. The pool estimator, the perfect Meera, ended with an error of 2.81 on new tasks. The model ended at 2.24. So the network had not learned a pure lookup table. It had learned something that mostly picks a known rule and partly adjusts it. But the adjustment was small, and the model never said anything like "this is a stranger". It always committed to an answer, and the answer was always close to an old one.

The lesson is that a narrowly trained model fails confidently. It produces a prediction with no sign of doubt, because nothing in its training ever rewarded doubt. The limit is that I measured only error, not anything like confidence, and a model trained to report uncertainty might behave differently. Meera, at least, could have said "I do not know this buyer". My model had no way to say it.

How the comparison was made

A comparison of this kind needs care, because two estimators can have similar error while making quite different predictions. So I recorded two numbers for each model and estimator. The first is the error, the average squared gap between prediction and truth. The second is the distance, the average squared gap between the model's prediction and the estimator's, prompt by prompt. A model can only be said to behave like an estimator if the distance is small, not just if the errors are close. All numbers are averages over the same 2,000 test prompts, so differences between models are not differences in luck.

The pool estimator needs one choice: how much noise to allow in the fit. I used a small allowance, so that it behaves almost like a lookup that picks the best-fitting of the 8 rules. Ridge needs a similar choice, a small penalty on large weights, which I set close to zero. Neither choice changes the picture much, but both are written in the scripts so anyone can vary them.

The model trained on endless tasks

The broadly trained model behaved like Arjun, and very nearly like the best possible Arjun. On new tasks, its error was 1.16 with three examples, 0.54 with four, 0.08 with seven and 0.03 with fifteen. Ridge regression was at 0.98, 0.05, 0.00 and 0.00. The model was a little slow to reach the exact answer at the point where four examples first make it possible, but the shape of its curve was ridge's. The average squared distance between its predictions and ridge's, at 15 examples, was 0.03, on predictions that typically have a size of about 4.

Figure 9. Trained on endless tasks, the model behaves like the best learner, not a few gradient steps. Gradient baselines start from zero with the best step size for each prompt length. At 8 examples: model 0.055, ridge 0.000, 10 steps 0.40, 1 step 1.65. Run in this paper's instrument. measured, synthetic data

The model was nowhere near gradient descent of the simple kind. To give the gradient-descent story a fair chance, I computed one step and ten steps of gradient descent from zero, choosing the best step size separately for each number of examples. With eight examples, one step had an error of 1.65 and ten steps had 0.40. The model had 0.055. Plain gradient descent with a few steps cannot explain it. More steps would eventually converge to the least-squares answer, which is ridge, but at that point "gradient descent" and "ridge regression" describe the same destination, and the difference is only in how the model gets there.

On seen tasks, the broad model did about as well as on new ones: 1.38 with three examples, against the narrow model's 0.17. It had never been told that 8 special rules existed, so it could not exploit them. That is the price of generality. Arjun is never fooled by the Dubai buyer, but he is never as quick as Meera with her regulars either.

Figure 10. Paler means closer: each model sits next to a different estimator. Mean squared distance between the model's predictions and each estimator's, at 3 and 15 examples (column labels give the estimator and the number of examples). The 8-task model is closest to the pool estimator on seen tasks; the unlimited model is closest to ridge everywhere. Run in this paper's instrument. measured, synthetic data

The heat map makes the comparison direct. Each cell gives the distance between a model's predictions and an estimator's. The narrow model on seen tasks is close to the pool estimator at three examples. The broad model is close to ridge everywhere. And the narrow model on new tasks is far from both, which fits Thought Experiment 3: it is neither a perfect recogniser nor a learner, but something in between that fails on strangers.

Static view: the model trained on 8 tasks follows the Bayesian estimator for those 8 on seen tasks and stays near an error of 2.2 on new ones. The model trained on unlimited tasks follows ridge regression on both seen and new tasks, reaching about 0.03 with 15 examples.

Figure 11. Choose a model and a test, and see whose curve it follows. Indigo: the model. Saffron dashed: ridge regression. Teal dotted: Bayesian estimator over the 8 training tasks. Coral dotted: one gradient step. Errors above 5 are clipped. Run in this paper's instrument. interactive, measured

Thought experiment 2

Three descriptions of one taster

Arjun's notebook, if you could read it, would show him estimating each buyer's weights from the cups by least squares. A statistician watching him would say he is computing the posterior mean under a Gaussian prior. A third observer, watching only his hands, would say he is making small adjustments to his guess after each cup.

Pause. Before you read on: which of these three descriptions of the broad model is the true one?

All three can be true at once, and for the broad model at least two of them are exactly the same thing. When the hidden weights are drawn from a Gaussian, as in my training, the Bayesian posterior mean is ridge regression. That is a standard result, not an interpretation. So a model that matches ridge is, by construction, performing Bayesian inference for its training prior. Whether it gets there by something like gradient steps, by a closed-form solve, or by a route nobody has named, is a separate question about mechanism.

The lesson is that the same behaviour can fit several mechanisms, and matching an estimator's output says nothing about which procedure produced it. The limit is that my measurements only compare outputs. To tell the procedures apart, one would need to look inside, as Akyürek and colleagues did with probes for intermediate quantities, or to find cases where the candidate procedures disagree, as Shen and colleagues did with example order.

Controls and a negative case

Two checks support these readings. First, the floor: guessing zero scored between 3.5 and 4.2 at every prompt length, so both models are doing real work from the first example on. Second, the narrow model on new tasks is itself the negative control for the claim that it learned an algorithm. If it had, its new-task error would fall towards zero as examples accumulate, as ridge's does. It stayed above 2.2. Whatever the narrow model does, it is not a general learning procedure that happens to be fast on familiar tasks.

One more detail matters for the argument. Both models have identical architecture, identical training length and identical optimiser settings. Nothing about the network decides whether it becomes Meera or Arjun. The training distribution decides. That is the cleanest single finding in this paper. It is also the one most often lost in arguments about what "transformers" do in context, as if the answer could be given without saying what they were trained on.

Part 4

From recognising to learning as the tasks multiply

If eight tasks give a Meera and endless tasks give an Arjun, the next question is what happens in between. I trained four more models with pools of 12, 16, 32 and 128 distinct tasks, keeping everything else the same, and tested each on new tasks. The bar chart gives the error with 15 examples, where ridge regression scores zero.

Figure 12. Between 8 and 32 training tasks, recognition gives way to learning. One seed per pool size, 6,000 training steps each. Ridge regression scores 0.00. This repeats in miniature the threshold found by Raventós et al. (2023). Run in this paper's instrument. measured, synthetic data

The switch came early and fast. With 8 tasks, the error on new tasks was 2.24. With 12 it fell to 0.73, with 16 to 0.33, and with 32 to 0.06, already close to the unlimited model's 0.03. With 128 tasks it was 0.03, the same as unlimited. Most of the change happened between 8 and 32 tasks, a much smaller pool than I had guessed before running the sweep.

The other side of the switch is just as clear. On seen tasks with three examples, the model trained on 8 tasks scored 0.17. The 12-task model scored 0.42, the 16-task model 0.69, the 32-task model 0.85 and the 128-task model 0.99, close to ridge's 1.06. Every gain on strangers was paid for on regulars. No model in the sweep was both an expert recogniser and a good learner. That is what the Bayesian reading predicts: a sharper prior helps exactly when the world matches it, and hurts when it does not.

The interactive lets you step through the same models one at a time and see the whole curve. At the narrow end the curve flattens early, at a high error, because more examples cannot help a recogniser whose candidate list does not contain the answer. At the broad end it drops to the ridge curve. The interesting models are in between, where the curve begins to fall with more examples but has not yet reached the floor.

Static view: on new tasks, models trained on few distinct tasks keep a high error however many examples they see. As the number of training tasks grows, the curve drops towards ridge regression, which reaches zero by 5 examples.

Distinct training tasks: 8

Figure 13. Slide from a few tasks to endless ones. Each position is a separate model, one seed each, tested on 2,000 prompts with tasks it never saw. Saffron dashed: ridge regression. Run in this paper's instrument. interactive, measured

What the switch means

A plausible reading of the switch is that, below some number of tasks, memorising the pool is the cheaper way to reduce training loss, and above it, learning a general method is cheaper. With 8 tasks, the model can store all 8 rules in its weights and simply pick one. With thousands, storing them all becomes harder than learning to compute an answer from the examples. Raventós and colleagues reported the same pattern at larger scale, and found that above the threshold the model generalised to tasks far outside the training pool.

The switch has an awkward implication for language models. Nobody knows how diverse their training "tasks" are, because text does not come labelled with tasks. Some patterns in text, such as translating between two common languages, appear in millions of variations. Others, such as an unusual code format, may appear a handful of times. A single model could be an Arjun for the first kind and a Meera for the second. That would explain why real models seem to learn some new mappings and ignore others.

Meera with thirty-two buyers

The tasting room makes the trade-off concrete. Suppose the broking house grows and Meera now has 32 regular buyers. Two cups are no longer enough to tell them apart, because several buyers now like similar things. She starts to work more like Arjun, weighing the cups and estimating what the buyer wants, with her known buyers as a rough guide. On Monday she is a little slower than she used to be. On Wednesday, when a stranger arrives, she does far better. With 128 buyers, she has in effect stopped recognising anyone and simply estimates.

This is a smooth change in the taster, with no moment at which she switches method. The sweep looks the same. There was no single pool size at which the model flipped from one behaviour to the other. The word "threshold", which Raventós and colleagues used, fits their larger models better than mine, where the change looked like a steep slope spread over a factor of four in pool size.

An analogy, and where it breaks

The difference is familiar from clothing. An off-the-rack shop holds a few sizes. A customer is quickly matched to the nearest one, and if the customer is unusual the fit is poor. A bespoke tailor takes measurements and cuts to them, which takes longer but fits anyone. A shop with a few sizes is Meera; a shop with thousands of sizes starts to behave like a tailor, because some size is always close. The analogy breaks in one useful place. A trained model does not just pick the nearest size. The narrow model's error of 2.24 on new tasks, better than the 2.81 of pure lookup, shows that it blends sizes. A shop cannot sell a customer the average of two jackets.

The failure boundary

The boundary of these results is easy to state. Each pool size was trained once, with one seed, for 6,000 steps. A threshold located from one run per size is a rough location, which is why prediction P18-1 asks for five seeds. Raventós and colleagues also found that the threshold depends on input dimension and training length, so my numbers for 4-number inputs say nothing directly about other sizes. And all of this is linear regression, the simplest task where the question can be asked. In nonlinear tasks, the best general learner is harder to define, and so is the comparison.

Part 5

Where gradient descent is literally true

None of this means the gradient-descent story is wrong. It means it is true of particular models. To see where, I trained the simplest model from the theory: a single layer of linear self-attention, with no softmax, no MLP and no stacking. It reads the examples as tokens containing both input and output, and the query as a token with the output left blank. I trained it on unlimited tasks for 6,000 steps, then compared its predictions with one step of gradient descent at a range of step sizes.

Figure 14. One linear attention layer, trained on regression, becomes one gradient step. One-layer linear self-attention trained on unlimited tasks, 6,000 steps, tested with 8 examples. Distance to one gradient step with step size 0.5: 0.006, against a typical prediction size of about 4. This repeats von Oswald et al. (2023). Run in this paper's instrument. measured, synthetic data

The match was almost exact. At a step size of 0.5, the average squared distance between the model's predictions and one gradient step, with 8 examples, was 0.006. At other step sizes it rose quickly, to 0.23 at 0.3 and 0.27 at 0.7. The model's error curve also matched the curve for one gradient step at that step size almost number for number: 2.20 against 2.21 with four examples, 1.60 against 1.59 with seven. The trained layer is one step of gradient descent, as von Oswald and colleagues found. Ahn and colleagues (2023) proved the general case: for a single linear attention layer, the global minimum of the training objective implements one step of preconditioned gradient descent.

Why a linear layer becomes a gradient step

The reason is simpler than it sounds. One gradient step on a regression problem, starting from zero, moves the weight guess in the direction of the sum of each input times its output. The prediction for a new input is then that sum, multiplied by the new input. A linear attention layer computes exactly this kind of quantity. Its query comes from the new input, its keys from the example inputs, and its values carry the outputs. Multiply query by key, weight each value by the result, and add: that is the sum of input times output, projected onto the new input. Training only has to find the right scaling, which is the step size.

That also explains why my four-layer softmax model did not look like gradient descent. The softmax changes the weighting from a plain product into something sharper and nonlinear, and four layers with MLPs can compute far more than one step. Ahn and colleagues showed that stacked linear layers can implement several steps of a refined gradient method. Nothing forces a softmax model to stay within that family, and mine reached ridge-like answers much faster than ten plain steps would.

One gradient step, by hand

Here is the step with real numbers, using the one-number version of the task from Part 1. Two cups arrive, with briskness 1 and 2 and scores 2 and 4, so the true weight is 2. Start the guess at zero. The gradient step adds, for each cup, its briskness times its score, giving 1 × 2 + 2 × 4 = 10. Divide by the number of cups and multiply by the step size; with a step size of 0.3, the new guess is 0.3 × 10 / 2 = 1.5.

A third cup arrives with briskness 1.5. The one-step prediction is 1.5 × 1.5 = 2.25, while the true score is 3. One step has moved the guess most of the way, but not all of it. Ridge regression, solving directly, gets a weight of almost exactly 2 and predicts about 3. That gap is what the gradient figure shows at scale: one step is a crude learner, and a single linear attention layer, which is one step, inherits its crudeness.

An analogy, and its break

Gradient descent is how a cook adjusts salt. Taste, add a pinch in the direction of what is missing, taste again. One linear attention layer is a cook allowed exactly one pinch, sized to be right on average. Ridge regression is a cook who can work out the right amount from the recipe directly. The analogy breaks because a cook tastes the dish after each pinch, while a single attention layer gets no second taste. Its one step is computed from the examples in a single pass. Several layers can take several steps, but each is fixed by training, not by tasting.

The lesson for the larger debate is that "transformers do gradient descent in context" is a true sentence about a specific model class, not a general law. It holds exactly for one linear layer trained on regression. It holds approximately for some deeper linear models. For softmax transformers trained on language, Shen and colleagues found that the theory's assumptions do not match, and that pretrained models respond to the order of examples differently from gradient descent. Taken together with my results, the safest statement is that gradient descent is one procedure a transformer can use, which training finds under particular conditions.

Part 6

Recognition, comparison and inference

Indian epistemology has a vocabulary for exactly the distinctions this paper has been drawing, and it is more precise than the English words "learning" and "recognising". The Nyāya school counted four means of valid knowledge, pramāṇa: perception, inference, comparison and testimony. Recognition, pratyabhijñā, was analysed separately, as a cognition that joins a present perception with a memory: "this is that". Each of these maps onto a different thing a model might do with its prompt.

Meera's method is recognition. Five cups arrive, and she knows them as the hotel chain's, joining what she tastes now with what she remembers. Arjun's method is inference, anumāna: from the signs in front of him, the scores, he infers what he cannot see, the buyer's rule. The Nyāya tradition insisted that inference rests on a known general connection between sign and signified, vyāpti. For Arjun, that connection is the assumption that scores are a weighted sum of qualities. For the broad model, it is the linear form of every rule in its training.

The prior as latent impression

There is a third idea that fits the training prior itself. In several Indian schools, past experience leaves saṃskāra, latent impressions that shape later cognition without being present to it. A trained model's weights are a fair example. The 8 rules of the narrow model are not stored anywhere as a list one could read. They are impressions spread across the weights, and they show up only in how the model responds to a prompt. When the model meets five examples, its saṃskāras decide what the examples can mean to it.

That framing makes one point vivid. The difference between Meera and Arjun is not in what they do with the cups in front of them. Both weigh the evidence against their expectations. The difference is in their impressions: eight sharp ones, or a broad, smooth expectation that any weighting is possible. What a model can learn from its prompt is set by what training left behind, and that is as true of the broad model as of the narrow one. Its "generality" is itself an impression, the Gaussian prior, and it would fail on rules of a kind it never saw, such as nonlinear ones.

Figure 15. The accounts lie on one line, not in two camps. Placements are my judgement. Task vectors from Hendel et al. (2023) and Todd et al. (2024); ridge and gradient descent from Akyürek et al. (2023) and von Oswald et al. (2023). illustrative, author judgement

Thought experiment 4

The rival who says recognition is just a lookup table

A sceptical engineer reads the last section and objects. "You are dressing up a lookup table in Sanskrit. The narrow model stores eight answers and picks the closest. Calling that pratyabhijñā flatters it. And the broad model learns genuinely new tasks, so there is nothing to recognise at all. Your Recognition Theory of Mind has nothing to say about the interesting case."

Pause. Before you read on: is the second objection right? Does the broad model's success on new tasks leave recognition out of the picture?

I think the second objection is largely right, and it is worth conceding plainly. On a brand-new rule, the broad model does not recognise anything as "that". It infers. If RTM claimed that all in-context learning is recognition, the broad model would refute it. The narrow model is a better fit, but the first objection lands too: what it does is closer to nearest-match lookup than to the Śaiva notion of a knower recognising an object as the same.

The lesson is that recognition, in the RTM sense, describes one regime of in-context learning, the retrieval regime, and only its shape. It does not describe the learning regime, and it does not credit either model with a recogniser. The limit is that the concession is about my toy models. In a language model, the two regimes probably run side by side, and the recognition framing may still pick out something real about how prompts select behaviours learned in pretraining.

Comparison, the third way

There is one Nyāya means of knowledge that the debate has not used, and it may be the most interesting. Comparison, upamāna, is defined in the Nyāyasūtra as knowing something through its likeness to something already known. The stock example is the gavaya, a wild ox. A villager who has never seen one is told by a forester that a gavaya is like a cow. Later, in the forest, he sees an animal that is like a cow, and knows: this is what "gavaya" names.

Thought experiment 5

The gavaya in the forest

Picture the villager at the edge of the forest. He has a description, "like a cow", and a memory of cows. He has never seen a gavaya. An animal steps out of the trees.

Pause. Before you read on: is the villager recognising, inferring, or doing something else?

The Nyāya answer was that he is doing something else, and that is why upamāna was counted as a separate means of knowledge. He is not recognising, because he has never seen this animal. He is not simply inferring, because he is not reasoning from a general rule. He is using a likeness to something known to fix the meaning of a new word. It is half Meera and half Arjun: an old impression, used as a template for something new.

The lesson is that the third option may describe language models best. A prompt with a few examples of an unusual task often resembles tasks the model knows well, and the model may use that likeness to build a new behaviour from old parts. Task vectors, which Hendel and colleagues (2023) found compress a prompt's examples "into a single task vector", might be one mechanism for this. The limit is that this is an analogy, not a finding. Nobody has shown that a model builds new task vectors by likeness to old ones, and the test would need careful design.

[AUTHOR NOTE: optionally add a sentence on how upamāna or saṃskāra has figured in your own reading or practice, if it connects]

Pratibhā, the sudden grasp of meaning

Bhartṛhari offers a third Indian frame, and it fits the recognition regime closely. For him, the meaning of a sentence is not built up word by word like a sum. It arrives as a whole, in a flash of understanding he calls pratibhā. The words are occasions for that flash, not its parts. A listener who has heard many sentences grasps a new one at once, because the words call up a meaning already shaped by long use.

The narrow model behaves a little like this. Three examples are enough for it to settle on a whole rule, not to estimate one weight at a time. The examples act as cues, and the answer comes from what training laid down. The break is plain, though. Bhartṛhari's pratibhā is a flash in a conscious listener, and it can grasp a sentence never heard before. The narrow model grasps only what it already holds, and it has no listener at all. The parallel shows the shape of the recognition regime; it says nothing about awareness.

Where this sits in the RTM layers

In the layers of the Recognition Theory of Mind, the hypothesis this series tests, in-context learning sits at L1 and L2 in every regime. The examples change the output, which is responsiveness, and they shape what is available to later computation, which is manifestation. In the retrieval regime there is also the formal shape of L4, recognition: a present input joined with a stored impression. What is missing in every regime is L3, vimarśa, reflective awareness. No model in this paper represents anything like "I do not know this task". The narrow model, faced with a stranger, never registered that it was one.

Figure 16. Recognition of a task, without anyone to recognise it. Recognition Theory of Mind layers applied to in-context learning. The dotted L4 rung marks a structural parallel in the retrieval regime, not a capacity. author hypothesis

That absence gives a clear test for any claim that in-context learning involves more than L2. A model with something like L3 would represent its own fit to the prompt, and use it: switching method when the prompt matches no known task, or reporting low confidence. The test is functional, and passing it would not show awareness in the Śaiva sense. But failing it, as all my models do, shows that the shape of recognition can be present with nothing that reflects on it.

Figure 17. The tasters model the behaviour, not the system. The analogy captures which estimator each model follows. It breaks where it credits the model with knowing what it does not know, or with choosing a method. analogy check

Part 7

Testing a real model with flipped and arbitrary labels

A pretrained language model cannot be retrained on a chosen pool of tasks, but it can be probed with prompts that pull the two regimes apart. The realistic worked case here is GPT-2 small, a 124-million-parameter model from 2019, on the simplest few-shot task I could design. Each prompt lists sentiment words with labels, one per line, such as "happy: positive" and "awful: negative", then gives a new word and asks for its label. I used 20 clearly positive and 20 clearly negative words, and ran 200 prompts per condition.

The test, step by step

Zero-shot baseline. With no examples, just "happy:" and a choice between "positive" and "negative", GPT-2 chose the right label 96 per cent of the time. It already knows which words are pleasant.
Gold labels. With 4, 8 or 16 correct examples, accuracy was 71 to 74 per cent, lower than zero-shot. The examples hurt.
Random labels. With labels assigned at random, accuracy, scored against true sentiment, was 75 to 80 per cent, no worse than gold. This repeats Min and colleagues.
Flipped labels. With every example label reversed, GPT-2 followed the flipped mapping in only 21 to 27 per cent of prompts. In the rest, it gave the label its prior expected.
Arbitrary labels. With "foo" for positive and "bar" for negative, it used the shown mapping in 37 to 41 per cent of prompts, at or below the 50 per cent of chance.

Figure 18. GPT-2 small reads the format, not the answers. Forty sentiment words, 200 prompts per bar. Random labels: score against the true sentiment. Flipped: share that followed the flipped mapping. Foo/bar: share that used the arbitrary labels as shown. Zero-shot with "positive" and "negative" scored 0.96. Dashed line: chance. Run in this paper's instrument. measured, GPT-2 small

Read through the lens of this paper, GPT-2 small is a Meera with a very strong prior. It reads the prompt to find the task and the label words, then answers from what it already knows about the words. It does not learn the mapping from the examples. When the examples contradict its knowledge, it mostly ignores them; when they use labels with no prior meaning, it cannot pick up the mapping. That is what Wei and colleagues reported for small models, and what Pan and colleagues would call task recognition without task learning.

The drop from zero-shot to gold deserves a word, because it looks wrong. My best guess is that the examples introduce biases that have nothing to do with the task: GPT-2 tends to repeat the most recent label, or the more common label in the prompt. Those biases cost more than the examples add, for a model that already knew the answer. I did not test this guess, and it is the kind of detail that would need its own controls.

What this does and does not show

This is one small model on one easy task. It does not show that larger models behave the same way, and the literature says they do not: Wei and colleagues found that overriding priors with flipped labels emerges with scale. What it does show is that the instruments work. Four prompt conditions, each cheap to run, are enough to place a model's in-context behaviour between recognition and learning, and the result for GPT-2 small is unambiguous.

An analogy, and its break

A new employee is given five examples of the firm's house style for client letters, and asked to write the sixth. A junior with no habits of her own will copy the examples. A senior who has written letters for twenty years will glance at the examples, recognise "a client letter", and write it the way she always has. If the examples happen to use a strange convention, the senior will miss it. GPT-2 is the senior. The analogy breaks because the senior could be told "follow the examples exactly" and would. GPT-2 small, without instruction tuning, has no channel for that instruction beyond the examples themselves.

What a good in-context learning claim contains

Claims that a model "learns from its examples" are easy to make and hard to check, so it helps to agree on what a report should include. From the tests in this paper, I would ask for five things. The first is a zero-shot baseline, so readers can see what the model knew before the examples. The second is the gold-label result, so readers can see what the examples add. The third is the random-label and flipped-label results, to separate recognition from learning.

The fourth is an arbitrary-label result, which asks whether the model can pick up a mapping with no prior support at all. The fifth is the number of prompts and the spread of results, because few-shot accuracy can swing widely with the choice and order of examples. My GPT-2 test meets the first four and reports 200 prompts per condition, without the spread. That is enough for the sharp pattern I found, but not for small differences.

What this means for writing prompts

For people who write few-shot prompts for a living, the findings suggest a change of emphasis. If a model mostly recognises tasks, the job of the examples is to name the task clearly, not to teach it. That means choosing examples that are typical, in the expected format, and spread across the kinds of input the model will meet. It matters less that each label is perfect, as Min and colleagues found, and more that the examples point unmistakably at one familiar task.

If the task is genuinely new to the model, the advice reverses. Then the examples are the only source of the mapping, and their labels matter a great deal. More examples help, and so does a larger model, since the evidence suggests task learning grows with scale. A quick way to tell which case you are in is the flipped-label test from this paper. If flipping the labels changes nothing, the model already knows the task, and your examples are a name tag. If flipping them changes the answers, the model is reading them, and they had better be right.

There is also a middle case that deserves care. A model may recognise a task that is close to, but not the same as, the one you want. A support team's own definition of "urgent" may differ from the general sense of the word. The model will then apply the general sense with great confidence, as the narrow model did with its nearest known task. The cure is to test with cases where the two definitions disagree.

Deciding what you are looking at

For a practitioner, the useful output is a short procedure. The tree below gives four tests in order. The first three separate recognition from learning. The fourth names the estimator the model matches, which only makes sense when you know its training prior, as in synthetic settings. In a pretrained model, the fourth test becomes a comparison with zero-shot behaviour, which stands in for the prior.

Figure 19. Four tests before you say a model learned from its examples. Steps 1 to 3 separate recognition from learning; step 4 names the estimator the model approximates. A model that passes 1 but fails 2 and 3 is mostly recognising. decision instrument

Thought experiment 6

The examples nobody read

A colleague is building a classifier for customer complaints on top of a small open language model. Her firm uses its own categories, and one of them, "billing", covers some complaints a general model would file under "delivery". She writes eight careful examples into the prompt to teach the distinction, tests on a few cases, and ships.

Pause. Before you read on: if her model behaves like GPT-2 small in this paper, what have her eight examples taught it?

Mostly the format and the label names. A model in the recognition regime will use the examples to identify "complaint classification" and then file each complaint where its prior says it belongs. The firm-specific distinction, the one thing she wrote the examples to teach, is exactly the kind of mapping such a model ignores. Her spot checks may pass, because most complaints are filed the same way under both schemes. The failures will be concentrated in the cases she cared about.

The uncomfortable lesson is that few-shot prompting can look like teaching while doing little more than naming the task. The defence is cheap: run the flipped-label test on her own examples and see whether the model follows them. If it does not, it is not learning from them, and she needs a larger model, fine-tuning, or a different design. The limit is that larger, instruction-tuned models follow examples much more readily, so the size of this risk depends on the model she chose.

[AUTHOR NOTE: optionally add a sentence on whether you have seen few-shot prompts in client systems that turned out to be ignored by the model]

Objections

The best case that I have this wrong

The strongest objection is that "both are Bayesian" makes the question disappear by definition. Any good predictor approximates the Bayesian answer for its training distribution, so saying a model is Bayesian adds nothing. The interesting question, on this view, was always about mechanism: is there something like a gradient step in the activations? My experiments answer that only for a one-layer linear model and leave the softmax model's mechanism unknown. That is a fair criticism. My reply is that the levels distinction does real work, because it shows that "Bayes or gradient descent?" was never a choice, and that the right questions are "Bayesian for which prior?" and "by which procedure?".

A second objection concerns scale. My models have four layers and inputs of four numbers. Raventós and colleagues worked with larger settings, but nobody has run the task-diversity experiment on anything like a language model, because language does not come in countable tasks. The leap from my sweep to claims about GPT-style models is an analogy, and the GPT-2 test only shows the recognition end of the range.

A third objection is about my gradient-descent baselines. I compared the softmax model with plain gradient descent from zero, with one and ten steps and the best step size. The literature includes preconditioned and multi-step variants, and some of those converge much faster. A cleverer variant might match my model as closely as ridge does. If so, claim C5 would fall, and the gradient-descent story would cover more ground than I allow.

A fourth objection is that my GPT-2 test is too easy to be informative. Sentiment of single words is something GPT-2 already knows, so it is no surprise that it relies on its prior. A task with no prior, where the examples are the only source, might show more learning. The foo/bar condition was meant to be that task, and it showed little. But a more careful design, with arbitrary but learnable mappings and more examples, might find a weak learning signal I missed.

A fifth objection is that I have read too much into one small model's indifference to its labels. Min and colleagues (2022) found that random labels barely hurt several models, which fits my GPT-2 result. But Wei and colleagues (2023) found that large models do follow flipped labels, overriding what they learned in pretraining. So the recognition picture may describe small models only. I accept this, and it is why C8 is written as a claim about scale. The objection does not touch the synthetic results, where the regime depends on training diversity, not size.

A sixth objection is philosophical. Calling a forward pass "Bayesian" may smuggle in a picture of a reasoner weighing beliefs, when all that happens is arithmetic on vectors. I agree that the word can mislead. I use it in the narrow sense of Marr's computational level: the outputs match those a Bayesian predictor would give. Nothing in this paper says the model has beliefs, weighs them, or knows that it is doing so. The same caution applies to "learning". When I say the broad model learns in context, I mean only that its predictions improve with examples on tasks it has never seen, in the way a learning procedure would. Whether anything in it deserves the richer sense of the word is a separate question, and Part 6 keeps it separate.

Tests

What would change my mind

Several results would make me revise this paper. If a five-seed rerun of the diversity sweep put the change far from 8 to 32 tasks, or showed no consistent order between pool sizes, I would treat C3 as a one-seed accident. Suppose probing showed a softmax transformer of the kind I used computing the intermediate quantities of a gradient method, not of a closed-form solve. Then I would treat gradient descent as a better description of its mechanism than I have here.

On real models, I would revise C7 if GPT-2 small followed flipped labels well on other tasks with more examples, say 32 or 64. That would mean my sentiment test was a special case where the prior is unusually strong. I would revise C8 if large pretrained models without instruction tuning were found to ignore flipped and arbitrary labels as GPT-2 does, since the shift with scale would then be an effect of instruction tuning, not of size.

On the sweep itself, I would also revise the trade-off claim if a model were found that was both an expert on seen tasks and a good learner on new ones. My sweep shows no such model: every pool that helped on strangers cost accuracy on regulars. A larger model, or a better training schedule, might escape that trade-off, and if so the Bayesian reading would need an extra ingredient.

Figure 20. What I do not know, placed by how soon it could be settled. Placement is my judgement. author judgement

On the RTM side, I would take the recognition framing more seriously if a model were found that detects when a prompt matches no known task and changes its behaviour as a result. That would be a functional sign of something like L3, and the narrow model's confident failures would then look like a limitation of small models, not of the architecture. Equally, if no such component were ever found in models that clearly learn new tasks, I would conclude that recognition is a description of the retrieval regime only, as Thought Experiment 4 conceded.

Figure 21. How sure I am of each claim, at a glance. Bar length is my subjective confidence; pattern and colour give the ledger category, so the chart reads in greyscale. author judgement

Ledger

Claim Ledger

Each main claim of this paper, with how sure I am, what kind of evidence carries it, and what would knock it down.

ID	Claim	Confidence	Evidence type	What would refute it
C1	A transformer trained on unlimited linear tasks predicts new tasks almost exactly as ridge regression does.	established	Garg et al. (2022); Akyürek et al. (2023); Raventós et al. (2023); this paper: error 0.03 at 15 examples, distance to ridge 0.03	A careful replication in which broad training does not approach ridge
C2	A transformer trained on a few fixed tasks behaves like a Bayesian lookup over those tasks: better than ridge with few examples, poor on new tasks.	established	Raventós et al. (2023); this paper: 0.17 against ridge 1.06 at 3 examples; 2.24 on new tasks at 15	Narrow-pool models that generalise to new tasks
C3	Which behaviour appears depends on training task diversity; in my setting the change from lookup to ridge-like behaviour happens between 8 and 32 tasks, and costs accuracy on seen tasks.	likely	Raventós et al. (2023); this paper's one-seed sweep: new-task error 2.24, 0.73, 0.33, 0.06 for 8, 12, 16, 32 tasks	Multi-seed sweeps showing no consistent change with pool size
C4	One layer of linear self-attention trained on regression implements one gradient step.	established	von Oswald et al. (2023); Ahn et al. (2023); this paper: distance 0.006 at step size 0.5	Trained linear layers that match no gradient step
C5	My four-layer softmax model is not doing one or ten plain gradient steps; it is much closer to ridge.	likely	This paper: 0.055 against 0.40 for ten steps at 8 examples	A gradient variant with few steps that matches the model as closely as ridge
C6	"Gradient descent or Bayes" mixes levels: ridge is the Bayesian answer for a Gaussian prior and can be computed by gradient steps.	likely	Standard Bayesian linear regression; Marr (1982); Panwar et al. (2024)	A case in which a trained model matches gradient descent and no Bayesian estimator for its prior
C7	GPT-2 small does mostly task recognition on a simple sentiment task: random labels do not hurt, flipped labels are ignored, arbitrary labels are not learned.	likely	Min et al. (2022); Wei et al. (2023); this paper: flipped followed 0.21 to 0.27; foo/bar 0.37 to 0.41	GPT-2 following flipped labels well on several tasks
C8	In pretrained models the mix shifts towards task learning with scale.	likely	Wei et al. (2023); Pan et al. (2023)	Large models that ignore flipped and arbitrary labels as GPT-2 does
C9	The retrieval regime has the formal shape of recognition (L4) without L3; the learning regime is closer to inference.	speculative	Conceptual analysis under RTM	A model component that marks a prompt as matching no known task and changes strategy

Dated predictions

ID	Prediction	Judged by	How it will be judged
P18-1	Rerunning this paper's task-diversity sweep with five seeds per pool size will place the switch from lookup to ridge-like behaviour between 8 and 32 training tasks for inputs of 4 numbers, with every pool of 32 or more ending below 0.2 error on new tasks at 15 examples.	This paper's scripts, rerun by me or anyone	Run e18b with seeds 0 to 4; report new-task error at 15 examples
P18-2	By the end of 2028, a published study will show a pretrained language model of under 1 billion parameters, without instruction tuning, following flipped sentiment labels above 70 per cent with 16 examples.	Peer-reviewed or widely cited work	Literature search on 31 December 2028

Words

Glossary

In-context learning: A model using examples in its prompt to perform a task, without changing its weights.
Ridge regression: Least-squares fitting with a small penalty on large weights; the Bayesian answer when weights are drawn from a Gaussian.
Gradient descent: Improving a guess by repeated small steps downhill on an error measure.
Bayesian estimator: The prediction that averages over possible tasks, weighted by how well each explains the examples and how likely it was to begin with.
Prior: The spread of tasks a learner expects before it sees any examples.
Task diversity: How many distinct tasks appear in training.
Linear self-attention: Attention without the softmax; easier to analyse and the setting of the gradient-descent constructions.
Task vector: A single internal vector that summarises the task shown by the examples.
Task recognition and task learning: Using examples to identify a known task, versus picking up a mapping not seen in training (Pan et al.).
pratyabhijñā प्रत्यभिज्ञा: Sanskrit: recognition; knowing a present thing as one known before. RTM layer L4.
upamāna उपमान: Sanskrit: comparison; in Nyāya, the means of knowing what a word names from a description of likeness.
anumāna अनुमान: Sanskrit: inference from a sign to what it signifies.
saṃskāra संस्कार: Sanskrit: a latent impression left by past experience that shapes later cognition.
pratibhā प्रतिभा: Sanskrit: in Bhartṛhari, the flash of understanding in which a sentence's meaning is grasped as a whole.
pramāṇa प्रमाण: Sanskrit: a means of valid knowledge, such as perception or inference.
vyāpti व्याप्ति: Sanskrit: the invariable connection between a sign and what it signifies, on which inference rests.
gavaya गवय: Sanskrit: a wild ox; the stock example for knowledge by comparison.
vimarśa विमर्श: Sanskrit: reflective awareness; RTM layer L3.

Conclusion

Which buyers exist

This paper asked whether in-context learning implements a learning algorithm or retrieves task priors. In small trained transformers, the answer is set by training. A model trained on 8 tasks recognised them, beating ridge regression with three examples, and failed on new tasks. A model trained on endless tasks behaved like ridge regression on any task. Between the two, the behaviour switched as the number of training tasks grew, as Raventós and colleagues had found.

The sweep added a detail I had not expected. The change came early, between 8 and 32 tasks, and it was a trade. Each step towards learning new tasks cost accuracy on old ones. No model in the sweep had the best of both. That is the clearest sign in my data that the two behaviours are two settings of one thing, a prior that can be sharp or broad, and not two separate abilities that a model could simply add together. A model, like a taster, cannot be maximally ready for strangers and maximally quick with regulars at once.

The two behaviours are not two kinds of mind. Both are Bayesian guesses, differing in the prior: a few sharp impressions, or a broad expectation that any rule is possible. Gradient descent is a description at another level, a procedure. It is exactly what a single linear attention layer does, and it is not what my four-layer softmax model does. In GPT-2 small, the examples mostly name the task, and the model answers from its prior, ignoring flipped labels in about three prompts out of four.

Figure 22. The paper on one page. Pin it next to the next claim that a model "learns in context". summary

For anyone using few-shot prompts, the practical message is to test before trusting. Run the prompt on tasks the model cannot know, flip the labels, and try arbitrary ones. If the model ignores them, its examples are naming a task, not teaching one. That is often useful, but it is not learning, and a system that depends on the difference should be checked for it.

Mr. Kurian's remark holds up. Meera and Arjun were both guessing what the buyer wanted, and they only disagreed about which buyers exist. A transformer's training decides which buyers exist for it. What none of my models could do is notice that a buyer was new. In the Pratyabhijñā tradition, recognition is the moment a knower sees the present as the past. In these models, there is the present, there is the past, and a join between them, but no one at the join.

The next paper in this series turns to another sudden change in training, grokking, where a model that has memorised its data abruptly starts to generalise, long after its training loss has stopped improving.

[AUTHOR NOTE: optionally close with a sentence on why the difference between being taught and being reminded matters to you]

Evidence

Sources

Primary research

Brown, T. B. et al. (2020). Language models are few-shot learners. arXiv:2005.14165. Verified.
Xie, S. M., Raghunathan, A., Liang, P. and Ma, T. (2022). An explanation of in-context learning as implicit Bayesian inference. arXiv:2111.02080. Verified; abstract quoted.
Min, S. et al. (2022). Rethinking the role of demonstrations: what makes in-context learning work? arXiv:2202.12837. Verified; abstract quoted.
Garg, S., Tsipras, D., Liang, P. and Valiant, G. (2022). What can transformers learn in-context? A case study of simple function classes. arXiv:2208.01066. Verified.
Akyürek, E., Schuurmans, D., Andreas, J., Ma, T. and Zhou, D. (2023). What learning algorithm is in-context learning? Investigations with linear models. arXiv:2211.15661. Verified; abstract quoted.
von Oswald, J. et al. (2023). Transformers learn in-context by gradient descent. arXiv:2212.07677. Verified; abstract quoted.
Ahn, K., Cheng, X., Daneshmand, H. and Sra, S. (2023). Transformers learn to implement preconditioned gradient descent for in-context learning. arXiv:2306.00297. Verified.
Raventós, A., Paul, M., Chen, F. and Ganguli, S. (2023). Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression. arXiv:2306.15063. Verified; abstract quoted.
Wei, J. et al. (2023). Larger language models do in-context learning differently. arXiv:2303.03846. Verified; abstract quoted.
Pan, J., Gao, T., Chen, H. and Chen, D. (2023). What in-context learning "learns" in-context: disentangling task recognition and task learning. arXiv:2305.09731. Verified; abstract quoted.
Shen, L., Mishra, A. and Khashabi, D. (2023). Do pretrained transformers learn in-context by gradient descent? arXiv:2310.08540. Verified; abstract quoted.
Hendel, R., Geva, M. and Globerson, A. (2023). In-context learning creates task vectors. arXiv:2310.15916. Verified; abstract quoted.
Todd, E. et al. (2024). Function vectors in large language models. arXiv:2310.15213. Verified.
Panwar, M., Ahuja, K. and Goyal, N. (2024). In-context learning through the Bayesian prism. arXiv:2306.04891. Verified.
Radford, A. et al. (2019). Language models are unsupervised multitask learners. OpenAI technical report (GPT-2). Model used in this paper's test, weights via Hugging Face.

Primary texts and background

Marr, D. (1982). Vision: a computational investigation into the human representation and processing of visual information. W. H. Freeman. The three levels of analysis.
Gautama, Nyāyasūtra 1.1.6, on upamāna (comparison) as a means of knowledge, with the stock example of the gavaya. Standard editions; paraphrased rather than quoted.
Utpaladeva, Īśvarapratyabhijñākārikā, with Abhinavagupta's Vimarśinī; the doctrine of recognition. Standard editions; paraphrased.

Reference

Cite this work

Mahapatra, R. R. (2026). Learning or recognising? What a transformer does with the examples in its prompt. Mind Frontier paper U0378. rajeshrmahapatra.com. https://rajeshrmahapatra.com/papers/mind-frontier/U0378-in-context-learning-algorithm-or-recognition.html. DOI: to be assigned.

@misc{mahapatra2026in,
  author = {Mahapatra, Rajesh Ranjan},
  title = {Learning or recognising? What a transformer does with the examples in its prompt},
  year = {2026},
  howpublished = {Mind Frontier paper U0378, rajeshrmahapatra.com},
  url = {https://rajeshrmahapatra.com/papers/mind-frontier/U0378-in-context-learning-algorithm-or-recognition.html},
  doi = {10.xxxx/placeholder}
}