Invented scenario Everything in this story is invented. It is here to give intuition, never evidence.

It is a wet Saturday in Basavanagudi, in south Bengaluru. Meenakshi Rao translates Kannada novels into English. She works at a long table by a window, with the Kannada book open on the left and her notebook on the right. Her head moves left and right as she writes, a few words at a time.

Farhan Sheikh sits across the table with a laptop. He builds machine translation for a start-up, and Meenakshi has agreed to test his oldest model. It reads a whole Kannada sentence, word by word, into a single list of numbers. Then a second network writes the English from that list alone. It never looks back at the Kannada.

On short sentences the model is fine. On long ones it falls apart. Kannada puts the verb at the end, so a sentence can run for thirty words before you learn what anyone did. Farhan's model gets the first few English words right, then drifts. By the end it is writing a plausible sentence about something else.

"It has to remember the whole sentence before it starts," Meenakshi says. "I never do that. I keep the page open. When I need the verb, I go and find it."

Her uncle, Venkatesh Shastri, has come for lunch. He teaches Nyāya and Mīmāṃsā at a small pāṭhaśālā in Mysuru. "In our śāstra," he says, "a word does not mean alone. It waits for the word it needs. The listener joins the words that need each other, that fit each other, and that come close enough together. Only then is there a sentence."

Three people and three ideas of how words find each other. Farhan's model squeezes everything into one fixed store and hopes. Meenakshi keeps everything and looks back for what she needs, when she needs it. Shastri describes a tradition that asked which words must be joined, and why, many centuries before either of them.

The question for this paper is exact. What problem does attention solve that recurrence could not, derived without folklore?

Here is the answer in plain words, before the argument. A recurrent network carries the past forward in a state of fixed size. Whatever the size, that state must decide what to keep before it knows what will be asked. Attention removes the decision. It keeps every past item and chooses among them at the moment of the question, by comparing what is wanted with what each item offers. The comparison is soft, so the network can learn it with gradients.

I tested this with a small instrument. A recurrent state of 64 numbers lost half its recall of a word after eight or nine steps. Asked to copy back a sequence of sixty words, it got about one in five right. A soft lookup over stored items recalled every one of 512 stored pairs. A fixed memory of 4,096 numbers, the kind that linear attention and fast-weight networks use, was perfect up to 64 pairs and fell to 32 per cent at 512.

The price is just as plain. Attention's memory grows with every item it keeps, and comparing every item with every other costs the square of the length. So the honest summary is a trade, not a victory. Attention buys exact access to the past by paying for storage and comparison. Recurrence saves the cost and accepts a bottleneck. Many arguments about new architectures are arguments about where on that trade to stand.

"Without folklore" matters as much as "derived". The word attention suggests a spotlight of the mind, an act of choosing what to notice. The mechanism has nothing of that. It is a weighted average with weights computed from similarity. It does not explain a network's decisions, as later sections show. It is not all you need, either; the networks that use it rely on much else.

[AUTHOR NOTE: if you have translated between Kannada, Hindi, Odia or another verb-final language and English, two or three sentences on how often you look back at the source while writing, and for what, would ground this scene.]

Start with the problem attention was invented to fix. A recurrent network reads a sequence one item at a time. At each step it combines the new item with its current state and produces a new state. The state is a list of numbers of fixed length, set when the network is designed. Everything the network knows about the past must live there.

The state is a suitcase#

Call the state size d. After reading a word, the network has d numbers. After reading a thousand words, it still has d numbers. If later steps might need any of those thousand words, the d numbers must somehow encode all of them. That is possible only if most of what was read can be thrown away, or squeezed hard.

Squeezing is not always bad. Paper 12 in this series argued that compression is close to what understanding means. A good summary keeps what matters and drops the rest. The difficulty is the word "matters". A recurrent state must decide what matters before it has seen the question. A translator cannot know which noun the verb will need until the verb arrives. If the noun was squeezed out, it is gone.

There are two ways this goes wrong. The state can forget items in proportion to their age, so that early items fade. Or it can keep many items, each blurred by the others, so that no single item can be read back cleanly. A design that fights one problem tends to worsen the other. The instrument shows both.

A first worked case: how fast a state forgets#

I built the simplest recurrent memory I could. The state has 64 numbers. Each step multiplies it by a fixed rotation, shrinks it by a factor ρ (rho) and adds a code for the new word. There are 16 possible words, so guessing gives 6.25 per cent. After 80 words, I trained the best possible linear readout to recover the word from k steps back, for each k from 1 to 60.

This is a fair test of the state, not of a training procedure. The readout was fitted by least squares on 2,000 sequences and scored on 1,000 others. If the information is in the state in any linearly readable form, the readout finds it. The rotation keeps the state's directions independent, which is the best case for a linear memory.

With ρ = 0.8, the last three words were recalled perfectly. Accuracy halved by lag 8 and reached 7.1 per cent, near chance, by lag 20. With ρ = 0.9, the memory lasted a step longer: 99.3 per cent at lag 1, half by lag 9, 10.5 per cent at lag 20.

The surprise was ρ = 0.97, the slowest decay. It kept more of the distant past: 24.8 per cent at lag 20, against 10.5 per cent for ρ = 0.9. But it recalled the most recent word only 60.1 per cent of the time. With slow decay, dozens of older words are still loud in the state, and they blur the new one. A fixed state trades the recent against the distant; it cannot have both.

Copying a whole sentence back#

Translation needs more than one word from the past. It needs, in some order, all of them. So I asked the same state for every word in sequences of length 5 to 60, and averaged the accuracy across positions. This is a crude stand-in for what Farhan's model must do: hold a whole sentence in the state and read it back out.

At five words both decay settings were perfect. At ten, they dropped to 84.5 and 89.6 per cent. At twenty, to about half. At sixty, to 21.0 and 23.3 per cent. A lookup that stores each word and its position separately reads every word back by construction; the capacity test in Part 3 confirms that such a lookup stays exact far beyond this length.

The pattern is the one Farhan sees. Short sentences are fine, long ones fail, and the failure is gradual. Nothing in the network breaks. The state simply runs out of room.

A trained network does better than my fixed rotation, because training can choose what to keep. Gated recurrent networks such as the LSTM learn when to write and when to hold. That raises the useful length a great deal. It does not change the arithmetic. A state of d numbers, read with finite precision, holds a bounded amount of information. Past that bound, something must be dropped.

An analogy: packing for a trip you cannot see#

A fixed state is like a suitcase packed before you know where you are going. You can pack light and well if you know the destination. If you do not, you pack a little of everything and hope. The longer and stranger the trip, the more likely you are to lack the thing you need.

Attention is like travelling with your whole wardrobe in a lorry that follows you. Whenever you need something, you open the lorry and choose. Nothing is ever missing. But the lorry gets bigger every day, and finding the right shirt takes longer as the pile grows.

A true story: the sentence that got too long#

In September 2014, Kyunghyun Cho and colleagues in Yoshua Bengio's laboratory in Montreal published a careful study of the new neural translation systems (Cho et al. 2014). These systems used exactly Farhan's design: an encoder network read the source sentence into a fixed-length vector, and a decoder wrote the translation from it. Cho's team asked a simple question. How does the quality depend on the length of the sentence?

The answer was blunt. The systems "performed relatively well on short sentences without unknown words", the abstract reported, but performance "degrades rapidly as the length of the sentence and the number of unknown words increase" (Cho et al. 2014). The fixed vector was the obvious suspect.

A co-author on that study was Dzmitry Bahdanau, a 23-year-old intern from Belarus who had not done research before (Wolchover 2026). He was, he later wrote, "super skeptical about the idea of cramming a sequence of words in a vector" (Bahdanau, email to Karpathy). His first attempt at a fix used two cursors, one moving through the source and one through the target, with every possible path between them summed by dynamic programming. It proved too hard to build in the five weeks left of his internship. A simpler version, with the two cursors moving in lockstep, worked a little but was, in his word, inelegant.

Then he thought of something else. He wanted the decoder to "learn to search where to put the cursor in the source sequence". The idea, he wrote, "was sort of inspired by translation exercises that learning English in my middle school involved. Your gaze shifts back and forth between source and target sequence as you translate" (Bahdanau, email to Karpathy). He wrote the soft search as a softmax, a weighted average over the encoder's states for each source word. It worked on the first try.

He called the architecture RNNSearch. The team rushed the paper onto arXiv because, Bahdanau recalled, they knew Ilya Sutskever and colleagues at Google were ahead with a much larger model (Bahdanau, email to Karpathy). The abstract states the diagnosis directly: "we conjecture that the use of a fixed-length vector is a bottleneck" (Bahdanau, Cho and Bengio 2014). Its fix lets the model "(soft-)search for parts of a source sentence that are relevant to predicting a target word". The word attention, Bahdanau says, was added to the conclusion by Bengio in one of the final passes (Bahdanau, email to Karpathy).

Meenakshi would recognise the method at once. The machine keeps the page open. Attention began as a fix for one measured failure: quality falling with sentence length because a fixed vector could not hold the sentence. It did not begin as a theory of mind, and its inventor's account of it is a description of a translator's habit.

What Part 1 establishes#

A recurrent state of fixed size holds a bounded amount. It must choose what to keep before the question arrives. It trades recent against distant memory. And whole-sequence recall falls as the sequence grows, which is the failure Cho's team measured in real translation systems. None of this says recurrence is useless. It says what recurrence must do, and so what a fix must change.

Now derive the fix, using only the problem as a guide. The aim is to see that each part of the attention formula answers a specific need. Nothing is added for decoration, and nothing needs a story about minds.

Step 1: keep everything#

The bottleneck came from forcing the past through d numbers. So stop doing that. Keep a separate record for every item: for a sentence of n words, keep n vectors. The memory now grows with the input. That is the whole price, and it is paid at the start.

The records need not be raw words. In Bahdanau's design, each record was the state of a bidirectional recurrent network at that position, so it already carried some context from both sides (Bahdanau, Cho and Bengio 2014). In a Transformer, each record is the output of the previous layer at that position (Vaswani et al. 2017). Either way, there are n records, one per position.

Step 2: choose by content, at the moment of need#

With n records, the decoder needs a way to pick the right one when it is about to write a word. It cannot pick by position alone, because Kannada and English order words differently. The verb that English needs second may sit last in the Kannada. So it must pick by content: by what the record says, compared with what the decoder is looking for.

That requires two vectors. The decoder forms a query, a description of what it wants next. Each record offers a key, a description of what it contains. A natural measure of match is the dot product of query and key: large when they point the same way, small or negative otherwise. Paper 11 in this series explained why dot products between random high-dimensional vectors are close to zero. That is what makes a match stand out.

Step 3: make the choice soft#

Picking the single best match is an argmax: find the largest score and take that record. The argmax has a fatal property for learning. Its output does not change at all when the scores change a little, until the winner changes, and then it jumps. Its gradient is zero almost everywhere. A network that chooses this way cannot learn how to choose.

The fix is the softmax. Raise e to the power of each score, then divide by the total. The result is a set of positive weights that sum to one. The largest score gets the largest weight, but every record gets some. The output is the weighted average of the records. The softmax is a differentiable stand-in for the argmax, and that is the only reason it is there.

Multiplying the scores by a number β (beta) before the softmax sets how sharp the choice is. With β = 0 every record gets the same weight, 1/n. As β grows, the weights pile onto the best match, and the softmax approaches the argmax. The network can learn β, or learn the size of the query and key vectors, which has the same effect.

Step 4: separate what you match on from what you take#

In Step 2 the same vector was used to decide whether a record matches and to supply what the record contributes. These are different jobs. A word's grammatical role might decide whether it matches, while its meaning is what the decoder needs. So give each record two vectors: a key for matching and a value for the content. The output is the weighted average of the values, with weights set by query–key matches.

End-to-end memory networks used this separation in 2015, giving each stored sentence one embedding for matching and another for output (Sukhbaatar et al. 2015). In a Transformer, queries, keys and values are all computed from the same inputs by three learned matrices (Vaswani et al. 2017). The separation costs little and lets the network decide, for each layer, what counts as relevant and what gets passed on.

Step 5: scale the scores by the square root of d#

The last part of the formula is the one most often left unexplained. The Transformer divides each dot product by √d, where d is the length of the key vectors (Vaswani et al. 2017). The reason is arithmetic. If the entries of a query and a key are independent with mean zero and variance one, their dot product has variance d. As d grows, raw scores spread wider, the softmax becomes extremely sharp, and almost all the weight lands on one record.

That would be fine if sharpness were the goal. It is not, at the start of training. A saturated softmax has tiny gradients: changing the scores slightly changes the weights hardly at all. Vaswani and colleagues gave exactly this reason. They suspected that for large d "the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients" (Vaswani et al. 2017). Dividing by √d brings the variance back to one, whatever d is.

The instrument checks this. With 64 random keys and random query, and no scaling, the largest weight rose from 0.28 at d = 4 to 0.95 at d = 1,024. The entropy of the weights fell from 2.77 to 0.14 nats, against a maximum of 4.16 for 64 equal weights. A simple proxy for the size of the gradient, the sum of p(1 − p) over the weights, fell from 0.85 to 0.08. With scaling, the largest weight stayed near 0.11, entropy near 3.68, and the gradient proxy near 0.96, at every d.

So the formula is complete, and each part has a job. Store n records (Step 1). Compare a query with each key by dot product (Step 2). Divide by √d so the comparison stays trainable as vectors grow (Step 5). Turn scores into weights with a softmax so the choice can be learned (Step 3). Take the weighted average of the values (Step 4). Written out, the output is softmax(QKᵀ/√d)V.

A second worked case: one English word from three Kannada ones#

Take the Kannada sentence "nānu pustakavannu ōdide", word for word "I book-the read", meaning "I read the book". Suppose the decoder has written "I" and is about to write the English verb. It needs "ōdide". Give the three source words two-number keys: nānu (2, 0), pustakavannu (0, 2) and ōdide (1, −2). The decoder's query is (0.5, −1.5).

The raw scores are the dot products: 1 for nānu, −3 for pustakavannu and 3.5 for ōdide. Divide by √2, since the keys have two entries: 0.71, −2.12 and 2.47. The softmax gives weights of 0.145, 0.009 and 0.847. If each value is simply a marker for its word, the output is 85 per cent "ōdide", 14 per cent "nānu" and under 1 per cent "pustakavannu".

Two things are worth seeing. The decoder found the verb by content, though the verb sits last and English needs it second. And the output is a blend, not a choice. Some of "nānu" leaks in. Without the √2 scaling, the weights would have been 0.076, 0.001 and 0.923, sharper but with smaller gradients. A trained network sets its query and key sizes so that the blend is as sharp as the task needs.

How sharp should the choice be?#

The blend matters when records are similar. I stored 64 random key–value pairs in 64 dimensions and queried each with its key plus noise. The noise moved the query about 0.12 of the way off its key in each dimension, so the query no longer matched exactly. Then I varied β from 0 to 64 and measured two things: whether the output was closest to the right value, and how far it was from that value.

At β = 0, the uniform-attention control, the output was the average of all values. It picked the right one 1.3 per cent of the time, close to the chance rate of 1.6 per cent. At β = 2 it was right 70 per cent of the time. At β = 4 and above, it was always closest to the right value. But being closest is not the same as being clean. At β = 4 the output was still 0.81 away from the right value, on a scale where the values have length one. Only at β = 16 did the error fall below 0.01.

Soft attention can point at the right record long before it can deliver that record cleanly. Between β = 4 and β = 16, the choice is right but the content is mixed. For a downstream layer that needs the exact value, such as copying a name, mixing is a failure even when the choice is correct.

A true story: turning the sentence backwards#

The other team Bahdanau mentioned was at Google. In September 2014, Ilya Sutskever, Oriol Vinyals and Quoc Le reported a large encoder–decoder system for English to French, built from deep LSTM networks (Sutskever, Vinyals and Le 2014). It had no attention. It still scored 34.8 BLEU on the test set, beating a phrase-based statistical system at 33.3. And, the abstract said, "the LSTM did not have difficulty on long sentences" (Sutskever, Vinyals and Le 2014).

That sentence is a useful check on any neat story. A bigger, well-trained recurrent network with a fixed state did handle long sentences acceptably. The bottleneck was real, but it was not a wall. The authors also found a trick. Reversing the order of words in every source sentence, but not the target, "improved the LSTM's performance markedly" (Sutskever, Vinyals and Le 2014). With the source reversed, the first source words sit closest to the first target words. That "introduced many short term dependencies", which made learning easier.

The trick shows what the fixed state was struggling with. It was not the number of words alone but the distance between a source word and the target word that needs it. Reversal shortens the distance for the start of the sentence and lengthens it for the end. Attention makes every distance one step, because any target word can reach any source word directly. The reversal trick is a hand-made fix for one sentence shape. Attention is a learned fix for all of them.

An analogy: the address book and the photograph#

A recurrent state is like a photograph of a crowded room. Everyone is in it, but the people at the back are small and blurred, and some are hidden behind others. To find one person, you study the photograph. If they are hidden, they are gone.

Attention is like an address book. Each person has an entry, and you look them up by name. Nobody is hidden. But the book is as long as the list of people, and looking someone up means checking entries.

Self-attention and the cost of the square#

Bahdanau's attention linked a decoder to an encoder. The Transformer turned the same operation inward (Vaswani et al. 2017). In self-attention, every position in a sequence forms a query and compares it with the keys of every other position. Each position's output is a blend of all the others' values. Stack such layers, with small feed-forward networks between them, and you have the Transformer.

Self-attention removes recurrence entirely. No state is carried from step to step. Information moves between any two positions in a single layer. And every position can be computed at once, which suits the parallel hardware on which networks are trained. Vaswani and colleagues stressed this: their model was "more parallelizable" and required "significantly less time to train" (Vaswani et al. 2017).

The cost is the square. For n positions, self-attention computes n × n scores in every layer. Doubling the length quadruples the work. When generating text one word at a time, the stored keys and values grow by one record per word, so memory grows with length and each new word must be compared with all earlier ones. These are the costs of keeping the page open.

The argument so far says that attention avoids the bottleneck by storing more. That invites an obvious objection. Of course a store that grows with the input beats one that does not. The fair question is what each design gets per number of memory, and where each one stops.

The task#

I used associative recall, the simplest task that needs content-based lookup. A memory is given n pairs of random vectors: a key and a value, each with 64 entries and length one. Then it is given one of the keys and must return the matching value. The answer counts as right if the returned vector is closer to the correct value than to any other stored value. Chance is one in n.

Neural Turing Machines were tested on exactly this kind of task, among others, in 2014 (Graves, Wayne and Danihelka 2014). It is a fair test of attention's central claim. If attention is a content-addressed lookup, it should do well here. If a fixed state can match it, the claim is weaker than it looks.

Three memories#

A softmax cache. Store all n keys and values. To recall, score the query against every key, sharpen by β = 16, apply the softmax and average the values. This is attention, with no learned parts. It uses 2 × n × 64 numbers: 512 at n = 4, and 65,536 at n = 512.

An outer-product memory. Add up the outer products of each value with its key, giving one 64 × 64 grid of 4,096 numbers, whatever n is. To recall, multiply the grid by the query. This is the state that linear attention keeps when it is run as a recurrent network (Katharopoulos et al. 2020). It is also the fast-weight memory of the early 1990s, which Schlag, Irie and Schmidhuber showed to be formally the same thing (Schlag, Irie and Schmidhuber 2021).

A holographic vector. Bind each key to its value by circular convolution and add the results, giving one vector of 64 numbers. To recall, unbind with the query. This is Plate's holographic reduced representation, a classic way to pack structure into a fixed vector (Plate 1995). It is the closest thing here to a small recurrent state.

Each design was tested on 20 random memories at each size, with 32 queries per memory, for n from 4 to 512.

The actual output#

The softmax cache recalled every query correctly at every size, up to 512 pairs. The outer-product grid was perfect up to n = 64, then fell: 98.3 per cent at 128 pairs, 75.9 at 256 and 31.6 at 512. The holographic vector was below perfect from the start: 98.4 per cent at 4 pairs, 53.6 at 16, 8.4 at 64 and 0.9 at 512, close to chance.

The numbers per unit of memory tell the story. At n = 32, the softmax cache and the outer-product grid used the same 4,096 numbers, and both were perfect. At n = 64 the grid was still perfect with half the memory of the cache. Past that point the cache kept growing and stayed perfect, while the grid stayed the same size and failed. A fixed memory of d × d numbers recalls cleanly up to about d pairs; beyond that, crosstalk between stored pairs swamps the signal.

Paper 11 in this series derived why. Random vectors in 64 dimensions are nearly orthogonal, but not exactly. Each stored pair adds a small amount of noise to every recall, of size about one over √d. With n pairs, the noise adds up to about √(n/d) against a signal of one. The signal stays ahead while n is near d, and loses once n is several times d. The measured curve has that shape.

Schlag and colleagues reached the same conclusion analytically for linear attention: purely additive outer products have a capacity limit, and the memory must be edited, as well as added to, to go further (Schlag, Irie and Schmidhuber 2021). Their fix replaces the plain sum with a delta-rule update that subtracts the old value before writing a new one.

The negative controls#

A good instrument shows that its success is not automatic. I ran two controls.

Shuffled query. The softmax cache was given a random query, unrelated to any key. Its accuracy was 24.7 per cent at n = 4 and 0.2 per cent at n = 512, matching chance at every size (25 per cent and 0.2 per cent). So the cache's success comes from the match between query and key, not from some quirk of the scoring.

Uniform attention. In the sharpness sweep of Part 2, β = 0 gave equal weight to every record. It picked the right value 1.3 per cent of the time against a chance rate of 1.6. So the softmax over scores, not the averaging alone, does the work.

A failure boundary#

Put the two results together and a boundary appears. On one side, the number of things that must be recalled is small compared with the state. A fixed state suffices, and is cheaper. On the other side, the number is large. A fixed state fails, and a growing store is needed. Between them is a band where a fixed state degrades gradually.

The boundary depends on what the task needs. If a task needs only a summary of the past, such as the average sentiment of a review, a small state is enough however long the input is. If a task needs exact recall of many items, such as copying a list of names, no fixed state is enough once the list is long. Attention earns its cost on tasks that need exact access to many items chosen late, and only on those.

Where recurrence fought back#

None of this means recurrence lost. Researchers looked for ways to get attention's benefits without its quadratic cost, and several came back to fixed states.

Katharopoulos and colleagues showed in 2020 that if the softmax is replaced by a product of feature maps, attention can be computed in time linear in the sequence length (Katharopoulos et al. 2020). Run step by step, this linear attention is a recurrent network whose state is the outer-product grid of the instrument. It is up to 4,000 times faster on long autoregressive sequences, they reported, with similar performance on their tasks. The instrument shows what it gives up: exact recall past about d items.

Gu and Dao's Mamba, in 2023, took a different route (Gu and Dao 2023). Its designers identified a "key weakness" of earlier subquadratic models: an "inability to perform content-based reasoning". Their fix was to make the recurrent update depend on the input, so the state can choose what to write and what to forget. This is a learned version of the gate in an LSTM, applied to a structured state. On many language tasks Mamba matched Transformers of similar size. A later paper by the same authors described a formal correspondence between such state-space models and variants of attention (Dao and Gu 2024).

Then came a sharp test. Jelassi and colleagues studied copying, the task where the instrument's fixed states failed (Jelassi et al. 2024). They proved that a two-layer Transformer can copy strings of length exponential in its size, while models with a fixed-size state are limited by that state, which bounds what they can hold. Empirically, pretrained Transformers "dramatically outperform" state-space models at copying and retrieving information from context.

So the picture is consistent across my small instrument and the published work. A selective fixed state can do most of what attention does when the task needs a summary, and falls behind when the task needs exact retrieval of much context. Hybrid designs, with mostly recurrent layers and a few attention layers, follow directly from that trade.

What the instrument does not show#

The instrument has no training. Keys and values were random, and the lookup was fixed. Trained networks learn their keys, and can arrange them to reduce crosstalk, so the outer-product grid's capacity in a trained model may be higher than the random case suggests. The bound of roughly d pairs per d × d grid is a scaling, not a fixed number.

It also tests one query at a time. Real tasks chain lookups: find a name, then find what that name did, then find where. The instrument says nothing about how errors in one lookup compound in the next. And it uses one sharpness, β = 16, for the cache. A trained Transformer must learn a sharpness for each head, and may not reach it.

Attention is a lookup. The previous parts showed what that buys. This part shows three things it does not buy, each of which is often claimed.

Access is not use#

Keeping every item available does not mean a model uses every item well. Liu and colleagues tested this directly in 2023 (Liu et al. 2023). They gave language models a long context with one relevant passage placed at different positions, and asked questions that needed that passage. Performance was "often highest when relevant information occurs at the beginning or end of the input context". It "significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models".

That is a striking result for a mechanism whose whole point is equal access. Attention's arithmetic treats every position alike. The trained network does not. Position encodings, the order of training data and the way earlier layers shape the keys all leave a bias. An architecture that can reach every position does not guarantee a trained model that does.

A counterexample: weight is not influence#

The second claim is that attention weights show what a model relied on. The heatmaps of attention weights in translation papers look like explanations. Bahdanau's own paper showed alignments that, its abstract said, "agree well with our intuition" (Bahdanau, Cho and Bengio 2014). It is tempting to read a large weight as "this word mattered".

I tested the simplest version of that reading. I built random attention layers with 16 records each, computed the weights, and then measured each record's real influence: how much the output moved when that record was removed. If weights are explanations, the record with the largest weight should be the one whose removal changes the output most.

When every value vector had the same length, that held almost always. The rank correlation between weight and influence was 0.997, and the top-weighted record was not the most influential in only 2 per cent of cases. Then I let the lengths of the value vectors vary, as they do in trained networks. With a spread of 0.5 on a log scale, the top-weighted record was not the most influential in 10.5 per cent of cases. With a spread of 1.5, in 19.5 per cent. The rank correlation fell only to 0.94, so the weights still looked like a good guide, while the top-ranked record was wrong one time in five.

The reason is simple once seen. Influence depends on the weight and on how different the record's value is from the rest. A record with a modest weight and a large, unusual value can move the output more than a record with a large weight and a value close to the average. The weight is half of the story.

Jain and Wallace reached a stronger conclusion with trained models in 2019 (Jain and Wallace 2019). Learned attention weights were "frequently uncorrelated with gradient-based measures of feature importance", and very different attention patterns could yield the same predictions. Wiegreffe and Pinter replied the same year (Wiegreffe and Pinter 2019). They argued that whether attention explains depends on what explanation means, and proposed tests, including a uniform-weights baseline like my β = 0 control. The exchange settled less than either side wanted. It did settle that a weight, read alone, is not an explanation.

An analogy: the spotlight#

The commonest picture of attention is a spotlight. The network shines it on part of its input and ignores the rest. The picture is useful for one thing: it conveys that attention selects.

What attention does enable: the induction head#

If not explanation, what? One clean answer comes from work on small Transformers. Olsson and colleagues described "induction heads", pairs of attention heads that implement a simple rule: if the sequence earlier contained A followed by B, and A appears again, predict B (Olsson et al. 2022). One head finds the previous occurrence of the current token. A second copies what came after it.

This is exactly the operation a fixed state struggles with. To predict B after the second A, the model must find a specific earlier position by content, then read the next item exactly. The copying results of Jelassi and colleagues and my instrument both say that fixed states are weak at this when the context is long. Olsson's team found that induction heads form at the same point in training as a sharp rise in the model's ability to use its context, and gave causal evidence for this in small attention-only models.

So the first-principles derivation and the mechanistic finding meet. Attention's distinctive power is content-addressed copying from far back, and the clearest circuits found inside trained Transformers use it for exactly that.

The derivation never needed the word attention. It needed a lookup that is soft, searched by content and computed at query time. The name came late, from Bengio, in the last edits (Bahdanau, email to Karpathy). It is worth asking what the name brings with it.

Ideas in the air#

Bahdanau was clear that his idea was not the only one of its kind. Alex Graves had used a soft window over a text sequence to generate handwriting in 2013, letting a network decide which letter it was writing at each moment (Graves 2013). Graves's Neural Turing Machine of 2014 read and wrote an external memory "by attentional processes", including content-based addressing (Graves, Wayne and Danihelka 2014). Weston's memory networks stored facts and retrieved them for question answering (Weston, Chopra and Bordes 2014).

Bahdanau wrote that his work in Montreal was independent of these, and that "it was indeed exactly the same idea" when he saw Graves's translation work six weeks later. He added that attention "is just the natural way to do flexible spatial connectivity in deep learning", "a nearly obvious idea that was waiting for GPUs to be fast enough" (Bahdanau, email to Karpathy). That is the inventor's own case against folklore. The mechanism is what you build when the problem is a bottleneck and the tool is gradient descent.

The older relative: content-addressable memory#

There is an older line still. A Hopfield network, introduced in 1982, stores patterns and retrieves the nearest one from a partial cue. That is content-addressable memory (Hopfield 1982). In 2020 Ramsauer and colleagues showed that a modern Hopfield network with continuous states has an update rule "equivalent to the attention mechanism used in transformers" (Ramsauer et al. 2020). It can store exponentially many patterns in the dimension of its space and retrieve one in a single step.

That result gives the derivation a second foundation. Attention is one step of retrieval in an associative memory whose energy has a minimum near each stored pattern. The β of Part 2 sets whether the minima are separate, so a single pattern is retrieved, or merged, so the output is an average. Ramsauer's team named these regimes: global averages, metastable averages over subsets, and single stored patterns. The sharpness sweep showed all three: β = 0 gave the global average, β of 4 to 8 a blend dominated by the right record, and β of 16 or more a single pattern.

Human attention is a different thing#

In psychology, attention is the selection of some information for further processing at the expense of other information, under limited capacity. The limited capacity is the point. People attend because they cannot process everything.

The mechanism in a Transformer has no limited capacity in that sense. Every head computes a weight for every position, every time. It processes everything and then averages. If anything, it is the recurrent network that resembles limited human capacity, with its fixed state that must choose what to keep. The analogy with human attention runs backwards at the level of capacity, though it holds at the level of a translator's gaze, which was Bahdanau's inspiration.

What survives the folklore check#

Strip away the name and three things remain. Attention is a soft, content-addressed lookup into a store that grows with the input. It is trainable because the softmax is smooth, and stays trainable at large widths because of the √d scaling. And it is one step of retrieval in an associative memory. Each of these can be tested, and Parts 2 to 4 tested them. None needs the claim that the network is attending in any sense a psychologist would recognise.

Venkatesh Shastri's remark at lunch came from a long Indian debate about how separate words become one sentence. The debate belongs to Mīmāṃsā, the school of interpreting Vedic injunctions, and was taken up by Nyāya and by the literary theorists. Its central question is close to the one attention answers. Given a string of words, which words should be joined to which, and on what grounds?

The problem the tradition faced#

A heap of words is not a sentence. If someone says four nouns in a row, nothing is understood, though each word has a meaning. Elisa Freschi puts the worry plainly: without criteria, "out of a random heap of words one would not know how to start to get to the sentence meaning" (Freschi 2018). Something must say which words connect.

The Mīmāṃsā answer is three conditions. In Malcolm Keating's summary, they are "expectancy of connections (ākāṅkṣā), semantic fit (yogyatā), and cognitive contiguity (āsatti or sannidhi)" (Keating, SEP). Words become a sentence when they need each other, fit each other and are close enough together.

Ākāṅkṣā, expectancy. A word's meaning leaves something open. A verb such as "brings" expects someone who brings and something brought. Freschi notes that the examples of expectancy "always refer to syntax rather than semantics and typically have a verb expecting a complement or vice versa" (Freschi 2018). In Ollett's rendering of the Prābhākara definition, it is when "a listener stands in need of knowing the correlate of a given word-meaning" (Ollett 2020).

Yogyatā, fitness. The joined meanings must make sense together. The stock counterexample is watering or sprinkling something with fire (Freschi 2018; Keating, SEP). The grammar is complete, and the meanings cannot combine.

Sannidhi or āsatti, proximity. The words must come close enough. In the simplest reading this is closeness in time: "a cow" and "runs", said on two different days, do not form a sentence (Freschi 2018). But Keating reports that the Prābhākaras understood it as cognitive proximity, and that the related words "may not be words which uttered in sequence, as a connected subject and verb may be separated by a dependent clause" (Keating, SEP). Ollett gives the definition as "presence in the listener's mind of a word-meaning when another word-meaning is present" (Ollett 2020).

That last move is the one that matters here. Proximity measured by position fails for exactly the sentences that broke Farhan's model: a subject at the start, a verb at the end, and thirty words between. The Prābhākara authors replaced closeness in the string with closeness in the hearer's mind, which is the move attention makes when it matches by content instead of position.

Two theories of how words combine#

The tradition split on what a word gives before the sentence is understood. On the Bhāṭṭa view, called abhihitānvaya, "the relation of [word-meanings] that have been expressed", words first express their meanings, and then those meanings are synthesised into a sentence meaning (Ollett 2020). Its defenders noted that "Bring the cow", "See the cow" and "Bring the sheep" must share something, the invariable meaning of each word (Keating, SEP).

The Prābhākara view is called anvitābhidhāna, "the expression of relational [word-meanings]" (Ollett 2020). On it, a word expresses a meaning "already related" to the meanings of the other words in the sentence. The Prābhākara author Śālikanātha argued that what is remembered on hearing an isolated word is not yet a meaning; meaning "has to do with the role that a word's cognition plays within a sentence" (Keating, SEP). Ollett quotes the Prābhākara teacher's blunter version: "in the practical activity of adults, no meaning at all is observed to be understood from just a word" (Ollett 2020).

The correspondence with language models is close. Older systems gave each word one fixed vector, the same in every sentence, and then combined vectors. That is the Bhāṭṭa order: express first, relate after. A Transformer's input embedding is also fixed per word. But after the first layer of self-attention, each position's vector already depends on every other word in the context. By the top layers, "bat" next to "dugout" and "bat" next to "eye" have different vectors. What a Transformer passes upward is meaning-in-relation, the Prābhākara position, with the Bhāṭṭa fixed meaning kept only at the input.

Freschi sketches how a contemporary Prābhākara might put it, and the sketch fits even better. On her reading, each word might give "some preliminary understanding of word-meaning", while "each new word adjusts the meaning of the previous one" through the three criteria, "in a hermeneutic circle" (Freschi 2018). Stack that adjustment in layers and you have a description of a deep self-attention network.

A whole sentence, defined by what is missing#

The Mīmāṃsā definition of a single sentence goes further. Ollett translates Mīmāṃsā Sūtra 2.1.46: "It is a single sentence if, upon being split up, there would be unresolved dependencies, on account of the fact that it has a single meaning/purpose" (Ollett 2020). A sentence is a unit because its parts need each other. Cut it, and expectancies are left hanging.

This is a test of completeness. It says when the joining is done. Attention has no such test. A Transformer processes a fixed window, and every position attends to the positions it is allowed to see. Nothing in the mechanism marks that an expectancy has been met and the unit is closed. The network may learn something like it; the mechanism does not supply it.

[AUTHOR NOTE: if you have studied śābdabodha with a teacher, or met these three conditions in the Tarkasaṅgraha or elsewhere, one or two sentences on how they were taught, and which examples were used, would ground this part.]

What this means for RTM#

The Recognition Theory of Mind, the hypothesis this series tests, has five layers. The fourth is pratyabhijñā, recognition: the act of knowing a present thing as one already known, "this is that". In the Kashmir Śaiva tradition it names the recognition of one's own nature. RTM borrows the structure, not the theology, and treats it as a claim about cognition that can be tested.

Attention gives that structure a mechanical form. A query is a present need. A key is a record from the past. A high score is a match: this present need is answered by that past record. In RTM's terms, attention is a mechanism of recognition, because it compares the present with stored records of the past and selects by match. A fixed recurrent state can recognise only what it chose to keep; attention can recognise anything it stored.

It is worth noting that Abhinavagupta, the great Pratyabhijñā author, also wrote on how sentences mean. The SEP reports that he accepted Jayanta's theory of purport, tātparya, as the function that unifies a sentence (Keating, SEP). I do not draw a doctrine from this. It shows that the tradition that named recognition also cared how parts become one meaning.

RTM makes a prediction here, which the ledger records as speculative. If recognition at layer four needs content-addressed access to stored records, then a system that keeps only a fixed summary should fail at recognition of items it did not mark as important when they arrived. In people, that predicts a pattern: recognition of incidental details should depend on memory systems that store episodes, not on working memory, which is fixed and small. That is broadly the existing picture in memory research, so the prediction is weak. A sharper test is in machines: a fixed-state model should fail at in-context recognition tasks that a Transformer of equal size passes, in proportion to the number of items to be recognised. The copying results already point that way (Jelassi et al. 2024).

The bridge breaks for RTM too. Pratyabhijñā in its home tradition is a recognition that transforms the one who recognises. An attention head transforms nothing about itself; it returns a blend and is done. If RTM needs recognition to change the recogniser, attention is at most a part of it, and the part that does the changing is elsewhere.

The strongest objection to this paper is that I have mislocated attention's advantage. On this view, the Transformer won because it trains fast on parallel hardware, not because it stores more. Vaswani and colleagues led with exactly that: their model was "more parallelizable" and needed "significantly less time to train" (Vaswani et al. 2017). Sutskever's recurrent system had already shown that long sentences were manageable without attention (Sutskever, Vinyals and Le 2014). Give a recurrent network the same compute and data, the objection runs, and it does as well.

A second form of the objection is theoretical. Recurrent networks with large enough states can, in principle, compute anything. A state of a million numbers holds a great deal. The capacity argument depends on the state being small compared with what must be recalled, and real states are not small.

Both points have force, and the first is partly right. Parallel training was decisive for the Transformer's adoption, and selective state-space models have closed much of the gap on language tasks (Gu and Dao 2023). If the question were "why did Transformers win in 2017", efficiency would be most of the answer.

But the question is what attention solves that recurrence could not. On that question, the capacity argument survives both objections. Jelassi and colleagues proved the gap for copying, independent of training speed, and found it in pretrained models of practical size (Jelassi et al. 2024). My instrument found it at equal memory: the 4,096-number grid lost to the cache past 64 pairs, whatever the compute. And a state of a million numbers is still fixed. Contexts of millions of tokens, each needing exact recall, exceed it.

There is a third objection, from the other side. Perhaps attention solves nothing deep, because a Transformer does not use its access well: lost in the middle shows that (Liu et al. 2023). My reply is that this separates the mechanism from the trained model. The mechanism removes the bottleneck. Whether training makes good use of the removal is a separate question, and the answer so far is "partly".

I would give up the central claim, that attention's distinctive contribution is exact, content-addressed access to a store that grows with the input, if any of the following were shown.

A fixed state that copies. A recurrent or state-space model with a state of fixed size that copies arbitrary strings much longer than its state can hold, with accuracy matching a Transformer. That would contradict the capacity bound, so it would mean the bound had been misapplied.

Equal-memory parity on retrieval. A careful comparison at equal memory in which a fixed-state model matches attention on in-context retrieval of many items. My instrument used random keys; trained keys might change the picture.

Attention that adds nothing. Evidence that removing the attention layers from hybrid models costs nothing on retrieval-heavy tasks. Current hybrids keep a few attention layers for a reason; if that reason vanished, so would part of my argument.

Weights that explain. A demonstration that attention weights, read alone, predict the effect of removing inputs across trained models as well as gradient or removal methods do. That would overturn Claim C15.8.

What I still don't know#

I do not know how far training raises the capacity of a fixed state. The instrument's keys were random. A learned outer-product memory with a delta-rule update may hold several times d pairs (Schlag, Irie and Schmidhuber 2021), and I have not measured how many.

I do not know why trained Transformers lose information in the middle of long contexts. Position encoding, training data and the shape of learned keys are all candidates, and the published work does not yet separate them.

I do not know whether a single attention head should have a way to say that nothing fits. Some designs add an extra slot that absorbs weight when no record matches; whether that is the tradition's anvayānupapatti in mechanical form, or just a convenience, is unclear to me.

I do not know whether RTM's layer four needs recognition to change the recogniser. If it does, attention is only part of the story, and I do not yet know which part of a network would supply the rest.

[AUTHOR NOTE: if you have used a long-context model in your own work and noticed it missing something in the middle of a long document, a brief first-person example would make the lost-in-the-middle finding concrete.]

Invented scenario The scene below continues the invented story.

A month later, Farhan brings his laptop back to Basavanagudi. He has added an attention layer to his old model. It still reads the Kannada sentence with the same recurrent network. But the decoder now keeps every encoder state and, for each English word, scores them all and takes a weighted blend.

The long sentences come out right. Meenakshi asks him to show the weights for one of them. The English verb, near the start, points at the Kannada verb, at the very end. "That is what I do," she says. "I go and find it."

"It does not know it is looking," Farhan says. "It just compares." He shows her a sentence where the largest weight sits on a small word that hardly matters, while a heavier word with a smaller weight carries the meaning. Venkatesh Shastri, reading over their shoulders, says the old authors would have asked a different question: not where the machine looked, but whether the words it joined were the ones that needed each other.

The answer to the paper's question can now be given in full. A recurrent network carries the past in a state of fixed size. It must decide what to keep before it knows what will be asked, it trades recent against distant memory, and its exact recall is bounded by its size. Attention solves this by keeping a record for every item and choosing among them at question time, by matching a query against keys. The softmax makes the choice learnable; the √d scaling keeps it learnable as vectors grow. Attention solves the problem of exact, late, content-addressed access to a past that a fixed state cannot hold. It pays for this with memory that grows with the input and comparison that grows with its square.

The rest is folklore, and the instrument shows why. Attention weights are not explanations; one time in five, the most heavily weighted item was not the most influential. Access is not use; trained models lose the middle. And attention is not attention in the psychologist's sense; it is the opposite of limited capacity. What the Mīmāṃsā authors add is a reminder of what the mechanism leaves out. They asked which words need each other, whether they fit, and when the sentence is whole. Attention answers the third of their questions, proximity, better than any recurrent state. It does not ask the other two.

Start with the failure. Cho and colleagues measured how translation quality fell with sentence length in the encoder–decoder systems of 2014 (Cho et al. 2014). Then read Bahdanau, Cho and Bengio, who proposed the soft search that became attention (Bahdanau, Cho and Bengio 2014), and Bahdanau's own account of how he came to it, published by Andrej Karpathy (Bahdanau, email to Karpathy). Sutskever, Vinyals and Le are the essential counterweight: a recurrent system without attention that handled long sentences well enough (Sutskever, Vinyals and Le 2014).

For the Transformer itself, read Vaswani and colleagues, paying attention to their reason for the √d scaling (Vaswani et al. 2017). Luong, Pham and Manning compared simple variants of attention for translation (Luong, Pham and Manning 2015). For the memory view, read Ramsauer and colleagues on modern Hopfield networks (Ramsauer et al. 2020) and Schlag, Irie and Schmidhuber on linear attention as fast weights (Schlag, Irie and Schmidhuber 2021).

For the case for recurrence, read Katharopoulos and colleagues on linear attention (Katharopoulos et al. 2020) and Gu and Dao on Mamba (Gu and Dao 2023). Then read Jelassi and colleagues on copying, which marks the limit (Jelassi et al. 2024). For what attention does not do, read Jain and Wallace and the reply by Wiegreffe and Pinter together (Jain and Wallace 2019; Wiegreffe and Pinter 2019), then Liu and colleagues on lost in the middle (Liu et al. 2023). Olsson and colleagues on induction heads show what attention does well (Olsson et al. 2022).

For the Indian side, Malcolm Keating's entry in the Stanford Encyclopedia of Philosophy sets out the theories of sentence meaning clearly (Keating, SEP). Freschi's short post on the three conditions is a good companion (Freschi 2018), and Ollett's presentation on the context principle gives the key definitions in translation (Ollett 2020). Paper 11 in this series explains the crosstalk behind the capacity results: High-dimensional geometry of representation.

Each source was opened and checked against its publisher page, DOI record, arXiv record or a primary-text edition before it was cited. Anything I could not confirm is marked “to verify”.

Primary research

1. Cho, K., van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014). On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. arXiv:1409.1259. https://arxiv.org/abs/1409.1259
2. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473; ICLR 2015. https://arxiv.org/abs/1409.0473
3. Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. arXiv:1409.3215; NeurIPS 2014. https://arxiv.org/abs/1409.3215
4. Graves, A. (2013). Generating Sequences With Recurrent Neural Networks. arXiv:1308.0850. https://arxiv.org/abs/1308.0850
5. Graves, A., Wayne, G., and Danihelka, I. (2014). Neural Turing Machines. arXiv:1410.5401. https://arxiv.org/abs/1410.5401
6. Weston, J., Chopra, S., and Bordes, A. (2014). Memory Networks. arXiv:1410.3916. https://arxiv.org/abs/1410.3916
7. Sukhbaatar, S., Szlam, A., Weston, J., and Fergus, R. (2015). End-To-End Memory Networks. arXiv:1503.08895; NeurIPS 2015. https://arxiv.org/abs/1503.08895
8. Luong, M.-T., Pham, H., and Manning, C. D. (2015). Effective Approaches to Attention-based Neural Machine Translation. arXiv:1508.04025. https://arxiv.org/abs/1508.04025
9. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention Is All You Need. arXiv:1706.03762; NeurIPS 2017. https://arxiv.org/abs/1706.03762
10. Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences 79(8), 2554–2558. https://doi.org/10.1073/pnas.79.8.2554
11. Plate, T. A. (1995). Holographic reduced representations. IEEE Transactions on Neural Networks 6(3), 623–641. https://doi.org/10.1109/72.377968
12. Ramsauer, H., Schäfl, B., Lehner, J., et al. (2020). Hopfield Networks is All You Need. arXiv:2008.02217; ICLR 2021. https://arxiv.org/abs/2008.02217
13. Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. arXiv:2006.16236; ICML 2020. https://arxiv.org/abs/2006.16236
14. Schlag, I., Irie, K., and Schmidhuber, J. (2021). Linear Transformers Are Secretly Fast Weight Programmers. arXiv:2102.11174; ICML 2021. https://arxiv.org/abs/2102.11174
15. Gu, A., and Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752. https://arxiv.org/abs/2312.00752
16. Dao, T., and Gu, A. (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. arXiv:2405.21060; ICML 2024. https://arxiv.org/abs/2405.21060
17. Jelassi, S., Brandfonbrener, D., Kakade, S. M., and Malach, E. (2024). Repeat After Me: Transformers are Better than State Space Models at Copying. arXiv:2402.01032; ICML 2024. https://arxiv.org/abs/2402.01032
18. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172; TACL 2024. https://arxiv.org/abs/2307.03172
19. Jain, S., and Wallace, B. C. (2019). Attention is not Explanation. arXiv:1902.10186; NAACL 2019. https://arxiv.org/abs/1902.10186
20. Wiegreffe, S., and Pinter, Y. (2019). Attention is not not Explanation. arXiv:1908.04626; EMNLP 2019. https://arxiv.org/abs/1908.04626
21. Olsson, C., Elhage, N., Nanda, N., et al. (2022). In-context Learning and Induction Heads. arXiv:2209.11895. https://arxiv.org/abs/2209.11895
22. Bahdanau, D. Email to A. Karpathy on the origin of attention, published with permission by Karpathy on X, December 2024; full text also reproduced on Hacker News. https://news.ycombinator.com/item?id=42310213

Primary texts and translations

1. Keating, M. The Literal-Nonliteral Distinction in Classical Indian Philosophy. Stanford Encyclopedia of Philosophy. Sections on ākāṅkṣā, sannidhi and yogyatā, abhihitānvaya and anvitābhidhāna. https://plato.stanford.edu/entries/literal-nonliteral-india/
2. Ollett, A. (2020). The Context Principle in Indian Philosophy of Language. Presentation, with translations from Mīmāṃsā Sūtra 2.1.46 and Prābhākara texts.

Secondary sources and leads

1. Freschi, E. (2018). Proximity, semantic fitness and syntactic expectancy as criteria for the sentence meaning. The Indian Philosophy Blog. https://indianphilosophyblog.org/2018/02/07/proximity-semantic-fitness-and-syntactic-expectancy-as-criteria-for-the-sentence-meaning/
2. Wolchover, N. (2026). Is 'Attention' the Essence of Intelligence? Quanta Magazine. https://www.quantamagazine.org/is-attention-the-essence-of-intelligence-20261012/

Mahapatra, R. R. (2026). The attention mechanism derived from first principles. Mind Frontier, paper 15, version 1.0. Rajesh Ranjan Mahapatra. https://rajeshrmahapatra.com/papers/mind-frontier/U0358-attention-from-first-principles.html (DOI to be assigned.)

@misc{mahapatra2026u0358,
  author       = {Mahapatra, Rajesh Ranjan},
  title        = {The attention mechanism derived from first principles},
  year         = {2026},
  howpublished = {Mind Frontier, paper 15},
  note         = {Version 1.0},
  doi          = {10.XXXX/placeholder},
  url          = {https://rajeshrmahapatra.com/papers/mind-frontier/U0358-attention-from-first-principles.html}
}