Coordinates and what they hide

Start with what a basis is. A layer of a network with 1,280 units produces, for each input, 1,280 numbers. We can treat those numbers as coordinates of a point in a 1,280-dimensional space, measured against 1,280 reference directions, one per unit. Those reference directions are a basis. But they are not the only one. Any set of 1,280 independent directions would do, and the same point would then have 1,280 different numbers. The point hasn't moved. Only its description has.

In a mathematics class, this is a reason to prefer statements that don't depend on the basis. In a neural network, there is a complication that makes the question interesting instead of trivial. The network doesn't just store vectors; it computes with them. The next layer multiplies the vector by a matrix of weights. If you change the basis of one layer's activations, you can absorb the change into the next layer's weights, and the network computes exactly the same function. Nothing it does has changed. So whether a basis is meaningful depends on whether anything in the network fails to absorb a change like that.

The figure shows the simplest case. A cloud of points, two clusters, described in two bases. In the original basis, the clusters are separated along the first axis: one coordinate seems to "detect" the difference. After a rotation of forty-five degrees, both coordinates mix the clusters, and neither detects anything on its own. But the clusters are as separable as before, the line that divides them is the same line, and the distance between any two points is unchanged. The coordinate's meaning moved; the geometry didn't.

Three kinds of change

It helps to separate three kinds of change of basis, because claims survive them differently.

A permutation shuffles the order of the units. Unit 412 becomes unit 7. Every serious claim survives this, because a label is only a label.

A rotation, or more generally an orthogonal transformation, turns the axes rigidly. Lengths and angles are preserved. Individual coordinates are not.

An invertible linear transformation can rotate, stretch and shear. It preserves which points lie on which side of a flat dividing surface, how many dimensions a set of points spans and which points can be linearly predicted from which. It doesn't preserve lengths or angles.

There is a fourth kind, which sits between the first two and matters for networks with ReLUs. A positive rescaling multiplies each unit by its own positive number. It preserves the sign of every coordinate, and so it preserves which units are active and which are silent, but it changes their sizes and every angle between vectors. In a ReLU network, you can rescale a unit's incoming weights by a positive factor and its outgoing weights by the reciprocal, and the function is unchanged. So even where the basis is privileged, the scale of each axis usually isn't. A claim such as "unit 412 is active for cricket" survives rescaling. A claim such as "unit 412 is twice as active for cricket as unit 413 is for football" does not, unless something fixes the scales.

Permutations, rotations and invertible linear maps form a nested family: every permutation is an orthogonal transformation of a special kind, and every orthogonal transformation is an invertible linear map. Positive rescalings sit inside the invertible maps but outside the rotations. The bigger the family a claim survives, the less it depends on choices someone made about how to describe the network.

What the next layer sees

One more piece of background makes the rest easier. When I say a change of basis can be "absorbed" by the next layer, I mean something concrete. Suppose a layer produces a vector h, and the next layer computes W times h, where W is its weight matrix. Now replace h with Q times h, for some invertible matrix Q, and replace W with W times the inverse of Q. The next layer computes W times the inverse of Q times Q times h, which is W times h. Nothing downstream can tell the difference. The network with Q inserted is a different set of numbers computing the same function.

This is why the question isn't idle. For any trained network, there is an enormous family of other networks, one for each invertible Q at each layer, that behave identically and have entirely different activations. Training happened to land on one member of the family. A claim about representations is a claim about that member unless it holds across the family. And a claim that holds for one member and fails for others is a claim about which member training chose, which may be interesting, but is a different claim from one about what the network computes.

The family shrinks wherever something blocks the absorption. A ReLU between h and W blocks it, because the ReLU applied to Q times h is not Q times the ReLU applied to h, unless Q merely permutes the units or scales them by positive numbers. So in a layer followed by a ReLU, the family of equivalent networks is much smaller: you can relabel units and rescale them, but you can't rotate them. That is the formal reason the unit basis is privileged there, and it is the reason the Network Dissection result, described below, came out as it did.

Three tiers of claim

Here is the classification this paper defends. It isn't original in its mathematics: anyone who has taken a course in linear algebra knows which quantities are invariant under which transformations. What I think is new is applying it systematically to the claims interpretability research actually makes, and asking what evidence each tier needs.

Tier one: claims about the land

A claim belongs to tier one if it survives every invertible linear change of basis. Examples: a concept can be linearly decoded from a layer with a given accuracy; the activations for a set of inputs span a subspace of a given dimension; two classes are linearly separable; projecting out a subspace changes the model's output in a given way; the representation of A can be linearly predicted from the representation of B. These are claims about the information present and about how the network uses it, and they don't depend on the grid. They are not trivially true. A linear probe can fail, a subspace can be large, an intervention can do nothing. But if they are true in one basis they are true in all.

There is a subtlety even here. A claim about linear decoding survives linear changes of basis, but it isn't a claim about the network unless the probe is constrained. John Hewitt and Percy Liang showed that probes trained to predict parts of speech can also learn "control tasks" that assign random outputs to word types, so high probe accuracy can reflect what the probe learned instead of what the representation encodes. They proposed selectivity, the gap between task accuracy and control accuracy, as the measure that "puts linguistic task accuracy in context with the probe's capacity to memorize". Tier one tells you the claim isn't hostage to coordinates. It doesn't tell you the claim is about the representation instead of the probe.

Tier two: claims about the ruler

A claim belongs to tier two if it survives rotations but not stretches. Cosine similarity is the standard example. "The vectors for king and queen have a cosine of 0.7" is unchanged if you rotate the space, but if you stretch one axis by a factor of ten, every cosine changes. Distances, norms, angles, "these two features are orthogonal", "this word is the nearest neighbour of that one": all tier two. They presuppose an inner product, a ruler for lengths and angles, and the default ruler is the Euclidean one inherited from the unit basis.

Is the Euclidean ruler justified? Nothing in the network obviously says so. If the next layer's weights absorb a stretch, the network computes the same function and every cosine changes. Kiho Park, Yo Joong Choe and Victor Veitch took this problem head on. They asked "how do we make sense of geometric notions (e.g., cosine similarity or projection) in the representation space?" and argued that the right inner product isn't the Euclidean one. They identified "a particular (non-Euclidean) inner product that respects language structure", which they call the causal inner product, and showed experimentally "the fundamental role of the choice of inner product". The lesson is that tier-two claims are meaningful once you have argued for the ruler, and not before.

Tier three: claims about the grid

A claim belongs to tier three if it changes under rotation. "Unit 412 fires for cricket" is tier three: rotate the space and no single coordinate need fire for cricket at all, though the information is all still there. So is "this layer has outlier dimensions", "activations are sparse", "this neuron is polysemantic" and "the kurtosis of the activations is high". All of these are statements about how the information is laid out along particular axes. They are meaningful only if the axes are.

Two findings that seemed to disagree

The two results look as if they contradict each other. One team says units are no more meaningful than random directions; the other says rotating the units destroys four-fifths of the detectors. Both used careful methods and both have stood up. How can both be true?

The answer is that they were testing different layers and different senses of meaning, and that the second team tested something the first did not: whether the network's own computation makes the unit basis special. In a convolutional network, each unit's output passes through a rectified linear unit, a function that sets negative values to zero independently for each coordinate. That operation does not commute with rotation. Rotate the inputs to a ReLU and you get a different result from rotating its outputs. So the coordinates that feed into a ReLU are privileged by the computation: the network, in effect, measures along those axes. Training then has a reason to align meaningful features with them, because a feature split across many units would be cut differently by each unit's threshold.

Bau's team measured detectors defined by thresholding individual unit activations in exactly such a layer, and found the alignment. Szegedy's team, using different analyses on different networks, found that images maximising a random direction looked as semantically coherent as those maximising a unit. Both can be true if units carry meaning above chance and random directions also carry meaning, because information is distributed: the space holds the semantics, and the units are a privileged but not exclusive way of slicing it. Ari Morcos and colleagues added a further twist: in networks they studied, "class selectivity is a poor predictor of task importance", and networks that generalise well depend less on single directions. A unit can be both privileged and not very important.

When a basis is privileged

So the question "does unit 412 mean cricket?" becomes the question "does anything in the network make unit 412's axis special?" There are three ways that can happen.

The first is an operation applied coordinate by coordinate. ReLU, GELU and other elementwise nonlinearities privilege the basis they act in. So does any per-coordinate operation: dropout applied to individual units, per-unit normalisation, quantisation that rounds each coordinate separately, and L1 penalties that favour sparsity along the axes.

The second is the training process. An optimiser that treats coordinates separately can privilege a basis even when the model itself would not. The widely used Adam optimiser keeps separate running estimates for each parameter and scales each update coordinate by coordinate.

The third is a convention outside the model: how its weights are stored, which hardware is used, how a downstream system reads it. These can make a basis special in practice without making it special to the model's function.

Where none of these applies, the basis is unprivileged, and tier-three claims about it are claims about notation. In a transformer, the main unprivileged space is the residual stream, the running vector that each layer reads from and adds to. Nothing in the residual stream's arithmetic acts coordinate by coordinate: layers read it through matrices and write to it through matrices, and those matrices can absorb any rotation. The hidden layer of each feed-forward block, after its nonlinearity, is privileged. So, in principle, "neuron 412 means cricket" could be a fact about a feed-forward neuron and a fact about notation if it were a residual-stream dimension.

A test, run

In principle, the residual stream should look the same in every basis. In practice, it doesn't. Anthropic's interpretability team tested why. In a 200-million-parameter transformer, they defined an outlier as a residual-stream value with absolute size above 6 and found that a typical layer had 20 to 60 outlier dimensions among 1,280. The activations were heavy-tailed. Then they ran Farida's test: they viewed the same activations in a fixed random orthonormal basis. The measured kurtosis was almost exactly 3, the value expected for a basis with nothing special about it.

So something was privileging the residual-stream basis, though nothing in the model's forward computation should. The team tested candidates in turn. Replacing LayerNorm with a rotation-invariant normalisation left the outliers in place. Randomly rotating the model's matrices to test floating-point effects hurt performance only by tiny amounts. Training models with random rotations between the residual stream and the computation gave the same performance and residual-stream kurtosis of almost exactly 3, while heavy tails reappeared in the basis where computation happened. Their provisional suspect was Adam, which normalises gradient updates coordinate by coordinate. They are careful to say the evidence implicating Adam is suggestive, not conclusive.

This is the most instructive experiment I know for the question this paper asks. It shows that a basis can be privileged by training in a space where the architecture privileges nothing, that the privilege is real in the sense that it shows up in the data, and that it can be removed without changing what the model can do. A tier-three claim about the residual stream of a model trained with Adam is therefore a true claim about that model, and a claim about an accident of its training. Whether it is meaningful depends on what you want it for.

The quantisation story is the perfect counterweight to the rotation thought experiment. Rotate the space, and those six dimensions dissolve into thousands. Yet in the model as trained, written down and run on real hardware, they are what make quantisation fail. So tier-three claims aren't idle. They are meaningful relative to an operation that acts on coordinates, here 8-bit rounding of each value. The claim "these six dimensions are special" is true of the model under per-coordinate quantisation, and false of the model's function in the abstract. Both statements are correct. The mistake is to say the first without naming the operation that makes it true.

The ruler problem

Tier two deserves a section of its own, because it is where most everyday interpretability claims live, and where the default choice is least often questioned. Cosine similarity is the workhorse of embedding analysis. Nearest neighbours, clustering, retrieval and many claims about "what the model thinks is similar to what" are cosines in disguise.

William Timkey and Marten van Schijndel found a striking problem with cosines in contextual language models. "A small number of rogue dimensions, often just 1-3, dominate these measures." Worse, there was "a striking mismatch between the dimensions that dominate similarity measures and those which are important to the behavior of the model." Their remedy was simple: standardising each dimension, subtracting its mean and dividing by its spread, corrected for rogue dimensions "and reveal[ed] underlying representational quality". Standardising is a change of ruler. It stretches each axis differently, and so it changes every cosine.

Notice what that means. The cosines researchers had been reporting depended on a few coordinates with large values, the same kind of coordinates Dettmers found. Those coordinates are a tier-three fact. The cosine, which looks like a tier-two quantity safe under rotation, inherits a tier-three dependence the moment the ruler is fixed to the unit basis without argument. An earlier finding fits the same pattern. Kawin Ethayarajh found that contextual representations in ELMo, BERT and GPT-2 "are not isotropic in any layer": they occupy a narrow cone, so any two random words have a high cosine. Some of what looks like similarity is the shape of the cone.

What makes a ruler right

If the Euclidean ruler isn't given, where should one come from? There are three serious proposals, and each ties the ruler to something outside the coordinates.

One is statistical: whiten or standardise the space so that every direction has the same variance across a reference set of inputs. This removes rogue dimensions, but it depends on the reference set, and it treats all variance as equally important.

A second is functional: define the ruler by what the next part of the network does with the vector. Two representations are close if the model's outputs are close when given them. This is the idea behind Park and colleagues' causal inner product, which they build so that causally separable concepts come out orthogonal, and which lets them construct "probes and steering vectors using counterfactual pairs".

A third is comparative: decide that similarity between two networks should be measured in a way that doesn't depend on either's basis. Simon Kornblith and colleagues showed a surprising limit here. They proved that "neither CCA nor any other statistic that is invariant to invertible linear transformation can measure meaningful similarities between representations of higher dimension than the number of data points." Too much invariance makes a measure blind. Their alternative, centred kernel alignment, gives up full invariance to arbitrary linear maps in exchange for being able to "reliably identify correspondences between representations in networks trained from different initializations".

That result is worth pausing on, because it complicates the picture so far. I have been saying that the more transformations a claim survives, the safer it is. Kornblith's theorem shows a limit. A similarity measure invariant to every invertible linear map can't distinguish meaningful similarity from coincidence when the space is large compared with the data. To learn anything, you have to break some symmetry, which means committing to some ruler. The question is never whether to choose a ruler but whether you have said why.

Meaning by exclusion

So far the argument has been mathematical. Here is where the Buddhist logicians come in, and why I think they help.

Dignāga, working in the fifth and sixth centuries, faced a problem that looks very like Farida's. His school held that only particulars are real: this pot, this moment of blue, each unique and momentary. Yet words are general. "Pot" applies to millions of pots. If there is no real universal "potness" for the word to name, what does the word mean? His answer was apoha, exclusion. As the Routledge Encyclopedia of Philosophy summarises it, "a word denotes its meaning through the exclusion of what is other (anyāpoha)". The word "pot" doesn't pick out a positive essence shared by all pots. It works by ruling out everything that is not a pot. As the publisher's description of the Columbia University Press volume edited by Mark Siderits, Tom Tillemans and Arindam Chakrabarti puts it, the theory rejects properties such as "potness" and claims "that the element uniting all pots is their very difference from all non-pots."

The Stanford Encyclopedia's entry on Dharmakīrti gives Dignāga's own formulation: "A word talks about entities only as they are qualified by the negation of other". It adds that Dignāga applied the analysis to things and to words alike: "non-non-blue is the universal-qua-object (arthasāmānya) signified by the term 'blue'".

Put that next to a neural network. A tier-three claim, "unit 412 means cricket", is a claim that cricket has a positive home: a particular coordinate whose value is the cricketness of the input. That is the analogue of potness, a positive property sitting in one place. A tier-one claim, "cricket can be linearly read out from this layer", is a claim about a contrast: there is a way of dividing the space so that cricket texts fall on one side and non-cricket texts on the other. The dividing surface is defined by what it excludes. It doesn't live in any coordinate. It survives every change of basis because it was never a property of the coordinates in the first place.

This is more than a pun on the word "exclusion". A linear probe is literally a hyperplane that separates "X" from "not X". A steering vector built from counterfactual pairs, as in Park and colleagues' work, is literally a difference: the vector from "the man" to "the woman", averaged over many pairs. Neither is a positive coordinate. Both are what Dignāga would call a non-non-X: a direction defined by what it rules out. The interpretability methods that survive a change of basis are, in structure, apoha methods.

Dharmakīrti's anchor

Dignāga's theory had a well-known weakness, pressed by his realist critics Uddyotakara and Kumārila. If "pot" means "not non-pot", how do we know what counts as a non-pot without already knowing what a pot is? The exclusion seems to presuppose the very thing it was meant to replace. Dharmakīrti's answer, in the seventh century, was to ground exclusion in causal efficacy. The Routledge entry puts his version this way: "Acting toward a real thing requires delimiting the scope of what one ignores so that just some of the differences are left over." A living being "can afford to ignore the differences between the differences that are left over in this exclusion because these differences don't impact the specific goal". What unites the things called "pot" is that they do the same work, holding water, for the purposes at hand.

That is the move Park and colleagues make with their causal inner product, translated into geometry. A bare exclusion, a hyperplane that separates X from not-X, is fixed by the data you happened to label. It survives changes of basis, but it doesn't yet have a ruler: which other directions count as "the same" or "different" depends on an inner product. Dharmakīrti's anchor says: let the ruler be set by effects. Two directions are orthogonal if changing one doesn't affect what the other does. That fixes the tier-two problem by grounding it in the network's behaviour, which is what Dharmakīrti did for Dignāga's theory by grounding exclusion in what things do.

The analogies that weren't

The most famous claims ever made about word vectors are claims about arithmetic. Tomas Mikolov and colleagues introduced efficient methods for learning word vectors and evaluated them partly on analogies: the vector for "king" minus "man" plus "woman" lands near "queen". Tolga Bolukbasi and colleagues used the same arithmetic to show bias, in a paper whose title quotes the result "Man is to Computer Programmer as Woman is to Homemaker?".

Analogy arithmetic is a claim about differences. "King minus man" is a direction defined by a contrast, an apoha object if anything is. Differences survive changes of basis. So it is natural to think that the analogy results are tier-one facts about the land. Mostly they are. But the readout that turned them into words wasn't, and that is a lesson in where coordinate-free claims can be spoiled.

The case matters for this paper because it shows the three-tier test is necessary but not sufficient. Passing Farida's test, surviving a change of basis, makes a claim independent of notation. It doesn't make it independent of every other choice in the pipeline. The readout rule, the probe's capacity, the reference set used for standardising: each can do work that gets credited to the representation.

The melody analogy

A melody is a useful second picture. Play "Happy Birthday" in C major, then in G major. Every note changes. The intervals between notes, the rhythm and the shape of the tune don't. Someone who says "the third note of this tune is E" has made a claim about the key, not the tune. Someone who says "the third note is a major third above the first" has made a claim about the tune. Tier one is the intervals. Tier three is the note names.

Superposition: when no basis will do

There is a case where the whole framing of this paper is strained, and it is becoming the central case in interpretability. Networks often seem to represent more features than they have dimensions. A layer with 1,280 units might track tens of thousands of distinct features, packed in so that each feature is a direction and many directions share each unit. Nelson Elhage, Chris Olah and colleagues built toy models in which this "polysemanticity can be fully understood, arising as a result of models storing additional sparse features in 'superposition'". They found "a phase change" between models that store features one per dimension and models that pack them in, and "a surprising connection to the geometry of uniform polytopes".

In superposition, there is no basis in which every feature has its own axis, because there are more features than axes. Tier-three claims about units are therefore guaranteed to mislead: a unit will respond to several unrelated features, and calling it the cricket neuron hides the others. But the features are still directions, and claims about them can be tier one or tier two. "Feature F is the direction d, and adding d to the residual stream makes the model talk about cricket" is a claim about a direction and an effect. It survives changes of basis, so long as you transform d along with everything else.

The difficulty is that finding the features requires a choice. Methods that look for them, such as sparse dictionary learning, search for an overcomplete set of directions in which activations are sparse. Sparsity is a basis-dependent notion: it counts how many coordinates are non-zero, in the dictionary's basis. So a basis comes back in, not as the units the network happens to have, but as a dictionary chosen because it makes the activations sparse. That choice is defended by its results: dictionary features are often more interpretable and more useful for intervention than units. But it means the features found are relative to a criterion, sparsity, that the network itself may or may not privilege. Superposition is where tier three returns by the back door.

Where this sits on RTM's ladder

This series locates questions about mind on a five-layer map, the Recursive Tantric Model (RTM). L1, spanda, is dynamics: the activity that can be measured. L2, prakāśa, is manifestation: the appearing of a content. L3, vimarśa, is reflexive grasp. L4, pratyabhijñā, is recognition, and L5, cit, is ground. The Sanskrit terms come from Kashmir Śaiva philosophy and are used as labels for kinds of question.

The basis question sits almost entirely at L1. A network's activations are its dynamics, and a change of basis is a change in how we describe them. But the three tiers map onto a distinction RTM cares about. Tier three describes L1 in a particular vocabulary, the unit coordinates. Tier one describes what L1 makes available: what can be read out, which is the precondition for anything appearing at L2. If a machine representation is ever to be a candidate for manifestation in RTM's sense, it will be a tier-one or tier-two fact, defined by what it makes available and how it is used, not a tier-three fact about which unit is active. That is a modest conclusion, but it rules out one popular style of argument: pointing at a single unit's activation as evidence that a machine "has" a concept, let alone experiences one.

What this means for claims about machine minds

This series is about minds, and the basis question matters for the claims about machine minds that are starting to be made. Some of those claims point at a unit or a coordinate: a "deception neuron", a "self" feature, a dimension that tracks the model's confidence. Others point at a direction found by contrast: the difference between activations when the model is told something is true and when it is told it is false, averaged over many cases. The three tiers sort them.

A claim of the first kind is tier three. Before it can bear any weight, someone has to show that the coordinate lives in a privileged space, and that the claim survives Farida's test or moves only as the privileging operation predicts. A claim of the second kind is tier one in its construction, because a difference of averages doesn't depend on the basis. But it is only a claim about readout until someone intervenes. If adding the direction changes behaviour in the predicted way, and removing it removes the behaviour, the claim has become a claim about function.

Anthropic's work on emergent introspective awareness is a useful example of the second kind. The researchers injected concept vectors, directions derived from the model's own activations, into its residual stream and asked whether it would notice and name them. They report success on about 20 per cent of trials at the best layer and injection strength. Whatever one thinks that shows about introspection, and the next papers in this series will take it up, the method is the right kind for this paper's purposes. The injected vectors are directions defined by contrast, tier one in construction; their effect is tested by intervention; and the claim survives any change of basis applied consistently to the vector and the stream. A claim built that way can be argued about on its merits, without worrying that it would vanish under a rotation.

What the tiers can't do is say whether any of this has to do with mind. Tier one tells you a claim is about the network's function and not its notation. It doesn't tell you the function is mental. That gap is the subject of much of the rest of this series, and nothing in this paper narrows it. What this paper can do is clear away a class of claims that never reached the gap at all: claims about a unit that would vanish if the network were written down differently.

The strongest case against my answer

The strongest objection is that the three tiers are too neat, and that in practice everything interesting in interpretability is tier three. Networks are not abstract linear spaces. They are trained with Adam, run with nonlinearities, quantised, pruned and stored in particular formats. Every one of these operations privileges a basis. So the unit basis is, in almost every real model, special, and the insistence on basis-free claims is a mathematician's purism that misses how networks are built. The Network Dissection result shows units are meaningful; the quantisation story shows coordinates matter; the BERT busters work by Olga Kovaleva and colleagues showed that "pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs". On this view, tier three isn't the weak tier. It is the tier where the engineering happens.

A second objection runs the other way. Even tier one isn't safe, because the networks are nonlinear. A change of basis is a linear reparametrisation, but networks also have nonlinear symmetries: rescaling the weights into and out of a ReLU unit by reciprocal factors leaves the function unchanged and changes every activation's scale. Claims that survive linear changes of basis might not survive these. So the three tiers give a false sense of security.

Both objections are right about something. To the first: I agree that in real models the unit basis is usually privileged, and I've tried to say so. My claim isn't that tier-three statements are false. It is that they need an extra premise, the operation that privileges the basis, and that leaving the premise unstated produces claims that are true of one model and silently false of an equivalent one. Network Dissection tested the premise, which is why its claim is strong. The quantisation work named the operation, which is why its claim is useful. To the second: yes. The tiers are about linear changes of basis, and a full account would also ask which claims survive the network's nonlinear symmetries. The per-unit rescaling symmetry of ReLU networks changes magnitudes along unit axes but not which side of zero a unit is on, so it threatens claims about activation size more than claims about which units are active. That is a refinement within tier three, not a refutation of the scheme.

A counterexample and a failure boundary

A counterexample would be a tier-one claim that turned out to depend on the basis. Is there one? Kornblith's theorem is close: a measure invariant to every invertible linear map fails when the dimension exceeds the number of data points. That isn't a tier-one claim depending on the basis; it is a tier-one measure becoming uninformative. But it shows that the safety of tier one can be bought at the price of saying nothing. A tier-one claim that is informative needs enough data relative to dimension, and that condition is not about coordinates at all.

The failure boundary is superposition with nonlinear readout. When features are packed into more directions than there are dimensions, and the network reads them out through nonlinearities that suppress interference, a linear probe can find a feature direction that the network doesn't use in the same way. Tier one says the information is present; it doesn't say the network uses it linearly. Past this boundary, claims need interventions, not just readouts.

A negative control

Every rotation test needs a negative control: a quantity that must not change, to show the rotation was done correctly. The model's outputs are that quantity. If you insert a rotation and its inverse and the logits change by more than floating-point noise, something is wrong with the test, not the model. The privileged-basis experiments used this control: rotating the matrices hurt test loss only by tiny amounts, much less than the variation between random seeds. A claim that moves under a rotation which leaves the outputs fixed is a claim about notation or about an operation outside the forward pass. A claim that moves along with the outputs means the rotation wasn't absorbed properly.

What would change my mind

What I don't know

I don't know whether Adam is the cause of basis alignment in the residual stream; the team that ran the experiments says the evidence is suggestive. I don't know whether the causal inner product generalises beyond the settings Park and colleagues tested, or whether a single ruler can serve every concept in a model. I don't know how much of what dictionary methods find is a property of the network and how much is a property of the sparsity criterion. I don't know whether the per-unit rescaling symmetry of ReLU networks undermines any published interpretability claim in practice. And I don't know whether the analogy between apoha and contrast-based methods says anything about human concepts, though it is the part of this paper I would most like a cognitive scientist to test.

The rotation test

Here is the instrument this paper offers. It turns Farida's question into a procedure that can be applied to any claim about a representation.

The simulation lets you run the rotation test on a toy representation.

Claim Ledger

Dated predictions

Glossary

Anyāpoha (अन्यापोह)

Exclusion of what is other: Dignāga’s account of word meaning.

Apoha (अपोह)

Exclusion; the Buddhist theory that words and concepts mean by ruling out what they are not.

Arthasāmānya (अर्थसामान्य)

The universal as object, for Dignāga a non-non-X constructed by exclusion.

Basis

A set of reference directions in which a vector’s coordinates are written.

Causal inner product

Park, Choe and Veitch’s inner product chosen so that causally separable concepts are orthogonal.

Change of basis

Rewriting the same vectors in different reference directions; the vectors do not move.

Cosine similarity

The cosine of the angle between two vectors under a chosen inner product.

Kurtosis

A measure of how heavy a distribution’s tails are; 3 for a Gaussian.

Linear probe

A linear classifier trained to read a property out of a representation.

Polysemantic unit

A unit that responds to several unrelated features.

Privileged basis

A basis made special by an operation that acts coordinate by coordinate.

Residual stream

The running vector in a transformer that each layer reads from and adds to.

Rogue dimension

A coordinate with large values that dominates similarity measures.

Spanda (स्पन्द)

Vibration or dynamism; in RTM, the layer of measurable activity, L1.

Superposition

Storing more features than dimensions as overlapping directions.

Svalakṣaṇa (स्वलक्षण)

The unique particular, which for Dignāga alone is real.

Turning the grid

Tara went home and ran the test. She inserted a random rotation and its inverse after the layer where she had found unit 412, checked that the model's predictions hadn't changed, and looked again. The cricket unit was gone. Cricket was still there: a linear probe found it as easily as before, and a steering direction built from pairs of cricket and non-cricket sentences worked as well in the rotated model as in the original. Then she checked where unit 412 lived. It was in the hidden layer of a feed-forward block, right after the nonlinearity. In the original model, the axis was privileged, and the unit was a real feature of how that layer computed. She told her aunt both things. "It's a cricket neuron in this model," she said. "And it's a cricket direction in every version of this model." Farida nodded. "So the field is real, and so is the boundary stone. But only one of them moves with the grid."

That is my answer to the question. Claims about neural representations come in three tiers. Claims about linear readout, dimension and the effects of interventions survive any invertible change of basis and are about the network's function. Claims about similarity, distance and angle survive rotations but need a justified ruler, and the evidence from rogue dimensions shows the default ruler often isn't one. Claims about individual coordinates survive nothing, and are meaningful only where an operation in the network or its training privileges the basis: the ReLU layers of Network Dissection, Adam's per-coordinate updates in the residual stream, 8-bit quantisation in the outlier story. Dignāga's apoha, meaning by exclusion, describes why the first tier is safe: its claims are about contrasts, not positive markers. Dharmakīrti's grounding of exclusion in shared effects describes how to fix the second.

The practical step is the rotation test. Before accepting any claim that a unit, coordinate or dimension "means" something, ask which tier the claim is in. If it is tier three, ask what operation makes the basis special, and whether anyone has turned the grid to check. If it is tier two, ask for the ruler and the reason for it. If it is tier one, ask whether the probe was selective and whether there was enough data. A claim that answers these questions is about the land. One that can't is about the flagstaff.

Sources

Primary research

• Lindsey, J. (2025). Emergent introspective awareness in large language models. Transformer Circuits Thread. transformer-circuits.pub/2025/introspection/index.html
• Szegedy, C. et al. (2013). Intriguing properties of neural networks. arXiv:1312.6199. arxiv.org/abs/1312.6199
• Bau, D., Zhou, B., Khosla, A., Oliva, A. and Torralba, A. (2017). Network Dissection: quantifying interpretability of deep visual representations. CVPR. arXiv:1704.05796. arxiv.org/abs/1704.05796
• Morcos, A. S., Barrett, D. G. T., Rabinowitz, N. C. and Botvinick, M. (2018). On the importance of single directions for generalization. arXiv:1803.06959. arxiv.org/abs/1803.06959
• Li, Y., Yosinski, J., Clune, J., Lipson, H. and Hopcroft, J. (2015). Convergent learning: do different neural networks learn the same representations? arXiv:1511.07543. arxiv.org/abs/1511.07543
• Kornblith, S., Norouzi, M., Lee, H. and Hinton, G. (2019). Similarity of neural network representations revisited. arXiv:1905.00414. arxiv.org/abs/1905.00414
• Park, K., Choe, Y. J. and Veitch, V. (2023). The linear representation hypothesis and the geometry of large language models. arXiv:2311.03658. arxiv.org/abs/2311.03658
• Timkey, W. and van Schijndel, M. (2021). All bark and no bite: rogue dimensions in transformer language models obscure representational quality. arXiv:2109.04404. arxiv.org/abs/2109.04404
• Ethayarajh, K. (2019). How contextual are contextualized word representations? arXiv:1909.00512. arxiv.org/abs/1909.00512
• Dettmers, T., Lewis, M., Belkada, Y. and Zettlemoyer, L. (2022). LLM.int8(): 8-bit matrix multiplication for transformers at scale. arXiv:2208.07339. arxiv.org/abs/2208.07339
• Kovaleva, O., Kulshreshtha, S., Rogers, A. and Rumshisky, A. (2021). BERT busters: outlier dimensions that disrupt transformers. arXiv:2105.06990. arxiv.org/abs/2105.06990
• Elhage, N. et al. (2023). Privileged bases in the transformer residual stream. Transformer Circuits Thread. transformer-circuits.pub/2023/privileged-basis/index.html
• Elhage, N. et al. (2022). Toy models of superposition. arXiv:2209.10652. arxiv.org/abs/2209.10652
• Hewitt, J. and Liang, P. (2019). Designing and interpreting probes with control tasks. arXiv:1909.03368. arxiv.org/abs/1909.03368
• Mikolov, T., Chen, K., Corrado, G. and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv:1301.3781. arxiv.org/abs/1301.3781
• Bolukbasi, T. et al. (2016). Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. arXiv:1607.06520. arxiv.org/abs/1607.06520
• Nissim, M., van Noord, R. and van der Goot, R. (2019). Fair is better than sensational: man is to doctor as woman is to doctor. arXiv:1905.09866. arxiv.org/abs/1905.09866

Primary texts and translations

• Siderits, M., Tillemans, T. and Chakrabarti, A. (eds.) (2011). Apoha: Buddhist nominalism and human cognition. Columbia University Press. cup.columbia.edu/book/apoha/9780231153607/

Secondary or leads

• Dharmakīrti, Stanford Encyclopedia of Philosophy (section 2.2, Aboutness: the theory of apoha). plato.stanford.edu/archives/fall2018/entries/dharmakiirti/
• Apoha, Routledge Encyclopedia of Philosophy. www.rep.routledge.com/articles/thematic/apoha/v-1
• Review of Apoha: Buddhist nominalism and human cognition, Notre Dame Philosophical Reviews. ndpr.nd.edu/reviews/apoha-buddhist-nominalism-and-human-cognition/