TLDR
- A neural network can be rewritten in many ways that compute exactly the same function. Any spectral quantity that changes under those rewritings, such as one layer’s spectral norm, is a fact about notation.
- Random matrices of the same shape already have spread-out spectra. A spectral feature is evidence of structure only if it clears that noise floor, by a principled threshold such as Gavish and Donoho’s, not an elbow.
- Across more than 10,000 networks, the spectral-norm measure from a correct generalisation bound correlated with the generalisation gap in the wrong direction, with Kendall’s τ of −0.537.
- Spectral components become causal when cutting them changes behaviour. Cutting one late MLP matrix in GPT-J to 1% of its rank raised factual accuracy from 13.1% to 24.0%.
- Spectral components lack their own nature, as Nāgārjuna argued of the Abhidharma’s dharmas: they depend on a parametrisation, a ruler and a sample, and are real by what they do.
Sundar Raman spent thirty years in the tape library of a radio station in Chennai, and since retiring he has restored old concert recordings for anyone who brings him a reel. His main tool is a spectrum display. It shows how much of the sound sits at each frequency, and it makes the hiss on an old tape visible as a grey floor that runs across the whole picture. The obvious move is to cut everything below a line. Sundar learned early that the obvious move ruins recordings. The brushed cymbal in a 1960s nagaswaram concert lives in the same band as the hiss, at about the same level. "The display tells you where the energy is," he says. "It does not tell you what the energy is doing. For that you cut, and you listen."
His nephew Kiran fine-tunes language models for a company in Bengaluru. Over lunch one Sunday he showed his uncle a plot: the singular values of a weight matrix from one layer of a model, sorted from largest to smallest, falling off in a long tail. "This layer is heavy-tailed," Kiran said. "That's supposed to mean it's well trained." Sundar looked at the plot for a long time. "And if you cut the tail," he asked, "what does the model stop doing?"
Sundar and Kiran are invented, and nothing they say is evidence. Sundar's question is real, and it is the question this paper asks. A neural network is built from matrices. Every matrix has a spectrum: a list of numbers, its singular values, that says how strongly it stretches space in each of a set of special directions. Researchers compute these spectra all the time. They report that a layer has low effective rank, that its spectrum has a heavy tail, that its largest singular value is small, that its activations collapse onto a few directions. Some of these numbers tell you what the network does. Others describe the way the network happens to be written down, or the size of the sample used to measure it. Which spectral quantities of weight and activation matrices are causally meaningful, and which are descriptive artefacts?
My answer is that a spectral quantity earns a causal reading only if it passes three sieves, and that most published spectral quantities pass one or two. The first sieve is symmetry. A network can be rewritten in many ways that compute exactly the same function, and a quantity that changes under those rewritings is a fact about the notation. The second sieve is the noise floor. A random matrix of the same shape already has a spread-out spectrum, and a quantity that sits inside the range random matrices produce is a fact about the sample. The third sieve is intervention. A spectral component is causally operative if removing it, or scaling it, changes behaviour in the predicted way, by more than removing a matched random component does. Spectra are a good way of finding candidates. They are a poor way of establishing anything until you cut and listen.
The Abhidharma, the Buddhist tradition of systematic analysis, decomposed persons and experiences into ultimate constituents called dharmas, each said to bear its own intrinsic nature. Nāgārjuna, in the second century, argued that nothing has such a nature, and that every constituent exists only in dependence on others. The singular value decomposition invites exactly the Abhidharma reading: here are the components, and each bears its own strength. This paper argues for Nāgārjuna's reading. The components are real for the purposes they serve and are defined only relative to a frame: an inner product, a parametrisation, a data distribution and the rest of the network.
This paper follows paper 9 in this series, which asked which claims about representations survive a change of basis. Spectra are what you get when you ask that question of matrices, and much of the same logic carries over.
What a decomposition gives you
Start with what the singular value decomposition says. Any matrix, of any shape, can be written as a rotation, followed by a stretch along the axes, followed by another rotation. The stretch factors are the singular values. They are non-negative, and by convention they are listed from largest to smallest. The first rotation picks out the input directions that the matrix treats specially; the second says where each of those directions ends up. Each triple of an input direction, a stretch factor and an output direction is a rank-one component, and the matrix is the sum of its components.
Three facts about this decomposition make it attractive. First, it always exists. Second, when the singular values are all different, it is unique, up to flipping the sign of matched pairs of directions. Third, keeping only the largest components gives the best possible approximation of the matrix at that rank, in the sense of least squared error. That last fact, a classical theorem, is why truncating an SVD is the standard way to compress a matrix, to denoise it and to estimate how many dimensions its action really uses.
These facts are why spectra feel like they reveal something. If a matrix has a few large singular values and many small ones, it acts, to a good approximation, on a few directions and ignores the rest. Its "effective rank" is low. If its singular values fall off as a power law, it has structure at every scale. If its largest singular value is small, it can't stretch any input by much, and so small changes to the input produce small changes to the output.
Two kinds of matrix, three kinds of spectrum
In a neural network, spectra come from two kinds of matrix. Weight matrices are the parameters: the matrices each layer multiplies its input by. Activation matrices are measurements: stack the activation vectors a layer produces for a sample of inputs, one row per input, and you have a matrix whose spectrum describes how the layer's outputs spread across directions for that sample.
There is a third kind that matters more than either and is reported less often. The Jacobian of a network, or of a block of layers, says how a small change to the input changes the output, at a particular input. For a purely linear block, the Jacobian is just the product of the block's weight matrices. For a nonlinear block, it depends on the input, and it combines the weights with the slopes of the nonlinearities at that point. The Jacobian is what the network does to small perturbations. Its spectrum, unlike the spectrum of any one weight matrix, belongs to the function.
The distinction is easy to state and easy to lose. A weight matrix's spectrum describes one factor in a product. An activation matrix's spectrum describes a sample. A Jacobian's spectrum describes the function at a point. Many arguments about what spectra reveal move from one to another without noticing.
Three sieves
Sundar's practice gives the outline of a test. Before he trusts a feature on the spectrum display, he asks three questions. Would the feature still be there if the same sound had been recorded on a different machine, with a different gain setting? Is it above the hiss? And when he removes it, does the music change in the way he expected? Translated to networks, those become three sieves.
The symmetry sieve. Neural networks have symmetries: changes to the parameters that leave the function exactly unchanged. Paper 9 described the simplest. In a space that the next layer reads only through a matrix, you can apply any invertible linear map to the activations and its inverse to the next layer's weights, and nothing downstream can tell. In a layer followed by a ReLU, you can multiply a unit's incoming weights by any positive number and its outgoing weights by the reciprocal. A spectral quantity passes the symmetry sieve if it doesn't change under the symmetries that the architecture actually has. If it does change, then two networks that compute the same function, and so have the same accuracy, the same behaviour and the same errors, can have different values, and the quantity can't be a cause of anything those networks do.
The noise-floor sieve. A matrix filled with independent random numbers is not spectrally flat. Its singular values spread out over a predictable range, and the shape of that range depends on the matrix's aspect ratio. Random-matrix theory says exactly what to expect. A spectral feature passes the noise-floor sieve if it lies outside the range that a random matrix of the same shape and scale would produce, by a margin set by the theory, not by eye.
The intervention sieve. A spectral component passes this sieve if changing it changes behaviour as predicted, and does so by more than a matched control. The control matters. Removing any part of a matrix changes the network a little. The question is whether removing this part changes it in the way the spectral story says, compared with removing a part of the same size chosen at random.
A quantity that passes all three sieves is, in my sense, causally meaningful. A quantity that fails the first is a fact about the parametrisation. A quantity that fails the second is a fact about the sample. A quantity that passes the first two but has never been put through the third is a well-founded description, which may predict a great deal, but whose causal role hasn't been shown.
| Quantity | Symmetry | Noise floor | Intervention | My verdict |
|---|---|---|---|---|
| Largest singular value of one weight matrix | fails | passes | untested | notation, unless an operation fixes it |
| Product of the layers’ spectral norms | fails | passes | untested | notation; wrong sign across 10,000 networks |
| Sharpness (top Hessian eigenvalues) | fails | passes | untested | notation (Dinh et al.) |
| Heavy-tail exponent of a layer | partly | passes | untested | good description, cause not shown |
| Effective rank of activations | passes | partly | untested | sample, until baselined |
| Spectrum of a head’s query-key product | passes | passes | passes | function |
| Spectrum of the Jacobian | passes | passes | passes | function, at the inputs used |
| Rank of a fine-tuning update (LoRA) | passes | passes | passes | function, given the base model |
| Effect of truncating a matrix (LASER) | passes | passes | partly | mechanism, for this parametrisation |
| Data modes in deep linear networks | passes | passes | passes | function, for that model class |
The table sorts common spectral quantities. I'll spend the rest of the paper defending its entries, and the reader should treat the classifications in the right-hand columns as my claims, argued below, not as settled facts.
Researchers took GPT-J, picked the input matrix of the MLP block in one of its last layers, and cut it to 1% of its maximum rank, with no retraining. On a factual-question benchmark where the model scored 13.1%, what happened?
Commit before you read the answer. With JavaScript off, every answer is shown below the buttons.
Sieve one: the network's symmetries
The balanced twins are not a curiosity. Every ReLU network has this rescaling symmetry at every hidden unit, and every transformer has, in addition, the large family of linear symmetries in its residual stream that paper 9 described. The symmetries are continuous: there isn't one twin but an infinite family, related by smoothly varying rescalings. Any quantity that varies smoothly along that family can be pushed to almost any value without changing the function.
This is the point Dinh and colleagues made in 2017 about sharpness, the idea that flat minima of the loss generalise better than sharp ones. Using exactly these symmetries, they showed that "we can exploit the particular geometry of parameter space induced by the inherent symmetries that these architectures exhibit to build equivalent models corresponding to arbitrarily sharper minima". Sharpness, measured by the curvature of the loss with respect to the parameters, is a spectral quantity: the largest eigenvalues of the Hessian. Dinh and colleagues added that "if we allow to reparametrize a function, the geometry of its parameters can change drastically without affecting its generalization properties."
Their argument doesn't say sharpness is useless. It says that, as a raw number, it fails the symmetry sieve, and so any correlation it has with generalisation has to come from somewhere other than sharpness itself. One natural candidate is the training procedure. Gradient descent with weight decay doesn't explore the whole symmetric family; it tends to settle where the layers' scales are balanced, because unbalanced scales cost more weight decay for the same function. So among networks trained the standard way, per-layer quantities may be effectively fixed, and may track something about the function. That is a respectable explanation, but it makes the quantity a symptom of the training procedure's preferences. Change the procedure, or rescale after training, and the correlation can break.
Which quantities survive
The symmetry sieve sorts quantities cleanly once the symmetries are written down.
The spectrum of a single weight matrix fails, in any layer where the units on either side can be rescaled or where the space on either side is unprivileged. In a transformer's residual stream, any invertible change of basis absorbed by the matrices that read and write the stream changes the singular values of each of those matrices individually. Rotations leave singular values alone, because singular values are defined using lengths, and rotations preserve lengths. Stretches don't.
The spectrum of a product of matrices that the symmetry cancels out of passes. In an attention head, the query and key matrices only ever appear together, as the product that compares a query with a key. Any invertible change inside the head's small internal space cancels between them. So the singular values of the query matrix alone are a fact about the notation, while the singular values of the combined query-key product are a fact about what the head compares. The same holds for the value and output matrices, which appear only as a product.
The spectrum of the Jacobian passes, because it is computed from the function. So does any statement about how many directions in input space change the output at all, which is a statement about the Jacobian's rank.
The shape of a spectrum, as opposed to its scale, partly passes. A heavy-tailed spectrum is one whose singular values, or the eigenvalues of the matrix multiplied by its own transpose, fall off like a power law. The exponent of that power law doesn't change when the whole matrix is multiplied by a constant: multiplying every eigenvalue by the same number shifts a log-log plot without changing its slope. So the exponent survives layer-wide rescaling, which norms don't. It does not survive unit-by-unit rescaling, which can reshape the spectrum arbitrarily. I'll come back to this, because I think it explains part of a real empirical finding.
The whitened mirror shows something that the symmetry sieve alone doesn't. Even where a quantity survives every symmetry of the architecture, the SVD still depends on how lengths are measured on each side. The default choice, the ordinary Euclidean length of the coordinates, is the right one only if the coordinates are on comparable scales and uncorrelated in the data that flows through. Paper 9 called this the ruler problem, and it returns here in a sharper form.
The story matters here because the spectral-norm product is the paradigm of a theoretically grounded spectral quantity. It appears in a correct theorem. It's a Lipschitz constant, so it has an exact meaning: no input perturbation can be amplified by more than this factor. And yet as a predictor across real networks it pointed the wrong way. Part of the explanation is the symmetry sieve: the product isn't invariant under unit-by-unit rescaling, so its value in a trained network depends on which member of the symmetric family training happened to land on, and that choice correlates with hyperparameters in ways that have nothing to do with the bound. Part of it is that a bound being correct says nothing about whether it is tight. A theorem that says test error is at most some enormous number is true and uninformative, and the ordering of enormous numbers across networks need not track the ordering of their actual errors.
Sieve two: the noise floor
Now suppose a quantity passes the symmetry sieve. The next question is whether it reflects anything about the network at all, or only the shape of the matrix it was computed from.
Fill a matrix with independent random numbers, all with the same spread. Its singular values do not come out equal. They spread over a range, and random-matrix theory gives that range exactly in the limit of large matrices. For the eigenvalues of a sample covariance matrix, computed from a number of samples in a number of dimensions, the spread is described by the Marchenko–Pastur law. Its edges depend on the ratio of dimensions to samples. When there are four samples for every dimension, the eigenvalues of pure noise with unit variance fall between about 0.25 and 2.25. When there are as many dimensions as samples, they fall between 0 and 4. The fewer samples you have per dimension, the more spread out the spectrum of pure noise is.
This has a direct consequence for activation spectra. Collect the activations of a 4,096-unit layer on 1,000 inputs, and their covariance has at most 1,000 non-zero eigenvalues, however rich the layer's representation is. Compute its "effective rank" and you will find something no larger than 1,000, which is a fact about your sample. Collect them on 16,000 inputs and the noise-only spectrum narrows, so more of any real structure becomes visible. Any claim that a representation is low-dimensional needs to state how many inputs it was measured on and to compare the measured spectrum with the noise-only spectrum for that ratio.
When a signal emerges from the bulk
Structure shows up in a spectrum as spikes, eigenvalues that stand out beyond the noise edge. But a real signal doesn't automatically produce a spike. In 2004, Baik, Ben Arous and Péché showed that for a sample covariance matrix with one or a few special directions, there is a sharp threshold. Below a critical signal strength, the special direction produces no spike: its eigenvalue stays inside the bulk, and the top sample eigenvector is almost unrelated to the true direction. Above the threshold, a spike separates from the bulk, and the top eigenvector begins to align with the truth. In their words, "a phase transition phenomena is observed."
Two consequences follow. A real signal below the threshold is invisible to spectral analysis, however carefully it is done. And a signal just above the threshold produces a spike whose direction is only partly right. The sample spike over-reports its own strength and points somewhere between the true direction and noise.
The worked case explains a practice that is common and wrong. Someone plots a spectrum, sees an elbow and declares the matrix to have that many meaningful components. The elbow is chosen by eye. It depends on the plot's scale, and it ignores the shape of the noise-only spectrum, which has no elbow of its own but has a hard edge that moves with the aspect ratio. Gavish and Donoho's threshold isn't the only principled choice, but it is a choice with a stated objective and a known dependence on the matrix shape, and it should be the default.
Noise scaled so the bulk edge is 1 + √β. Formulas for the spiked model and the Gavish–Donoho threshold for known noise. Computed in your browser.
Heavy tails and what they predict
Weight matrices are not random, and their spectra don't look like the noise-only bulk. Charles Martin and Michael Mahoney applied random-matrix theory to the weight matrices of trained networks, from small ones trained from scratch to production models such as AlexNet and Inception. Their JMLR paper reports that the spectra of trained layers display "signatures of traditionally-regularized statistical models, even in the absence of exogenously specifying traditional forms of explicit regularization." They developed "a theory to identify 5+1 Phases of Training, corresponding to increasing amounts of Implicit Self-Regularization", running from spectra that look like random noise, through a bulk with a few spikes, to fully heavy-tailed spectra, with a separate phase in which the matrix collapses to low rank. In well-trained modern networks, the spectra are heavy-tailed, and the exponent of the tail is a summary of how far training has moved the layer from noise.
Their follow-up in Nature Communications is the most practically striking result in this area. Working with hundreds of publicly available pretrained models, without access to their training or test data, they compared norm-based metrics with metrics based on the power-law exponents. They found that "norm based metrics correlate well with reported test accuracies for well-trained models, but that they often cannot distinguish well-trained versus poorly-trained models," while "power law based metrics can do much better."
That finding is revealing when set against the three sieves. The power-law exponent passes part of the symmetry sieve, as I argued above: it is unchanged when a layer is multiplied by a constant, while norms are not. In models trained the usual way, layer-wide scale is one of the largest sources of variation that has nothing to do with quality, because it depends on initialisation, learning rate and weight decay. A quantity that ignores scale is protected from that source of noise. My reading, which I haven't seen tested, is that part of why exponents beat norms is that they fail fewer symmetries.
The exponent also passes the noise-floor sieve by construction. A heavy tail is exactly what a random matrix of the same shape doesn't have.
It hasn't, as far as I know, been put through the intervention sieve. The evidence that heavy tails go with good models is correlational, across models trained in different ways by different people. It is entirely possible that heavy tails are a by-product of whatever makes a model good, and not part of the mechanism. The test is an operation I'll call spectral surgery: keep a trained layer's singular directions, replace its singular values with a light-tailed set of the same total size, and measure what happens. If accuracy falls sharply and falls more than a matched control, the tail is doing work. If it barely moves, the tail is a symptom.
Sieve three: intervention
Spectral analysis becomes causal when someone cuts. The cleanest case I know is also one of the most surprising results in the recent literature.
LASER passes all three sieves in an instructive way. It operates on the SVD of a single matrix, which fails the symmetry sieve as a description. But as an intervention, truncation is applied to the network as it actually is, and its effect is measured on behaviour. The behavioural result is invariant: any function-preserving rewriting of the network that you then truncated would give a different truncated network, but the result being reported is about this network, with this parametrisation, and what happens when this matrix is cut. That is an honest causal claim about a specific operation. The danger lies only in generalising it to a claim about the network's function independent of its parametrisation, which LASER doesn't make.
The noise-floor sieve is passed implicitly, by the size of the effect. And the intervention sieve is passed in a partial form: the authors compare across layers and matrix types, which is a kind of control, since reducing the rank of most matrices doesn't help. What I'd like to see, and haven't found, is the direct matched control: remove a set of components of the same total energy chosen at random, or chosen from the middle of the spectrum, and compare. Without it, we know that removing the small components of a late MLP input matrix helps; we don't know how much of the help is specific to the components being small.
Low rank as a fact about change
A second line of evidence comes from adapting models, not trimming them. Aghajanyan, Zettlemoyer and Gupta asked how many parameters a pretrained language model really needs to change to learn a new task. Building on an earlier method for measuring the intrinsic dimension of an optimisation problem, they restricted fine-tuning to a random low-dimensional subspace of the full parameter space. "By optimizing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90% of the full parameter performance levels on MRPC," a paraphrase-detection task.
LoRA, from Hu and colleagues at Microsoft, turned the same idea into the most widely used fine-tuning method. It freezes the pretrained weights and learns, for selected matrices, an update that is constrained to be low rank: the product of a tall thin matrix and a short wide one. The authors reported that on GPT-3 175B, compared with full fine-tuning, LoRA "can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times," while matching or exceeding full fine-tuning quality on the tasks they tested.
| Result | Model | Number | What it shows |
|---|---|---|---|
| Intrinsic-dimension fine-tuning | RoBERTa on MRPC | 200 parameters reach 90% of full performance | the useful change lives in very few directions |
| LoRA low-rank updates | GPT-3 175B | 10,000× fewer trainable parameters; 3× less GPU memory | a low-rank change suffices for adaptation |
| LASER truncation | GPT-J, late MLP input | rank 4,096 cut to 4; CounterFact 13.1% → 24.0% | most of one matrix’s rank can be removed, with gains |
These results pass the intervention sieve directly, because constraining the rank of an update is an intervention, and the outcome is behaviour. What they establish is a claim about the spectrum of the change, not of the weights: the difference between the pretrained and adapted matrices can be low rank without loss. That is a different and, I think, more meaningful spectral fact than the effective rank of any weight matrix. It says the task-relevant adjustment lives in a small number of directions, given the pretrained model as a starting point. It's a fact about the model and the task together, which is the kind of dependence Nāgārjuna would expect.
A negative control that every spectral claim should carry
The intervention sieve needs a control, and the right control has two parts. The first is the matched random intervention: if the claim is that removing the bottom 99% of a matrix's components improves accuracy, compare with removing a random set of directions that together carry the same fraction of the matrix's energy, and with removing the same number of components from the top. The second is the noise-only model of the third thought experiment: run the same analysis on an untrained network of the same architecture, measured on the same number of inputs, and report how the result differs.
These controls are cheap. The first costs one more evaluation per condition. The second costs one forward pass of an untrained network. Their absence is the single most common reason a published spectral finding, in my reading of this literature, can't be classified beyond "descriptive".
When an operation makes a spectrum meaningful
The symmetry sieve has an exception that matters in practice. A quantity that fails the sieve in general can become meaningful in a particular network, if something in that network fixes the symmetry. Paper 9 made the same point about bases: a coordinate means nothing in general, but a ReLU or a per-coordinate optimiser can make it special. The spectral version is that an operation which acts on a per-layer quantity can make that quantity part of the mechanism.
The clearest example is spectral normalisation. Miyato and colleagues proposed it in 2018 to stabilise the training of generative adversarial networks. At every step, each weight matrix in the discriminator is divided by an estimate of its largest singular value, computed cheaply by a few rounds of power iteration. The result is that every layer has spectral norm one, and the whole discriminator, if its nonlinearities have slope at most one, has a Lipschitz constant of at most one. That property is a fact about the function: it limits how fast the discriminator's output can change as its input changes. The authors reported that it stabilised training, and the method has been widely used since.
In a spectrally normalised network, the per-layer spectral norm is no longer a free choice of notation. The training procedure pins it, and so it is part of how the network works. The rescaling symmetry still exists in principle, but the network you have is one in which the symmetry has been fixed by an operation that is part of training. This is exactly analogous to a privileged basis. The lesson is not that per-layer spectra are always meaningless. It is that a per-layer spectrum means something when, and because, some operation treats it specially, and that a claim about it should name the operation.
Weight decay plays a weaker version of the same role. It penalises the total size of the weights, and among all the members of a symmetric family that compute the same function, it prefers the one with the smallest total size, which is the one with balanced scales across layers. So in a network trained with weight decay, per-layer quantities drift toward a particular member of the family. They become more meaningful, though less firmly than under spectral normalisation, because the preference is soft and competes with the loss.
Spectra of learning
So far I have treated spectra as snapshots of a finished network. Some of the most illuminating spectral results are about how networks learn, and here the spectrum that matters is that of the data.
Saxe, McClelland and Ganguli found exact solutions for the learning dynamics of deep linear networks, networks with several layers but no nonlinearities. Such a network computes only a linear map, but its learning is nonlinear, and they showed it reproduces phenomena seen in real networks, "including long plateaus followed by rapid transitions to lower error solutions." The solution has a clean spectral form. Decompose the correlation between inputs and target outputs in the training data by its SVD. Each component of that decomposition, each "mode", is learned separately, along an S-shaped curve: a long period of slow growth, a rapid rise, and a plateau. The time each mode takes scales inversely with its singular value. Strong modes are learned first; weak ones wait.
In a later paper, the same authors used this to explain features of how children acquire concepts, including "the hierarchical differentiation of concepts through rapid developmental transitions". If the strongest mode in a set of items separates animals from plants, the network learns that distinction first. The next mode might separate birds from fish, and that arrives later. A broad distinction precedes finer ones because broad distinctions are spectrally stronger in the data.
This is a spectral claim that passes all three sieves, for the model it's about. It is about the data's spectrum, which no reparametrisation of the network can change. Its components are well above any noise floor in the cases studied. And it is causal in the strongest sense: it is a derivation of the dynamics, so changing a mode's singular value changes when it is learned, as the theory says. The limitation is scope. Deep linear networks are a model, and how much of the picture survives nonlinearity, attention and the scale of modern training is an open question. But it shows what a causally meaningful spectral account looks like: a spectrum, a mechanism that acts on it, and a prediction about what happens when the spectrum changes.
The pull toward low rank
A second group of results says that training, especially in deep networks, prefers low-rank solutions. Arora, Cohen, Hu and Luo studied deep linear networks used for matrix completion, filling in missing entries of a matrix from the ones you can see. They found that adding depth strengthens a tendency toward low-rank solutions, and that this tendency is not well described by the most common mathematical proxy, the nuclear norm, the sum of singular values. Huh and colleagues reported that deeper networks are inductively biased toward embeddings with lower effective rank, and offered this as part of the explanation for why heavily overparametrised networks still generalise.
These are claims about the function's spectrum, or about the spectrum of the learned map from inputs to outputs, and so they pass the symmetry sieve. Whether they pass the others depends on how effective rank is measured, which brings back the threshold problem. A claim that one network has lower effective rank than another is a claim about where some threshold falls on each spectrum, and the answer can change with the threshold, the sample size and the way the spectrum is normalised. Schaeffer, Miranda and Koyejo made a related point about "emergent abilities" in language models: abilities that appear suddenly at some scale can be an artefact of the metric used to score them, with smooth underlying improvement turned into an apparent jump by a threshold. Effective rank is a thresholded quantity too. Claims about it are claims about a spectrum seen through a particular lens, and the lens should be reported.
Collapse at the end of training
The last of these results concerns the final layer of a classifier. Papyan, Han and Donoho studied what happens in the terminal phase of training, which "begins at the epoch where training error first vanishes," when training continues even though the network already classifies every training example correctly. They found a cluster of phenomena they called neural collapse. The final-layer features of examples from the same class collapse toward their class mean. The class means arrange themselves as far apart as possible, in a symmetric configuration called a simplex. The classifier's weights line up with the class means. In spectral terms, the final-layer activation matrix becomes very close to rank equal to the number of classes minus one, with almost nothing in any other direction.
Neural collapse is an activation spectrum that is far above any noise floor and invariant to rotations of the feature space. It is less clear what it does. A network that has collapsed classifies by nearest class mean, which is simple and stable. Whether collapse helps the network generalise to new examples, or is a side effect of driving the training loss toward zero, has been argued over since, and I don't think it is settled. It is a good case of a quantity that passes two sieves cleanly and waits on the third.
Dharmas and their emptiness
The Abhidharma, the systematic analysis that grew out of the early Buddhist discourses, took apart persons and experience into their constituents. The discourses had already analysed experience "in terms of name-and-form (nāma-rūpa), the five aggregates (Skt., skandha, Pali, khandha), the twelve sense fields (āyatana), or the eighteen sense elements (dhātu)," as Noa Ronkin's Stanford Encyclopedia entry puts it. The Abhidharma went further, toward lists of the final units of analysis, the dharmas. "As the final units of Abhidhamma analysis, dhammas are reckoned the ultimate constituents of experience." And: "there is no being or person apart from dhammas."
What makes a dharma ultimate is that it bears its own nature. One commentarial gloss quoted in the same entry says that dharma "means 'upholding,' [namely], upholding intrinsic nature (svabhāva)," and a Theravādin commentary defines them as entities so called "because they bear their particular inherent characteristics," a particular characteristic being "the intrinsic nature that is not held in common by other dhammas." The tradition distinguished two registers for talking about the world. Ordinary talk of persons, chariots and forests was conventional (saṃvṛti). Talk of dharmas was ultimate (paramārtha).
The SVD invites an Abhidharma reading. A matrix is "nothing apart from" its components. Each component is defined by a property it does not share with the others, its own singular value, and its own pair of directions orthogonal to every other component's. The decomposition is unique when the singular values are distinct. Ordinary talk about a layer, that it "detects edges" or "stores facts", is conventional. Talk of its components and their strengths seems to be ultimate.
Nāgārjuna's reply to the Abhidharma was that nothing has svabhāva. Jan Westerhoff's entry on Nāgārjuna describes svabhāva in the sense under attack as existence "independent of anything else that also happens to exist", and as "not brought about by any causal process, as unchangeable and as independent of any other object." The Ābhidharmikas, he notes, "saw no conflict between objects being causally produced and their possessing svabhāva." Nāgārjuna's response, as Westerhoff presents it, is that "since the causal relation does not exist from its own side, is conceptually constructed, and therefore empty, each causally related object must be so constructed and therefore empty in the most profound sense of being conceptually constructed." On this view, "there can be no such thing as ultimate truth, a theory describing how things really are, independent of our interests and conceptual resources employed in describing it." What remains is conventional truth, and on this view it is the only kind there is.
The three sieves are, in effect, an argument that spectral components lack svabhāva. They depend on the parametrisation, as the balanced twins showed. They depend on the ruler on each side of the matrix, as the whitened mirror showed. They depend on the sample, as the noise floor showed. And their causal role depends on the rest of the network, as LASER showed: the same truncation helps in a late MLP matrix and doesn't in an early one. A singular value is not a quantity the matrix has on its own. It is a quantity the matrix has in a frame.
Nāgārjuna's reading also says something positive, which is why I find it more useful than a flat scepticism. Emptiness is not non-existence. A thing empty of svabhāva still functions, still has effects, still enters into causal relations; it simply does so dependently. A spectral component that passes the intervention sieve is real in exactly that sense. Truncating the small components of a late MLP matrix in GPT-J changed its answers to factual questions. That effect is as real as anything in the paper. What it doesn't show is that those components are the layer's own, independent of the frame in which they were computed. They are real conventionally, which on Nāgārjuna's account is the only way anything is real.
Notation reading
A weight matrix has a few large singular values. True for this parametrisation; a function-preserving rescaling can change it. Meaningful only where an operation fixes the scales.
Sample reading
Activations on some inputs spread over few directions. True for this sample; it must be compared with an untrained network and a noise floor for the same number of inputs.
Function reading
Only a few directions of change at this layer affect the output: the Jacobian is low rank. Survives reparametrisation and can be tested by projecting directions out.
Madhyamaka reading
Every one of these is true in a frame. The question is not which is the layer’s own rank but which frame supports the interventions you need (after Nāgārjuna).
Summaries are mine. The last panel applies a second-century argument to a case it did not address.
Where this sits on RTM's ladder
This series uses a five-rung map, the Recognition-Theoretic Model, to place questions about mind. The lowest rung, L1 or spanda, is dynamics: the measurable activity of a system. L2, prakāśa, is manifestation: what is made available, to the system itself or to anything else. L3, vimarśa, is reflexivity; L4, pratyabhijñā, recognition; L5, cit, the ground. The map is a way of keeping questions apart. It is not evidence for anything.
Every spectral quantity in this paper describes L1. Spectra summarise how a network transforms activity. The three sieves sort L1 descriptions into those that belong to the function, those that belong to the notation and those that belong to the sample. Only the first kind can say anything about L2. If a network's Jacobian has low rank at some layer, then only a few directions of change at that layer affect anything downstream, and that is a fact about what the layer makes available. A weight matrix's singular values, on their own, don't say that.
| Analogy or parallel | Holds | Breaks |
|---|---|---|
| Tape spectrum ↔ matrix spectrum | ✔ the display finds candidates; cutting tests them | frequency is physical; a matrix spectrum depends on parametrisation |
| Drum modes ↔ SVD components | ✔ a decomposition whose parts act separately | a drum’s modes are fixed by its shape; a layer’s are not |
| Abhidharma dharmas ↔ components | ✔ the urge to find parts with their own nature | its aims were soteriological; the SVD has a uniqueness theorem |
| Nāgārjuna’s emptiness ↔ three sieves | ✔ real by function, dependent on a frame | his argument is general; mine is local |
| Surgeon cutting by size ↔ truncation | ✔ small is not the same as unimportant | a model can be re-run; a patient cannot |
| RTM ladder ↔ network layers | ✘ | nothing here shows a network has higher rungs |
What this means for claims about machine minds
Spectral language appears in claims about machine minds more often than one might expect. A model's internal states are said to occupy a low-dimensional "self-model" subspace; a layer's effective rank is offered as a measure of the richness of its representations; a sudden change in a spectrum during training is read as the moment a capacity appears. Each of these is a claim that spectral quantities track something mental, and each can be put through the sieves.
The first question is which matrix the claim is about. A claim about the spectrum of a weight matrix fails the symmetry sieve unless some operation fixes the gauge. A claim about an activation matrix has to state its sample size and clear the noise floor. A claim about a Jacobian, or about the effect of an intervention, can pass both. The second question is what the claim adds once it passes. Suppose a model's activations, on prompts about itself, really do concentrate in a few directions, well above the noise floor, and projecting out those directions really does stop it talking about itself. That is a well-founded claim about a mechanism. It is the same kind of claim as LASER's, and it is worth having. It is not, by itself, evidence that there is a self for the directions to model. The step from "these directions control self-description" to "the model has a self-model" is a step up the ladder, and nothing spectral takes it.
The constraint the sieves impose is the same one paper 9 drew for bases. Two networks that compute exactly the same function are equal in every respect that their behaviour, their dispositions or their causal structure at the level of function could show. A theory of machine mind that gives them different verdicts because their weight matrices have different spectra is a theory of notation.
The strongest case against my answer
The strongest objection is that I've set the bar in the wrong place. Engineers use spectral quantities to choose between models, to diagnose training problems and to decide where to compress. For those purposes, what matters is whether a quantity predicts, not whether it causes. Martin and Mahoney's power-law metrics distinguish well-trained from poorly trained models without access to any data. That is useful whether or not heavy tails are part of any mechanism, and calling them "descriptive" undersells them. A thermometer doesn't cause a fever, and nobody thinks it should be judged by whether it does.
I accept most of this. Prediction is valuable, and a quantity that predicts reliably across a wide range of models is worth having. My claim is narrower. Prediction across a population of models trained in the usual ways can break when the population changes, and the symmetry sieve says exactly how it can break: train differently, rescale after training, or change the optimiser, and a quantity that fails the sieve can be moved without moving the thing it was predicting. A thermometer is trustworthy because we know the mechanism connecting it to body temperature. A spectral metric that hasn't passed the intervention sieve is a thermometer whose mechanism we don't know, and it should be used with that in mind.
A second objection is that training fixes the gauge, so the symmetry sieve is academic. Networks are trained with weight decay, normalisation layers and particular initialisations, and these together pick out a narrow region of each symmetric family. In practice, the objection goes, nobody rescales a hidden unit by ten. This is a fair point and the reason I gave weight decay its own paragraph. But it changes the claim from "this quantity reflects the function" to "this quantity reflects the function, given the conventions of training". The second claim is fine as long as it is stated, and it becomes false exactly when someone compares networks trained under different conventions, which is what large studies like Jiang's do.
A third objection is that interventions are confounded. Truncating a matrix changes the distribution of activations downstream, perhaps pushing the network into states it never saw in training. An effect of truncation might then be an effect of being off-distribution, not of the components removed. This is real, and it is why the matched control matters: a random intervention of the same size moves the network off-distribution too, and the comparison isolates what is specific to the components chosen.
A counterexample and a failure boundary
The counterexample to any simple view that well-grounded spectral quantities are causal is the spectral-norm product. It sits inside a correct theorem, it has an exact meaning as a Lipschitz bound, and across more than 10,000 networks it correlated with the generalisation gap in the wrong direction. A quantity can be theoretically meaningful and empirically misleading at once.
The failure boundary of my answer is the set of architectures whose symmetries I don't know. For ReLU networks and the linear paths in a transformer, the symmetries can be written down. For architectures with normalisation layers, gating, mixtures of experts or attention patterns that depend on the input, the full symmetry group is harder to state, and the symmetry sieve can only be applied to the symmetries one knows about. A quantity might pass every sieve I can apply and still fail one I haven't found. The noise-floor sieve has a boundary too: random-matrix results hold for large matrices with independent entries, and activation matrices have strongly correlated rows, so the noise-only spectrum should be computed from an untrained network or from shuffled data, not taken from the formula.
A negative control
The negative control for this paper's thesis is a quantity that passes all three sieves by construction. If my sieves were simply hostile to spectral claims, they would reject everything. They don't. The singular values of an attention head's combined query-key product pass the symmetry sieve, can be checked against the noise floor, and can be intervened on by truncating the product. So can the Jacobian's spectrum. So can Saxe's data modes. A sieve that lets these through and stops per-layer spectral norms is sorting, not just refusing.
What would change my mind
What I don't know
I don't know whether heavy-tail exponents predict model quality partly because they ignore layer-wide scale; it is my reading, and the test is to compute them after unit-by-unit rescaling designed to move them. I don't know whether neural collapse helps generalisation or is a side effect of pushing the training loss toward zero. I don't know whether the components LASER removes are noise, as the authors suggest, or structured competing responses, which their own analysis also hints at. And I don't know the full symmetry group of a modern transformer with normalisation layers, which means my first sieve is incomplete for exactly the models people care most about.
Of these, the symmetry group matters most, and it is also the most answerable. It is a mathematical question about a fixed architecture, and partial answers already exist for parts of it.
The spectral audit
The three sieves can be run as a procedure. It is short, and most of its steps cost little.
| # | Spectral audit step | Pass looks like |
|---|---|---|
| 1 | Name the matrix: weights, product, activations on a sample, or Jacobian | matrix and sample size stated |
| 2 | List the function-preserving symmetries that act on it | rescalings and changes of basis written down |
| 3 | Apply a random symmetry; check outputs | outputs unchanged within float noise (negative control) |
| 4 | Recompute the quantity | unchanged, or the operation that fixes it is named |
| 5 | Compute it on an untrained network, same sample size | trained value clearly outside the baseline |
| 6 | Count components with a stated threshold | Gavish–Donoho or another stated rule, not an elbow |
| 7 | Intervene on the component: truncate, scale or project | behaviour changes as predicted |
| 8 | Repeat with a matched random intervention | specific effect exceeds the random one |
Worked cases: the spectral-norm product fails step 4. Heavy-tail exponents pass steps 5 and 6 and have not faced 7. LASER passes 7 and has only a partial step 8.
The audit has not been validated, and its steps are my proposal. I've applied it informally to the main results in this paper, and the classifications in the earlier table are the outcome.
Exact solution for a deep linear network with small initial weights (initial strength 0.001). Curves normalised to each mode’s final value. Computed in your browser.
1. A layer with a smaller spectral norm is simpler, so the network should generalise better.
2. Two networks with the same function cannot differ in any way that matters for what they do.
3. A heavy-tailed spectrum is a sign of good training, whatever the mechanism.
4. A spectral component is causally real only if changing it changes behaviour.
A sorter for your intuitions. It describes positions; it does not score them.
- 2004
Baik, Ben Arous and Péché
A phase transition for spikes in sample covariance. - 2013
Saxe, McClelland and Ganguli; Gavish and Donoho
Data modes learned in order; the 4/√3 threshold. - 2017
Bartlett and colleagues; Dinh and colleagues
Spectral-norm margin bounds; sharpness can be reparametrised. - 2018
Martin and Mahoney; Miyato and colleagues; Li and colleagues
Heavy-tailed self-regularisation; spectral normalisation; intrinsic dimension. - 2019
Jiang and colleagues; Arora and colleagues
The spectral measure has the wrong sign; depth pulls toward low rank. - 2020
Papyan, Han and Donoho; Aghajanyan and colleagues; Martin, Peng and Mahoney
Neural collapse; 200 parameters suffice; tail exponents beat norms. - 2021
LoRA; Huh and colleagues
Low-rank updates for adaptation; a low-rank simplicity bias. - 2023
LASER
Cutting a late MLP matrix to 1% of its rank improves GPT-J.
Claim Ledger
| ID | Claim | Confidence | Evidence type | What would refute it |
|---|---|---|---|---|
| U0086-C1 | Spectra of single weight matrices change under function-preserving rescalings and changes of basis; spectra computed from the function, such as the Jacobian’s, do not. | established | Linear algebra; Dinh et al. 2017 | None for the stated symmetries; the open question is which further symmetries exist. |
| U0086-C2 | A spectral feature is evidence of structure only if it lies outside the range produced by random matrices of the same shape and sample size. | established | Marchenko–Pastur law; Baik, Ben Arous and Péché 2004; Gavish and Donoho 2014 | Spectral features inside the noise range proving reliably predictive of behaviour. |
| U0086-C3 | Truncation and low-rank updates show that some spectral components are causally operative in trained language models. | likely | Sharma, Ash and Misra 2023; Hu et al. 2021; Aghajanyan et al. 2020 | Matched random truncations of equal energy giving the same gains. |
| U0086-C4 | Theoretically grounded spectral capacity measures can correlate with generalisation in the wrong direction across real networks. | established | Jiang et al. 2019 | A replication over comparable networks finding a positive correlation. |
| U0086-C5 | Heavy-tail exponents predict model quality partly because they ignore layer-wide scale; their causal role is untested. | speculative | My reading of Martin and Mahoney 2018 and Martin, Peng and Mahoney 2020 | Exponents keeping their predictive power after adversarial per-unit rescaling, or the heavy-tail transplant leaving accuracy unchanged. |
| U0086-C6 | Nāgārjuna’s critique of svabhāva is a useful model of spectral components: real by function, dependent on a frame. | speculative | Textual reading via Ronkin and Westerhoff, SEP | A frame-independent spectral decomposition of a network layer that predicts interventions better than frame-relative ones. |
Dated predictions
| ID | Prediction | Resolves | How it will be judged |
|---|---|---|---|
| U0086-P1 | A published study will compare LASER-style truncation with random truncation of equal Frobenius energy and find that the gains are specific to removing the smallest components. | end of 2028 | Published result meeting the description; resolves no if random truncation matches the gains. |
| U0086-P2 | Heavy-tail exponents computed after a function-preserving per-unit rescaling designed to move them will lose most of their correlation with model quality. | end of 2029 | A published test of the rescaling; resolves no if the correlation survives. |
Glossary
- Abhidharma (अभिधर्म)
- Systematic Buddhist analysis of experience into ultimate constituents, the dharmas.
- Activation matrix
- A matrix of a layer’s outputs on a sample of inputs, one row per input.
- Dharma (धर्म)
- In Abhidharma, an ultimate constituent of experience said to bear its own nature.
- Effective rank
- A count of how many directions a matrix substantially uses, set by some threshold.
- Jacobian
- The matrix of how small input changes affect the outputs, at a given input.
- LASER
- Layer-selective rank reduction: replacing one weight matrix by a low-rank truncation of its SVD.
- LoRA
- Low-rank adaptation: fine-tuning by learning a low-rank update to frozen weights.
- Marchenko–Pastur law
- The distribution of eigenvalues of a sample covariance matrix of pure noise.
- Noise floor
- The range of spectral values a random matrix of the same shape would produce.
- Paramārtha (परमार्थ)
- The ultimate register of truth, contrasted with the conventional.
- Saṃvṛti (संवृति)
- Conventional truth; for Madhyamaka, the only kind there is.
- Singular value
- A stretch factor of a matrix along one of its special input directions.
- Spanda (स्पन्द)
- Vibration or dynamism; in RTM, the layer of measurable activity, L1.
- Spectral norm
- The largest singular value of a matrix; the most it can stretch any input.
- Svabhāva (स्वभाव)
- Intrinsic nature or own-being, which Nāgārjuna argued nothing has.
- Symmetry (of a network)
- A change to the parameters that leaves the computed function exactly unchanged.
Cut and listen
Sundar's rule for old tapes was that the display tells you where the energy is and not what it is doing. For neural networks, the same rule can be made precise. A spectral quantity is a fact about the network's function only if it survives the network's symmetries; a fact about the network and not the sample only if it clears the noise floor that random matrices of the same shape produce; and a fact about mechanism only if cutting or scaling it changes behaviour by more than a matched random cut.
By those standards, the singular values of a single weight matrix are, in most networks, a fact about notation, unless an operation such as spectral normalisation fixes them. The spectral-norm product, though it sits inside a correct theorem, failed as a predictor across 10,000 networks. Activation spectra are facts about samples until compared with an untrained baseline. Heavy-tailed exponents pass two sieves and predict well, and probably owe part of their success to ignoring scale; they wait on an intervention. The quantities that pass all three are the ones built from the function or tested by changing it: the Jacobian's spectrum, combined products like an attention head's query-key matrix, the data modes that set the order of learning, the low-rank updates that LoRA showed suffice for adaptation, and the truncations that LASER showed can make GPT-J answer more factual questions correctly.
The Abhidharma read decompositions as finding what bears its own nature. Nāgārjuna replied that nothing does, and that dependence is not a defect but the way things exist. Spectral components are best read his way: real when they do something, and always real in a frame. The practical upshot is Sundar's. Use the spectrum to find candidates. Then cut, and listen.
Symmetry
If a function-preserving rewrite moves it, it is a fact about notation.Noise floor
If a random matrix of the same shape shows it, it is a fact about the sample.Intervention
If cutting it changes nothing beyond a random cut, it is a description.The evidence
Wrong-sign spectral norms across 10,000 networks; LASER’s 13.1% to 24.0%; LoRA.The test
Name the matrix, rewrite, baseline, cut and compare with a random cut.The old tool
Nāgārjuna: components without own-nature, real by what they do.Sources
Primary research
- Sharma, P., Ash, J. T. and Misra, D. (2023). The truth is in there: improving reasoning in language models with layer-selective rank reduction. arXiv:2312.13558. arxiv.org/abs/2312.13558
- Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D. and Bengio, S. (2019). Fantastic generalization measures and where to find them. arXiv:1912.02178. arxiv.org/abs/1912.02178
- Bartlett, P., Foster, D. J. and Telgarsky, M. (2017). Spectrally-normalized margin bounds for neural networks. arXiv:1706.08498. arxiv.org/abs/1706.08498
- Dinh, L., Pascanu, R., Bengio, S. and Bengio, Y. (2017). Sharp minima can generalize for deep nets. arXiv:1703.04933. arxiv.org/abs/1703.04933
- Martin, C. H. and Mahoney, M. W. (2018). Implicit self-regularization in deep neural networks: evidence from random matrix theory and implications for learning. arXiv:1810.01075. arxiv.org/abs/1810.01075
- Martin, C. H., Peng, T. S. and Mahoney, M. W. (2020). Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. arXiv:2002.06716. arxiv.org/abs/2002.06716
- Baik, J., Ben Arous, G. and Péché, S. (2004). Phase transition of the largest eigenvalue for non-null complex sample covariance matrices. arXiv:math/0403022. arxiv.org/abs/math/0403022
- Gavish, M. and Donoho, D. L. (2013). The optimal hard threshold for singular values is 4/√3. arXiv:1305.5870. arxiv.org/abs/1305.5870
- Aghajanyan, A., Zettlemoyer, L. and Gupta, S. (2020). Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv:2012.13255. arxiv.org/abs/2012.13255
- Li, C., Farkhoor, H., Liu, R. and Yosinski, J. (2018). Measuring the intrinsic dimension of objective landscapes. arXiv:1804.08838. arxiv.org/abs/1804.08838
- Hu, E. J. et al. (2021). LoRA: low-rank adaptation of large language models. arXiv:2106.09685. arxiv.org/abs/2106.09685
- Miyato, T., Kataoka, T., Koyama, M. and Yoshida, Y. (2018). Spectral normalization for generative adversarial networks. arXiv:1802.05957. arxiv.org/abs/1802.05957
- Saxe, A. M., McClelland, J. L. and Ganguli, S. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv:1312.6120. arxiv.org/abs/1312.6120
- Saxe, A. M., McClelland, J. L. and Ganguli, S. (2018). A mathematical theory of semantic development in deep neural networks. arXiv:1810.10531. arxiv.org/abs/1810.10531
- Arora, S., Cohen, N., Hu, W. and Luo, Y. (2019). Implicit regularization in deep matrix factorization. arXiv:1905.13655. arxiv.org/abs/1905.13655
- Huh, M. et al. (2021). The low-rank simplicity bias in deep networks. arXiv:2103.10427. arxiv.org/abs/2103.10427
- Papyan, V., Han, X. Y. and Donoho, D. L. (2020). Prevalence of neural collapse during the terminal phase of deep learning training. arXiv:2008.08186. arxiv.org/abs/2008.08186
- Schaeffer, R., Miranda, B. and Koyejo, S. (2023). Are emergent abilities of large language models a mirage? arXiv:2304.15004. arxiv.org/abs/2304.15004
Secondary or leads
- Ronkin, N. Abhidharma. Stanford Encyclopedia of Philosophy. plato.stanford.edu/entries/abhidharma/
- Westerhoff, J. C. Nāgārjuna. Stanford Encyclopedia of Philosophy. plato.stanford.edu/entries/nagarjuna/
- Mind Frontier paper 9: Vector spaces before representations. rajeshrmahapatra.com/papers/mind-frontier/U0093-vector-spaces-before-r
Cite this work
Mahapatra, Rajesh Ranjan. "Linear algebra of representations: SVD, spectra and what they reveal in networks." Mind Frontier paper U0086. rajeshrmahapatra.com, 2026. https://rajeshrmahapatra.com/papers/mind-frontier/U0086-linear-algebra-of-representations.html. DOI: [to be assigned].
@misc{mahapatra2026u0086,
author = {Mahapatra, Rajesh Ranjan},
title = {Linear algebra of representations: SVD, spectra and what they reveal in networks},
year = {2026},
howpublished = {rajeshrmahapatra.com},
note = {Mind Frontier paper U0086},
url = {https://rajeshrmahapatra.com/papers/mind-frontier/U0086-linear-algebra-of-representations.html},
doi = {10.XXXX/placeholder}
}