The directions inside a transformation

A matrix is easiest to understand when we stop reading it as a grid of numbers and watch what it does to directions.

Take any input vector and imagine it as an arrow. A matrix rotates, reflects, stretches, shrinks or collapses that arrow. Different arrows can experience different gains. Some arrive at distinct output directions. Others become redundant because the matrix maps them onto the same line or sends them to zero.

The rank of a matrix counts how many independent output directions can be produced. A rank-one matrix can only paint along one output line, no matter how many input coordinates it receives. A rank-two matrix can paint across a plane. In an m by n matrix, rank can never exceed the smaller of m and n.

This makes rank different from matrix size. A 10,000 by 10,000 matrix contains one hundred million entries, yet it can still have rank three if every column is a combination of three patterns. Conversely, a small 20 by 20 matrix can have full rank and use all twenty independent channels. Parameter count tells us how much is written down. Rank tells us how many independent directional actions survive.

Consider a sensor wall with ten thousand readings. Suppose every reading is generated by only three latent changes: overall temperature, pressure gradient and vibration. A linear map from the wall may have ten thousand columns, but if it responds only to those three coordinated patterns, its realised action is three-dimensional. This is the first source of low-rank economy: many coordinates can carry variations of the same underlying direction. The caveat is immediate. A fourth pattern that is normally absent can still become decisive during a fault.

SVD finds the privileged axes

For an arbitrary rectangular matrix, the input axes that reveal these gains are rarely the coordinate axes printed on the page. Singular value decomposition finds an orthonormal input basis, an orthonormal output basis and a non-negative gain for each matched pair:

The expression on the right is especially revealing. Each term σᵢuᵢvᵢᵀ is a rank-one channel. It listens along vᵢ, scales what it hears by σᵢ, then writes along uᵢ. The full matrix is the sum of these channels. SVD turns a dense transformation into a ranked list of directional actions.

Orthogonality makes that list clean. The input directions do not overlap, so an input can be decomposed into independent coefficients along the vᵢ. The output directions do not overlap either. Energy therefore separates across channels, which is why squared singular values add so neatly. This convenience is mathematical, not semantic. A singular direction may mix thousands of human-interpretable features, and equal singular values permit the corresponding basis to rotate without changing the matrix’s action. When two gains tie, asking which individual singular vector is “the concept” has no unique answer.

flowchart TD
  A["The direction that carried<br/>almost no energy"] --> B["The directions inside a<br/>transformation"]
  B --> C["The theorem is exact, the<br/>question often is not"]
  C --> D["Adaptation is a low-rank<br/>hypothesis"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The theorem is exact, the question often is not

Truncated SVD solves a precise optimisation problem. Trouble begins when we answer a different problem with the same number.

Order the singular values so that σ₁ ≥ σ₂ ≥ … ≥ 0. Keep the first r rank-one channels and call the result Wᵣ. The Eckart–Young result, later generalised by Mirsky, says that no other rank-r matrix is closer to W under the spectral or Frobenius norm.[1][2]

The theorem’s force comes from its narrowness. It specifies the object, the rank constraint and the loss. If those match your operational question, truncated SVD is the answer. If they do not, optimality does not transfer by courtesy.

A three-direction example makes the two norms tangible. Let the singular values be 9, 3 and 1. A rank-one truncation discards gains 3 and 1. Its spectral error is 3 because there exists a unit input whose output is wrong by three units, and no unit input is wrong by more. Its Frobenius error is √10 because the squared errors, 9 and 1, are added. A rank-two truncation reduces the two errors to 1 and 1. Neither number says how often the corresponding inputs occur.

The theorem also clarifies a common misuse of “variance explained”. Squared singular values can be converted into a cumulative energy fraction, but the threshold is a design choice. Ninety-five per cent is not a natural law. A threshold that is adequate for image reconstruction may be indefensible for a diagnostic gate. The norm tells you how error is aggregated; the application tells you which aggregation is legitimate.

The hidden assumption: isotropic use

The Frobenius norm treats every input direction as though it were presented with equal second-moment weight, then treats every output direction as equally costly. That is often a useful neutral default. It is not a law of deployment.

Let E = W − Ŵ be the approximation error. Let x be the inputs the system actually receives. Let M = E[xxᵀ] be their uncentred second-moment matrix. Let Q weight output errors according to the decision loss. Then expected squared operational error is:

This equation yields a useful distinction. Algebraic rank counts non-zero singular channels. Spectral concentration describes how quickly their gains decay. Task-conditioned action rank is the smallest rank that keeps operational risk below a declared tolerance on the distributions and losses that matter. It is not a new decomposition theorem. It is a decision definition that exposes the assumptions hidden by a global norm.

The sample version is straightforward. Feed a calibration matrix X through both W and its approximation, then compare the outputs. If rows of X reproduce the deployment mix, the measured error estimates input-weighted action. If a separate critical set is scored with larger output penalties, it estimates loss-weighted action. The method is simple enough to tempt overconfidence: the estimate is only as representative as the samples and weights that created it.

Why activation-aware compression exists

Neural-network weights need not be globally low rank for their realised action to be low dimensional. Hidden activations can occupy a narrow region of the input space. A weight-only SVD spends rank preserving directions that the model rarely visits, while discarding directions that become large after activation scaling.

DRONE formalised a data-aware decomposition for NLP models by exploiting low-dimensional layer representations, even when the weight matrices themselves were not low rank.[6] Later activation-aware and truncation-aware SVD methods for large language models similarly calibrated decomposition to observed activations and layer sensitivity.[7][8] These results support a limited conclusion: the right object to compress may be the matrix as exercised by data, not the matrix in isolation.

Adaptation is a low-rank hypothesis

Compression asks whether an existing transformation can lose directions. Adaptation asks whether the change required by a new task can be expressed through a few new directions.

Same algebra, opposite counterfactuals

Compression and adaptation can both end with two thin factors, but they test opposite interventions. Compression begins with a known transformation W and asks: if some of its channels are removed, does its useful action survive? Adaptation begins with a frozen W and asks: if change is permitted only through a rank-bounded path, can the new behaviour be reached? The first is a deletion test. The second is a reachability test.

The distinction changes the evidence required. A compression claim compares W with an approximation Ŵ under a declared input distribution and loss. An adaptation claim compares the configured system W + BA with a target behaviour that may have no known dense update. A small reconstruction error for W therefore says nothing by itself about whether a new task is reachable through BA. A successful low-rank update says nothing by itself about whether W was low rank.

The shared factorisation does not make compression and adaptation the same claim. In deployment, record which counterfactual is under test. Otherwise a spectral property of the base model can be mistaken for evidence about task reachability, or a successful adapter can be mistaken for evidence that the underlying transformation was redundant.

Suppose a pretrained layer uses weight matrix W. Full fine-tuning replaces it with W + ΔW, allowing every entry of ΔW to move. Low-rank adaptation instead writes the update as BA, where A compresses the input into an r-dimensional bottleneck and B expands it back:

For a square 4,096 by 4,096 matrix, full fine-tuning exposes 16,777,216 trainable weights. A rank-eight adapter trains 4,096 × 8 + 8 × 4,096 = 65,536 weights, 256 times fewer for that matrix. Optimiser-state and gradient memory fall with the trainable parameter count. The adapter can also be merged into the base weight for inference.

The saving does not apply uniformly to the whole model. It depends on which projections receive adapters, whether biases or normalisation parameters are trained, and how optimiser states are stored. Rank eight on query and value projections is a different intervention from rank eight on every linear layer. Any comparison must therefore record the configured target modules, scaling, initialisation and trainable parameter count, not just the label “LoRA”.

That economy is real. The explanation is often overstated. LoRA does not prove that the model, the task or intelligence itself is low dimensional. It imposes a low-rank structure on selected weight updates. The empirical question is whether those constrained updates reach a sufficiently good function for the intended task.

Three different kinds of “small”

Fine-tuning discussions often merge three notions that should remain separate.

Aghajanyan and colleagues found that some language-model fine-tuning problems could be solved effectively in low-dimensional random subspaces of the full parameter space.[9] That supports the claim that useful solutions may be reachable with far fewer degrees of freedom than the model has parameters. It does not imply that the update to each weight matrix has the same low rank.

The original LoRA study reported strong results across several language models while training low-rank updates.[10] Theoretical work later showed that adapters distributed across depth can express more than a single rank-r matrix would suggest, with rank requirements depending on architecture and target function.[11] Composition matters: local bottlenecks do not translate into a simple global rank bound on the nonlinear network.

Rank is capacity, not optimisation

Even when rank is sufficient in principle, training can fail to find the useful factors. BA is bilinear: changing A changes what the bottleneck listens for, while changing B changes how that signal is written back. LoRA+ argued that using the same learning rate for both factors can be suboptimal at large width, and proposed different rates.[13] DoRA separated weight magnitude from direction and used LoRA for directional updates, reporting gains over standard LoRA in its evaluated settings.[14]

These are rival explanations for a weak adapter. The rank may be too low. The adapter may be attached to the wrong matrices. The optimiser may learn one factor faster than the other. The data may not excite the required directions. Or the target change may require broad modification rather than a narrow correction. A failed rank-eight run is not evidence that low-rank adaptation is impossible; a successful one is not evidence that rank eight is intrinsically correct.

Rank allocation across depth adds another choice. Giving every layer the same rank assumes equal need for adaptation. Yet early layers may already provide adequate features while a few middle or late layers carry most of the task-specific change. A fixed total budget can therefore be distributed uniformly, by measured sensitivity or by learned allocation. The budget is not only “how much rank”, but “where may new directions enter the computation?”

When two low-rank updates meet

Adapters are often trained separately and later combined. Algebraically, the sum of a rank-r₁ update and a rank-r₂ update has rank at most r₁ + r₂. If their active subspaces overlap, the merged rank can be smaller. If they push the same direction in opposite ways, they can cancel. If they write into different directions that downstream layers entangle, they can interfere even though each adapter works alone.

This creates a second rank question: not only whether each task has a narrow update, but whether several task updates share a compatible subspace. A common adapter bank is economical when tasks reuse directions. It becomes a bottleneck when task-specific directions compete for the same narrow channel. The revealing experiment is to compare separate performance, naive summation, joint training and a higher-rank merge while holding the base model fixed.

A broad empirical comparison found that standard LoRA settings underperformed full fine-tuning on substantial code and mathematics adaptation, while preserving more out-of-domain capability. The authors also reported that full fine-tuning updates had much higher rank than typical LoRA configurations.[12] The finding is useful precisely because it resists a simple verdict. A narrow update can be a feature when preservation matters and a defect when the new domain demands broad change.

stateDiagram-v2
  [*] --> Claim
  state "SVD finds the privileged axes" as Claim
  state "The hidden assumption,<br/>isotropic use" as Test
  state "Why activation-aware<br/>compression exists" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Measure the directions the task actually uses

The practical question is not “What rank is fashionable?” It is “At what rank do the behaviours and invariants I care about stop changing?”

A task-conditioned rank assay

The assay below combines three views. Weight-space SVD measures global spectral concentration. Activation tests measure error on realised inputs. Critical-slice tests apply explicit loss to rare or consequential cases. A rank is acceptable only when all declared gates pass.

Before running the sweep, define the gates numerically. A useful gate packet states the ordinary error tolerance, the maximum critical-slice regression, the acceptable forgetting budget and the minimum improvement that justifies another unit of rank. Without predeclared gates, teams tend to stop where the chart looks pleasing. The assay then becomes a narrative device rather than a decision instrument.

Repeat the sweep across seeds and at least one plausible shift. Low-rank factors can rotate or exchange scale while representing similar products, so factor-level similarity is not a reliable stability test. Compare outputs, task metrics and the subspace spanned by the product update. Preserve the base-model version, data hash, target modules, optimiser settings and evaluation slices as the receipt for the rank decision.

Read the shape of the rank curve

A rank sweep is more informative than a winning rank. An early plateau suggests that the tested action concentrates sharply. A long gradual slope suggests diffuse change or several weak directions. A cliff at one rank often means a particular distinction or threshold has finally been restored. A split curve, where ordinary accuracy plateaus but critical error keeps falling, reveals that aggregate metrics are hiding a separate operational requirement.

These shapes guide the next intervention. For an early plateau, verify shifts and stop. For a gradual slope, compare broader adaptation or move adapters to different layers. For a cliff, trace the examples that cross it and test whether one causal direction explains the change. For a split curve, keep separate gates rather than averaging the metrics back together. The curve is evidence about where capacity is missing, rather than only a recipe for choosing a larger number.

The smallest useful implementation needs only one candidate matrix, two input slices and three ranks. It does not need a frontier model. Start with a synthetic matrix whose spectrum you control, reproduce the failure in the browser lab, then replace synthetic inputs with authorised activations. This progression separates coding mistakes from modelling mistakes before expensive training begins.

Runnable Python assay

The browser lab shows the smallest counterexample. The script below scales the same logic to any matrix and synthetic or authorised activation samples. It reports spectral energy, ordinary output error, critical-slice error and loss-weighted error for a rank sweep.

The implementation calls NumPy’s SVD routine, which uses established numerical decomposition machinery exposed through LAPACK.[5] Modern practice builds on the bidiagonalisation route developed by Golub and Kahan.[3] Numerical provenance makes the factorisation reproducible. It does not make the operational loss appropriate.

For very large matrices, a full decomposition can itself be expensive. Partial and randomised SVD methods estimate the leading subspace without materialising every singular vector, often by probing the matrix with random directions and refining the captured range.[4] That accelerates the proposal stage. It does not remove the need for end-to-end validation, because an approximate leading subspace can still miss a low-energy critical direction.

from __future__ import annotations

from dataclasses import dataclass
import numpy as np


@dataclass(frozen=True)
class RankResult:
    rank: int
    spectral_energy: float
    ordinary_nrmse: float
    critical_nrmse: float
    weighted_nrmse: float


def truncated_svd(weight: np.ndarray, rank: int) -> np.ndarray:
    """Best rank-r approximation in the unweighted Frobenius norm."""
    matrix = np.asarray(weight, dtype=float)
    if matrix.ndim != 2 or not np.all(np.isfinite(matrix)):
        raise ValueError("weight must be a finite matrix")
    if not 0 <= rank <= min(matrix.shape):
        raise ValueError("rank is outside the valid range")
    u, singular, vh = np.linalg.svd(matrix, full_matrices=False)
    return (u[:, :rank] * singular[:rank]) @ vh[:rank, :]


def normalised_output_error(
    weight: np.ndarray,
    approx: np.ndarray,
    inputs: np.ndarray,
    *,
    output_weights: np.ndarray | None = None,
    sample_weights: np.ndarray | None = None,
) -> float:
    """Root weighted squared error, normalised by full-output energy."""
    matrix = np.asarray(weight, dtype=float)
    estimate = np.asarray(approx, dtype=float)
    samples = np.asarray(inputs, dtype=float)
    if estimate.shape != matrix.shape:
        raise ValueError("approx must have the same shape as weight")
    if samples.ndim != 2 or samples.shape[1] != matrix.shape[1]:
        raise ValueError("inputs must have shape [samples, input_dimension]")

    q = np.ones(matrix.shape[0]) if output_weights is None else np.asarray(output_weights, dtype=float)
    s = np.ones(samples.shape[0]) if sample_weights is None else np.asarray(sample_weights, dtype=float)
    if q.shape != (matrix.shape[0],) or s.shape != (samples.shape[0],):
        raise ValueError("weight vectors do not match output or sample dimensions")
    if np.any(q < 0) or np.any(s < 0) or not np.all(np.isfinite(q)) or not np.all(np.isfinite(s)):
        raise ValueError("loss weights must be finite and non-negative")

    full = samples @ matrix.T
    residual = samples @ (matrix - estimate).T
    loss_scale = s[:, None] * q[None, :]
    numerator = float(np.sum((residual**2) * loss_scale))
    denominator = float(np.sum((full**2) * loss_scale))
    if denominator <= 1e-12:
        raise ValueError("normalisation is undefined because full output energy is zero")
    return float(np.sqrt(numerator / denominator))


def rank_sweep(
    weight: np.ndarray,
    ordinary_inputs: np.ndarray,
    critical_inputs: np.ndarray,
    *,
    critical_loss_multiplier: float,
    output_weights: np.ndarray | None = None,
) -> list[RankResult]:
    """Compare spectral, ordinary, critical and decision-weighted errors."""
    if not np.isfinite(critical_loss_multiplier) or critical_loss_multiplier < 0:
        raise ValueError("critical_loss_multiplier must be finite and non-negative")

    singular = np.linalg.svd(weight, compute_uv=False)
    total_energy = float(np.sum(singular**2))
    if total_energy <= 1e-12:
        raise ValueError("rank sweep is undefined for a zero-energy matrix")

    combined = np.vstack([ordinary_inputs, critical_inputs])
    sample_weights = np.concatenate([
        np.ones(len(ordinary_inputs)),
        np.full(len(critical_inputs), critical_loss_multiplier),
    ])

    results: list[RankResult] = []
    for rank in range(1, min(weight.shape) + 1):
        approx = truncated_svd(weight, rank)
        results.append(
            RankResult(
                rank=rank,
                spectral_energy=float(np.sum(singular[:rank] ** 2) / total_energy),
                ordinary_nrmse=normalised_output_error(weight, approx, ordinary_inputs),
                critical_nrmse=normalised_output_error(weight, approx, critical_inputs),
                weighted_nrmse=normalised_output_error(
                    weight,
                    approx,
                    combined,
                    output_weights=output_weights,
                    sample_weights=sample_weights,
                ),
            )
        )
    return results


def synthetic_case(seed: int = 7) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
    """Create a matrix whose smallest direction is rare but decision-critical."""
    rng = np.random.default_rng(seed)
    dimension = 12
    left, _ = np.linalg.qr(rng.normal(size=(dimension, dimension)))
    right, _ = np.linalg.qr(rng.normal(size=(dimension, dimension)))
    singular = np.array([12, 7, 4, 2, 1.2, .8, .5, .3, .2, .12, .08, .05])
    weight = left @ np.diag(singular) @ right.T

    ordinary_coeff = rng.normal(size=(2_000, 3))
    ordinary_inputs = ordinary_coeff @ right[:, :3].T
    critical_coeff = rng.normal(size=(100, 1))
    critical_inputs = critical_coeff @ right[:, -1:].T
    return weight, ordinary_inputs, critical_inputs


if __name__ == "__main__":
    w, ordinary, critical = synthetic_case()
    results = rank_sweep(
        w,
        ordinary,
        critical,
        critical_loss_multiplier=5_000_000.0,
    )
    print("rank  energy  ordinary  critical  weighted")
    for row in results:
        print(
            f"{row.rank:>4}  {row.spectral_energy:>6.1%}  "
            f"{row.ordinary_nrmse:>8.3f}  {row.critical_nrmse:>8.3f}  "
            f"{row.weighted_nrmse:>8.3f}"
        )

Run with Python 3.10+ and NumPy. Replace the synthetic activations only when the data is authorised and representative. The script deliberately keeps ordinary and critical samples separate.

Consequence

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The direction that carried almost no energy
  M->>E: SVD finds the privileged axes
  E-->>M: compare rival explanations
  E->>D: Let behaviour choose the rank
  D-->>Q: revise scope or act

Let behaviour choose the rank

The wrong decision question is “How much of the singular-value energy can I keep?” That question is useful for a first scan, but it silently substitutes an unweighted matrix norm for the behaviour the system must preserve.

The better question is: Which input-output directions must remain active, under which inputs, with what consequence if they disappear? For compression, that means calibrating on activations, shifts and rare slices before setting a rank. For adaptation, it means treating LoRA rank, factor placement and optimiser design as hypotheses to test against full fine-tuning or broader alternatives.

A low-rank design is justified when task-conditioned error falls quickly, critical invariants remain intact, the result is stable across seeds and shifts, and the saved cost matters. It is not justified merely because the matrix is large, the spectrum is steep or a conventional rank worked elsewhere.

This changes the role of SVD. It is no longer a compression spell that declares small directions irrelevant. It becomes an instrument for proposing which directions might be removable, followed by an operational test of whether the proposal survives contact with the task.

Appendix

Glossary

Rank

The number of independent output directions a linear transformation can produce.

Singular value

The gain applied to one privileged input direction before it is written into a matched output direction.

Truncated SVD

The approximation formed by keeping only the largest singular channels.

Frobenius norm

The square root of the sum of squared matrix entries, also the square root of summed squared singular values.

Spectral norm

The largest amplification a matrix can apply to a unit input.

Intrinsic dimension

The effective number of optimisation degrees of freedom needed to reach a target performance from a specified starting point.

LoRA

A method that freezes pretrained weights and learns additive low-rank factor matrices.

Task-conditioned action rank

The smallest rank that satisfies declared error tolerances under specified inputs, losses and critical slices.

Appendix

References

1. Eckart, C. and Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika, 1, 211–218.
2. Mirsky, L. (1960). Symmetric gauge functions and unitarily invariant norms. Quarterly Journal of Mathematics, 11, 50–59.
3. Golub, G. and Kahan, W. (1965). Calculating the singular values and pseudo-inverse of a matrix. Journal of the Society for Industrial and Applied Mathematics, Series B, 2(2), 205–224.
4. Halko, N., Martinsson, P.-G. and Tropp, J. A. (2011). Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2), 217–288.
5. NumPy. numpy.linalg.svd documentation.
6. Chen, P., Yu, H.-F., Dhillon, I. S. and Hsieh, C.-J. (2021). DRONE: Data-aware low-rank compression for large NLP models. NeurIPS 34.
7. Yuan, Z., Shang, Y., Song, Y., Wu, Q., Yan, Y. and Sun, G. (2024). ASVD: Activation-aware singular value decomposition for compressing large language models.
8. Wang, X., Alam, S., Wan, Z., Shen, H. and Zhang, M. (2025). SVD-LLM V2: Optimizing singular value truncation for large language model compression. NAACL, 4287–4296.
9. Aghajanyan, A., Zettlemoyer, L. and Gupta, S. (2021). Intrinsic dimensionality explains the effectiveness of language model fine-tuning. ACL-IJCNLP, 7319–7328.
10. Hu, E. J. et al. (2022). LoRA: Low-rank adaptation of large language models. ICLR.
11. Zeng, Y. and Lee, K. (2024). The expressive power of low-rank adaptation. ICLR.
12. Biderman, D. et al. (2024). LoRA learns less and forgets less. Transactions on Machine Learning Research.
13. Hayou, S., Ghosh, N. and Yu, B. (2024). LoRA+: Efficient low-rank adaptation of large models. ICML, 17783–17806.
14. Liu, S.-Y. et al. (2024). DoRA: Weight-decomposed low-rank adaptation. ICML, 32100–32121.