Matrices as Machines

Part IRead the machine from its columns

A table of numbers encourages an entry-by-entry reading. That is useful for bookkeeping and poor for intuition. Take a matrix with two columns, a₁ and a₂. The first column is where the horizontal unit direction goes. The second is where the vertical unit direction goes. Once those two destinations are known, linearity fixes the destination of every point in the plane.

Consider the small operator

This minimal example contains the whole mechanism. The unit square becomes the parallelogram spanned by the two columns. Parallel lines remain parallel because the same weighted-sum rule applies everywhere. The origin stays fixed because a linear map sends the zero combination to zero. Straight lines stay straight, although their lengths and angles may change.

Rows ask questions; columns route causes

For an m by n matrix, the dimensions are an interface contract. The machine accepts n input coordinates and emits m output coordinates. Each of its n columns describes the output caused by one unit input. Each of its m rows describes one weighted question asked of the whole input. In the sensor average, the single row asks, “what is the common level?” It never asks, “do the probes disagree?” The missing question is the missing evidence.

This row view is useful whenever a matrix represents measurements rather than motion. A camera pixel, a portfolio factor, a neural feature and a linear sensor channel are all weighted summaries. Two rows that ask nearly the same question add little independent evidence. A row aligned with a required contrast can make that contrast visible even when the original coordinates are inconvenient. The same numbers therefore admit two complementary readings: columns reveal how input causes propagate; rows reveal what the output is capable of observing.

The distinction also clarifies rectangular matrices. A wide matrix with fewer rows than columns must compress an input into fewer reported coordinates. It can still preserve a particular finite set of task contrasts, but it cannot preserve every direction of the full continuous input space. A tall matrix can add measurements, yet duplicated or dependent rows do not create new information. Output width alone is not evidence of independent observation. Rank records the number of genuinely independent directions, while the rows and columns explain where those directions come from.

Matrices become systems through composition. If A transforms raw inputs and B transforms the result, the combined machine is BA. The order matters: BA means “apply A, then B”, and it is generally different from AB. Dimensions enforce the wiring. An output type from one stage must match the input type of the next. This makes matrix multiplication less mysterious: it is the algebra of connecting linear machines.

An affine layer adds a bias, y=Ax+b. Bias moves the whole output cloud but cancels when two cases are compared: (Ax+b)-(Ax′+b)=A(x-x′). It can change whether a later threshold fires, but it cannot by itself restore a distinction already erased by A. This is why difference vectors are the right probe for the present argument. They remove a common offset and expose the part of the layer responsible for preserving separability.

The rule depends on the chosen coordinates, but the underlying transformation does not. Change the input basis and the column values change because the machine is being described with different handles. Change the output basis and the coordinate labels change again. This distinction prevents a common mistake: treating a large coefficient as an intrinsic effect when it may partly reflect units or basis choice.

Linearity is the constraint that makes this compression possible:

That constraint is powerful and severe. A linear map cannot create a bend, a threshold or a context-dependent rule by itself. Neural networks gain non-linear behaviour by alternating matrix operations with non-linear activations. PyTorch’s official definition of a linear layer makes the affine rule explicit as a matrix transformation plus a bias.¹¹ The matrix still controls which local combinations and directions are available to the activation that follows.

Reading columns answers “where do the basis directions go?” It does not yet answer which directions are stretched most, whether any are nearly lost, or how stable an inverse will be. Those questions require a basis-independent view.

Part IIThe geometry hidden by the entries

Imagine three rooms with flexible floors. In the first, the floor rotates by 30 degrees. In the second, it stretches tenfold east-west and shrinks tenfold north-south. In the third, each northward step also slides east. All three can preserve the area of every painted tile. They do not preserve the same geometry.

This thought experiment varies one causal feature while holding another fixed. Each operator below has determinant 1. The rotation preserves lengths and angles. The reciprocal stretch turns circles into thin ellipses and magnifies measurement error along one axis during inversion. The shear preserves area yet changes angles and couples coordinates. A determinant is a volume summary, not a complete account of distortion.

For a square matrix, the absolute determinant gives the factor by which area or volume changes.⁴ A zero determinant proves that some dimension collapsed. A non-zero determinant proves exact invertibility in exact arithmetic. It says little about whether inversion is sensible under finite precision or noise. A determinant of one can hide a factor of a million in one direction and a factor of one millionth in another.

The right geometric instrument is the singular value decomposition:

Think of a unit circle painted on elastic material. The first orthogonal stage turns the material without changing its shape. The diagonal stage stretches the selected axes. The final orthogonal stage turns the resulting ellipse. The ellipse’s longest radius is the largest singular value, σ_max; the shortest surviving radius is σ_min. Every other unit direction lands between those extremes.

The SVD gives a clean answer to the room experiment. Rotation and the orthogonal mixer have singular values all equal to one. They can look busy in coordinates while preserving every distance. The reciprocal stretch has singular values far apart. It preserves area because their product is one, yet makes one direction fragile. The shear is a combination of rotations and unequal stretches even though its diagonal entries are both one.

The determinant hides where the volume went

For a square full-rank matrix, the absolute determinant equals the product of its singular values. That identity explains both the usefulness and the blindness of determinant. It reports the total volume multiplier, while forgetting how the multiplier was distributed among directions. Singular values (10, 0.1) and (1, 1) both multiply area by one. The first machine creates an elongated, fragile ellipse; the second leaves a circle unchanged. A product cannot tell us whether one factor became dangerously small because another became equally large.

The same warning applies to summaries such as average column norm or average activation size. Averages can remain stable while a narrow direction collapses. In a learning system, common examples may occupy the strong directions and dominate the metric, while a rare boundary case lies close to a weak singular direction. The layer then looks healthy under aggregate statistics and fails precisely where evidence is scarce. The geometry must be inspected by direction, not only by total energy.

Singular values are invariant under orthogonal changes of coordinates, which makes them less dependent on how axes are named. They are not invariant to arbitrary rescaling of features. Measuring income in pounds rather than thousands of pounds changes the numerical geometry unless preprocessing compensates. Before interpreting a small singular value as a property of the world, ask whether it is a property of the chosen units. A useful audit keeps the engineering-unit transformation and the model-space transformation together rather than reporting one detached spectrum.

Composition sharpens the risk. For compatible matrices, the strongest possible amplification of the product cannot exceed the product of the strongest amplifications of its stages. More importantly, a direction that becomes exactly zero at any stage stays zero through every ordinary downstream function that receives only that representation. Near-collapse can also accumulate. Several moderate squeezes aligned across layers can produce a severe end-to-end squeeze even when no single layer appears extreme. Conversely, rotations between squeezes can move the weak direction and change which input contrasts are affected.

This is why a layer spectrum is diagnostic rather than dispositive. It can locate a bottleneck and suggest a mechanism, but the end-to-end operator on a specific contrast still matters. In a purely linear stack it is the matrix product. In a non-linear network it is approximated locally by a Jacobian, and the approximation is valid only around the cases and perturbations used to form it. The SVD supplies the machine’s directional anatomy; an intervention supplies the semantic diagnosis.

This is the serious negative control for the claim that “mixing destroys information”. Let

Coordinate mixing and information erasure are different mechanisms. Mixing asks where a distinction is represented. Erasure asks whether any non-zero distinction maps to zero, or becomes too small to survive the operating noise. The next part separates those cases.

Part IIIWhen information disappears

Return to the two probes. Their disagreement is the vector d = (1, −1)^T. The averaging matrix sends d to zero. Therefore every pair of readings that differs only by some multiple of d has the same average. The set of all such invisible directions is the null space:

Erasure is an impossibility result, not a weak prediction

Suppose two possible states are x and x′, and the matrix produces the same representation for both: Ax=Ax′. Let g be any downstream procedure that receives only that representation. It may be a rule, a classifier, a language model or a human-facing dashboard. Because its inputs are identical, g(Ax)=g(Ax′). The downstream procedure may guess from prior frequencies, but it cannot base the distinction on evidence that the matrix removed.

This proof isolates the causal burden. Poor downstream performance is sometimes blamed on model capacity, training data or decision policy. Those explanations remain possible when the representations differ. They become insufficient when the relevant cases have collided exactly. The repair must alter the measurement, retain another channel, or provide independent evidence. More computation after the collision changes neither the equality nor the missing information.

The column space gives the companion boundary on outputs. It contains every vector the machine can produce. If a requested output lies outside that space, the equation Ax=b has no exact solution. Least squares can choose the nearest attainable output, but closeness is an optimisation criterion, not proof that the missing component was observed. The null space tells which input variations are invisible; the column space tells which output claims are reachable. Together they define the evidential envelope of the linear machine.³

Near-collision needs a quantitative version of the same argument. Let the observed output be y=Ax+e, where e is perturbation from measurement noise, rounding or upstream error. If two states differ by d, their noiseless outputs differ by Ad. When that separation is no larger than the possible difference between their perturbations, the observed output sets overlap. No deterministic decoder can then guarantee the correct state for every permitted perturbation.

Conditioning describes the global sensitivity of inversion, while task retention describes one declared direction. They answer different questions. A large condition number warns that some direction is fragile, even if the application never uses it. A small value of ‖Ad‖ for a named d warns that this application’s contrast is fragile, even when the global condition number appears acceptable. Global conditioning is a searchlight; task retention is the inspection at the actual fault line.

The rank counts how many independent output directions survive. If an n-dimensional input matrix has rank r, then n−r independent input directions lie in its null space. Software cannot decide numerical rank without a tolerance because floating-point calculations rarely return an exact zero. NumPy’s matrix_rank therefore counts singular values above a threshold tied to the largest singular value, matrix dimensions and machine precision.⁷

That numerical convention hints at a deeper operating issue. Exact rank asks whether a direction is mathematically zero. A deployed measurement pipeline needs to ask whether the direction remains large relative to noise, quantisation and the downstream decision margin. A singular value of 10⁻⁸ is non-zero. If the output is rounded to 10⁻³, the corresponding distinction has effectively vanished.

Consider a second thought experiment. A press can be set to thickness η. At η = 0 it turns a clay square into a line, so depth is exactly unrecoverable. At η = 0.01 the output is technically a very thin rectangle. In a noiseless world with infinite precision, depth can be recovered by multiplying by 100. Add surface noise of ±0.02, and the attempted inverse amplifies that noise until it dominates the recovered depth. The causal change from 0 to 0.01 restores algebraic invertibility without restoring practical evidence.

NumPy likewise computes the 2-norm condition number from the SVD and warns that an ill-conditioned inverse may be inaccurate even when no singularity error is raised.⁸⁹ The issue is not an implementation quirk. It follows from the machine’s geometry. Recovering a direction shrunk by a factor of one hundred requires amplification by one hundred, including amplification of whatever perturbation occupies the same output direction.

Exact erasure is a property of the null space; practical erasure is a relation among the matrix, the task and the noise floor. This explains why a single global rank threshold is never enough for high-consequence feature compression. The relevant tolerance comes from units, expected perturbations and the distinction the system must preserve.

There is another boundary. The smallest singular value is a worst-case measure across all directions. A system may safely crush an irrelevant direction while preserving every task-relevant one. Conversely, a matrix with a respectable global condition number can still attenuate the one semantic contrast that matters because the input was poorly scaled or the task direction lies near its weakest axis. That is why the audit must become task-specific.

Part IVAudit the distinctions that matter

Suppose a synthetic equipment-monitoring encoder receives four standardised inputs: temperature probe A, temperature probe B, vibration and pressure. Its first output is the average temperature. Its second is a scaled disagreement channel. The remaining outputs carry vibration and pressure unchanged:

For the task-relevant difference d = (1, −1, 0, 0)^T, the transformed difference is (0, η, 0, 0)^T. The average channel cannot see it. The disagreement channel carries all of it, scaled by η. If η is learned towards zero because most training rows contain agreeing sensors, a rare fault signature can disappear while average reconstruction or prediction loss remains good.

This is a worked scenario, not a reported deployment. Its purpose is to expose a common causal pattern: an aggregate metric rewards common variation, while an operationally decisive contrast lives in a low-variance direction. The matrix alone cannot tell us which contrast matters. The task definition supplies that information.

Make the contrast set part of the system specification

The four-sensor encoder should not be reviewed as an isolated array. Its specification needs a contrast register. One contrast may represent equal-and-opposite probe drift. Another may represent a one-unit pressure change at fixed temperature. A third may encode an invariance, such as adding the same calibration offset to both temperature probes when only their disagreement matters. Required distinctions should stay separated; declared nuisance variations may be attenuated or deleted.

That register turns an abstract representation claim into tests. For each contrast, record its source, plausible magnitude, units, consequence if lost, and expected behaviour. Run a positive case that must survive, a negative control that may disappear, and a scale test around the operating boundary. Test both signs because a later non-linearity can respond asymmetrically even when the matrix is symmetric. Repeat the calculation after quantisation, export and hardware conversion, since numerical changes can move a marginal direction across the threshold.

Training introduces another comparison. Evaluate the matrix before training, at checkpoints and at release. If a required retention ratio declines while average loss improves, the optimisation objective is buying common-case performance with a hidden loss of evidence. That is not automatically wrong. It is a decision that needs an owner. Possible responses include an auxiliary contrast loss, a constrained singular floor on a protected subspace, a skip connection, a wider bottleneck, or an explicit deterministic channel outside the learned compression.

The last option is often the clearest. If probe disagreement has a fixed operational meaning, compute it directly and carry it beside the learned features. A neural representation can still model complex interactions without being the sole custodian of a simple high-consequence signal. This is the same placement discipline used elsewhere in reliable systems: keep a deterministic fact in a typed channel when its semantics and calculation are stable, and use learning for the residual uncertainty.

A useful audit starts with difference vectors rather than raw examples. Each vector names a controlled contrast and its required fate: sensor disagreement that must remain observable, a class margin that must survive, a nuisance perturbation that should disappear, or a control variable that must remain recoverable. For a non-zero preservation contrast d, define its retention:

The ratio has no universal pass mark. If two output observations can each carry perturbation with Euclidean norm at most ε, their separation can be obscured by as much as 2ε. A conservative deterministic test is therefore ‖Ad‖₂ > 2ε. A probabilistic system should replace that bound with a distributional error model and a required confidence level. A classifier may instead compare the transformed distinction with its decision margin. A reversible codec may compare reconstruction error with application tolerance.

The threshold must be stated in the units of the decision. Matrix diagnostics become misleading when features with kilograms, milliseconds and percentages are combined without scaling. Standardisation may help numerical analysis, but it is not neutral: it embeds a judgement about which variation counts as comparable. Record the scaling transformation alongside the matrix and run the distinction test in both engineering units and model coordinates.

A release protocol for a learned projection

A practical review can proceed in six passes. First, confirm the matrix’s input and output semantics, including every scaling step. Second, inspect columns and rows for unintended coupling. Third, compute rank, singular values and condition estimates with a declared numerical tolerance. Fourth, run the named contrast register through the operator and compare retained separation with the applicable noise or margin model. Fifth, use an orthogonal or identity baseline to separate harm caused by compression from harm caused merely by coordinate mixing. Sixth, repeat the decisive tests end to end so that later non-linear stages cannot hide behind a healthy local diagnostic.

The baseline matters. Suppose a learned projection and an orthogonal mixer have the same output width and similar downstream latency. If both change the coordinate names but only the learned projection loses a protected contrast, compression geometry is implicated. If both pass locally and the full model fails, the cause lies later. If the identity baseline also fails, the contrast may not be present in the input evidence at all. These matched controls prevent every error from being attributed to the most visually complex matrix.

Store the results as a small decision receipt: matrix or model version, preprocessing version, contrast-set version, numerical tolerance, noise assumptions, measured retentions, negative controls, end-to-end outcomes and reviewer decision. Re-run it when any of those inputs changes. The receipt does not claim that the model is globally reliable. It establishes a bounded proposition: under the stated units and perturbations, this transformation retained these declared distinctions above these thresholds.

For a stack of learned layers, locate the first material attenuation. Let d₀ be the input contrast and d_k its representation after stage k. Plot ‖d_k‖ and, where meaningful, the decision margin attributable to that contrast. A sharp fall identifies the stage to investigate. A gradual decline points to cumulative geometry. A rebound after a non-linearity may restore separability on sampled cases, but it does not prove robust recovery under nearby perturbations, so intervention tests must accompany the trace.

The release decision should be contrast-specific. One matrix may be acceptable for forecasting aggregate load and unacceptable for detecting sensor disagreement. Another may intentionally remove identity-related variation while retaining variables required for legitimate performance. “Good representation” is therefore incomplete without a downstream use, a nuisance model and a consequence model.

There is precedent for caring about singular geometry in learning systems. Glorot and Bengio connected training difficulty to how activations, gradients and Jacobian singular values behave across layers, motivating an initialisation scheme that improved convergence in their experiments.¹² That published result concerns optimisation, not semantic retention. The shared mechanism is that repeated linear transformations can amplify some directions and suppress others. The task audit adds the missing question: which of those directions correspond to distinctions the application cannot afford to lose?

Executable matrix notebook

The notebook below is the article’s operational artefact. It accepts a 2 × 2 matrix, a task-relevant difference vector and a bounded output-noise level. It draws the transformed grid, calculates determinant, singular values, numerical rank, condition number and distinction retention, then applies the conservative separation test ‖Ad‖₂ > 2ε.

Its assumptions are explicit: real-valued two-dimensional inputs, Euclidean norms, exact matrix entries, and a worst-case bound ε on each output observation. The expected positive case is the orthogonal mixer, which preserves every difference. The negative case is the averaging collapse, which deletes (1, −1). The near-collapse preset is full rank but fails when the retained signal lies within the selected noise bound.

const trace = a*a + b*b + c*c + d*d;
const determinant = a*d - b*c;
const disc = Math.sqrt(Math.max(0, trace*trace - 4*determinant*determinant));
const sigmaMax = Math.sqrt(Math.max(0, (trace + disc) / 2));
const sigmaMin = Math.sqrt(Math.max(0, (trace - disc) / 2));

const out1 = a*dx + b*dy;
const out2 = c*dx + d*dy;
const retainedSeparation = Math.hypot(out1, out2);
const retention = retainedSeparation / Math.hypot(dx, dy);
const survives = retainedSeparation > 2*epsilon;

Failure boundary: where the operator audit stops helping

The audit does not turn every machine-learning problem into linear algebra. It characterises a declared linear or locally linear stage. A non-linear activation can restore separability after one matrix mixes coordinates, and it can also destroy a distinction that the matrix preserved. Data-dependent operations such as attention, normalisation in training mode and routing change the effective operator with context. End-to-end intervention remains necessary.

The method also depends on choosing meaningful differences. If the task-relevant contrast is unknown, misspecified or outside the observed environment, a clean audit can certify the wrong set. Singular values offer a global worst case, but they do not attach semantics to directions. Domain evidence must provide the contrasts, units, noise model and recovery cost.

Finally, a small singular value can be intentional. Compression, denoising and invariance often require deleting variation. The question is whether the deleted direction is nuisance variation under the target decision, not whether every input can be reconstructed. The mechanism ceases to help when preservation itself is not the objective or when the effective transformation is materially non-linear and context-dependent.

Compact glossary

Column space

All outputs the matrix can produce.

Null space

Input directions that map exactly to zero.

Rank

The number of independent output directions that survive.

Determinant

Signed area or volume scale for a square matrix.

Singular value

A principal directional stretch in the SVD.

Condition number

The ratio between greatest and least surviving stretch.

Orthogonal matrix

A rotation or reflection that preserves Euclidean lengths.

Distinction retention

The task-specific ratio ‖Ad‖/‖d‖ for a chosen contrast.

Choose matrices by the distinctions they keep

A matrix is often introduced as a rectangular arrangement of numbers because that is easy to write down. It is more useful to meet it as a machine. Its columns route basis directions. Its determinant summarises volume. Its singular values expose directional gain. Its null space records exact oblivion. Its condition number estimates the price of trying to reverse a severe squeeze.

The central decision follows. Do not accept a transformation because it has full rank, a respectable determinant or good average loss. Name the differences the downstream task must still recognise. Push those vectors through the matrix. Compare their retained separation with noise, quantisation, margin and recovery cost. Keep the audit with the preprocessing, units and matrix version so that a later change can be tested against the same obligations.

This changes model design in a concrete way. An encoder that averages two channels must retain a separate disagreement channel when disagreement carries fault evidence. A learned projection that suppresses a rare but consequential contrast needs a constraint, an auxiliary loss or a wider representation. An inverse problem with a large condition number should seek new measurements rather than pretend that algebra can manufacture missing evidence.

The grid picture is therefore more than a teaching aid. It is a causal diagram of what the system can still distinguish. Rotation relocates evidence. Stretch changes its scale. Mixing changes its coordinates. Collapse removes it. The matrix earns its place only when the distinctions that matter survive the trip.