Prediction and interpretation pull in different directions
Ordinary least squares chooses coefficients to minimise the residual sum of squares. Its estimates can have low bias but large variance, especially when predictors are numerous or correlated. A fitted model may track the training data closely and still predict new observations poorly. Shrinking coefficients towards zero can trade some bias for lower variance and improve expected prediction error. The practical question is not simply whether to fit, but how to control the instability introduced by fitting.
Two established responses occupied opposite corners of the design space. Best-subset selection retained or dropped variables, producing a compact model that could be read and discussed. Yet the discrete decision made it unstable: a small data perturbation could swap one subset for another. Ridge regression replaced selection with a continuous L2 constraint. It shrank coefficients smoothly and often improved stability, but generally left every coefficient non-zero. The result could be harder to interpret when the predictor list was long.
Tibshirani proposed the least absolute shrinkage and selection operator: Lasso: as a deliberate hybrid. It keeps the continuous least-squares objective but constrains the L1 size of the coefficient vector. As the budget t falls from the total absolute size of the least-squares estimate towards zero, coefficients shrink; some become exactly zero. The method therefore tries to inherit subset selection’s readable sparsity and ridge regression’s continuous regularisation.
subject to Σⱼ|βⱼ| ≤ t
intercept α is not charged to the coefficient budget
The paper standardises predictors and treats the intercept separately. This is essential rather than cosmetic. A constraint on coefficient magnitude depends on the units of each predictor. Without standardisation, measuring one variable in metres rather than centimetres changes its numerical coefficient and therefore changes how much budget it consumes. Excluding the intercept prevents the mean level of the response from competing with predictor effects for L1 capacity.
Worked example 1 · Units can spend the budget unfairly
Suppose height is recorded once in metres and once in centimetres. The physical relationship is unchanged, but the coefficient measured per metre is one hundred times the coefficient measured per centimetre. An unstandardised L1 budget penalises those two encodings differently. Standardising each predictor to a common scale before fitting makes the budget compare effects on a shared numerical basis.
Probe 1 · Why can a little bias improve prediction?
Least squares uses sample fluctuations when it estimates every coefficient without restraint. A constraint pulls estimates towards a smaller region, introducing bias but reducing how far they move when the sample changes. If the variance reduction exceeds the squared-bias increase, expected prediction error falls. The causal chain is constraint → reduced coefficient sensitivity → lower prediction variance, not constraint → truth.
A diamond gives the loss somewhere sharp to land
For two coefficients, the Lasso constraint |β₁|+|β₂|≤t is a diamond. Lines of constant residual sum of squares form ellipses centred at the unconstrained least-squares estimate. The constrained solution is the first point where an expanding ellipse meets the diamond. Because the diamond has corners on the coordinate axes, contact often occurs where β₁ or β₂ equals zero. Ridge replaces the diamond with a circular or elliptical L2 region. Its smooth boundary provides no equally prominent axis corners, so contact generally leaves both coordinates non-zero.
This picture explains a mechanism, not a guarantee that the correct scientific variables have been selected. The axes come from the supplied coordinate system. Rotate a correlated predictor space and the corners refer to different linear combinations. L1 sparsity therefore depends on representation. A zero means that the optimum of this loss, scaling and budget lies on an axis; it does not establish that the corresponding phenomenon has no causal role.
Orthonormal design turns geometry into a formula
When the predictor columns are orthonormal, the least-squares coefficients decouple. The Lasso solution soft-thresholds each coefficient: subtract a shared threshold from its magnitude, keep its sign, and truncate at zero. Best-subset selection hard-thresholds instead: it keeps a coefficient unchanged once it clears a cut-off and deletes it otherwise. Ridge multiplies every coefficient by a smooth factor and does not create zeros. The non-negative garotte shrinks relative to the original estimate and inherits its sign.
(u)⁺ = max(u,0)
one threshold γ is chosen so the absolute-value budget is met
Soft thresholding joins continuity and deletion. A coefficient just above γ emerges small; a coefficient just below γ is zero. This avoids the full jump of hard selection, although the derivative changes at the threshold. Large coefficients are still biased towards zero by γ. The method does not grant important predictors immunity from shrinkage; it deliberately pays that price for variance control.
Worked example 2 · Three orthogonal coefficients
Let the least-squares coefficients be 3.0, −1.2 and 0.4, and let γ=0.8. Soft thresholding returns 2.2, −0.4 and 0. The signs of surviving coefficients are preserved, every surviving magnitude loses 0.8, and the smallest variable disappears. A hard threshold at 0.8 would return 3.0, −1.2 and 0, retaining the full noisy magnitudes of the survivors.
Probe 2 · Why does a corner create selection?
The loss contour moves outward from its unconstrained minimum until feasibility stops it. A smooth boundary offers a continuum of sloped contact points. A corner concentrates many supporting directions at one axis location, so a range of loss gradients share the same optimum there. Geometry therefore maps many nearby data configurations to a coefficient equal to zero. Change the constraint shape or rotate the coordinates and that mapping changes.
The budget reveals an ordered sequence of models
Tibshirani indexes the path by a normalised budget s=t/Σ|β̂ⱼ OLS|. At s=1 the least-squares solution is available; at s=0 all penalised coefficients vanish. Between those endpoints, coefficient curves show which variables enter, leave, change sign or exchange influence as the constraint relaxes. A single selected model is one vertical slice through this larger object.
Paths make trade-offs inspectable. A variable that enters early persists across a broad budget range in that sample. A variable appearing only near the least-squares end depends on a generous allowance. Yet persistence is not a causal ranking and need not be stable under resampling. Correlated predictors can substitute for one another, so one path may give weight to the first and another path to the second.
Worked example 3 · One selected model hides nearby alternatives
Suppose cross-validation selects s=0.45, where four predictors are non-zero. At s=0.42 only three survive; at s=0.50 a fifth enters with a small coefficient. Reporting only the four-variable result makes the membership appear categorical. Reporting the nearby path reveals that two decisions are fragile while three are stable across the tested neighbourhood.
Probe 3 · Does entering early mean being important?
Early entry means that a predictor reduces residual loss efficiently under the chosen scaling while little L1 budget is available. It can also stand in for a correlated group or exploit sample-specific noise. Importance for prediction, stability under resampling and causal relevance are different claims. The path directly supports only the first, conditional on the data and representation.
Predictors can compete for the same explanatory work
The orthonormal formula is illuminating because it removes interaction among predictors, but most data are not orthogonal. With correlated columns, changing one coefficient changes the residual relationship seen by another. The Lasso solution must be found jointly. In a two-predictor analysis, Tibshirani shows that the form of the solution depends on the correlation and on where the least-squares estimate lies relative to the L1 region.
Ridge tends to move correlated coefficients towards one another because its squared-norm penalty spreads weight smoothly. The Lasso can concentrate weight, set one coefficient to zero, or even produce a sign different from the corresponding least-squares estimate. That possibility is not an algebraic defect. It follows from solving a coupled constrained problem. It is also a warning against interpreting each coefficient as if all other paths were fixed independently.
Highly correlated predictors create substitutability. If two columns carry nearly the same information, several coefficient allocations make similar predictions. A small data perturbation can redirect the L1 budget from one column to its neighbour without much change in fitted values. Sparsity can therefore make the displayed equation look decisive precisely when variable identity is uncertain.
Worked example 4 · Two thermometers, one temperature
Two sensors record almost the same room temperature with small independent noise. Either can predict energy use. Under a tight L1 budget, the Lasso may place nearly all temperature weight on sensor A and set sensor B to zero. In a bootstrap sample, B may replace A. Prediction remains similar because the columns substitute, but the selected-variable story changes. Treating A as uniquely causal would exceed the evidence.
Probe 4 · Why can sparsity reduce interpretive honesty?
A short equation looks decisive, but correlation can make many short equations predict almost equally well. The L1 corner forces one allocation to be displayed, suppressing nearby alternatives. If interpretation concerns variable identity, resampled paths or grouped summaries are needed. Sparsity improves legibility; it does not automatically quantify ambiguity.
The budget is estimated, not revealed
A Lasso fit is incomplete until its constraint level has been chosen. The paper discusses three approaches. Fivefold cross-validation repeatedly fits on four-fifths of the data and predicts the held-out fifth. The budget with the smallest aggregated prediction error is selected. Generalised cross-validation approximates this process using residual sum of squares and an estimate of effective parameters. A Stein unbiased-risk route is derived for appropriate normal-mean and fixed-design settings.
These procedures answer related but non-identical questions. Cross-validation estimates performance under a particular resampling scheme. Generalised cross-validation uses a local ridge approximation and a trace expression for effective dimension. The Stein estimate relies on distributional conditions and, in the orthonormal derivation, counts coefficients whose least-squares magnitudes clear the threshold. Each route introduces assumptions beyond the L1 optimisation itself.
The cross-validation grid is part of the evidence. If it never includes a sufficiently strong penalty, a null or very sparse model cannot win. If folds are formed carelessly in time, groups or repeated measurements, leakage makes estimated error optimistic. Tibshirani notes that resampling is computationally demanding because the programme must be solved repeatedly; later algorithmic advances changed that cost but should not be retrofitted into 1996.
GCV(t) = RSS(t) / [N(1 − p(t)/N)²]
selected budget = tested value minimising the chosen estimate
Worked example 5 · A minimum with a broad floor
Suppose five tested budgets have cross-validation errors 1.12, 0.94, 0.91, 0.92 and 1.08. The formal minimum is the middle value, but the two neighbours are practically close and may select different variables. Reporting the error curve and active counts reveals the trade-off. Reporting only the winning budget makes a noisy comparison look exact.
Probe 5 · What does cross-validation cause?
Cross-validation does not improve an already fixed coefficient vector. It selects which vector from a path will be used. Fold assignments change estimated prediction errors; those estimates change the selected budget; the budget changes shrinkage and membership. This selection chain can be studied by repeating folds, but it does not prove that the surviving variables are the true data-generating causes.
Standard errors after a non-smooth fit
The Lasso estimate is nonlinear and non-differentiable at zero, making ordinary standard-error calculations difficult. The paper constructs a local ridge approximation using weights related to current coefficient magnitudes and derives an approximate covariance matrix. It also uses bootstrap calculations in the prostate example. Both approaches expose a central tension: selection creates boundaries at which small perturbations can change model dimension.
The paper explicitly notes difficulty near coefficients equal to zero. A fixed active set permits a smooth local approximation; crossing a selection boundary does not. A standard error computed conditional on one selected configuration can miss the additional uncertainty of having selected that configuration. Later selective-inference work addresses related issues, but it is not part of this primary and is not smuggled into the historical result.
The penalty can be read as a prior, but only at the mode
The constrained problem has a penalised form: minimise residual sum of squares plus λΣ|βⱼ|. Under a Gaussian likelihood and independent double-exponential priors on coefficients, that objective is proportional to the negative log posterior. The Lasso estimate is therefore the posterior mode for this model. Ridge corresponds to independent normal priors and a quadratic penalty.
The double-exponential density has a sharp peak at zero and heavier tails than a normal distribution with comparable central scale. The peak supports strong concentration near zero; the tails allow occasional larger coefficients. Yet a continuous prior assigns probability zero to a coefficient being exactly zero. Exact zero arises in the posterior mode because the log density has a kink, not because the full posterior contains a discrete spike at zero.
Worked example 6 · Mode is not mean
For a weakly identified coefficient, the posterior density under a double-exponential prior can peak at zero while remaining spread across positive and negative values. The mode reports zero, but the posterior mean may be non-zero and the interval wide. Calling the Lasso estimate a Bayesian posterior mode is correct; calling it the entire Bayesian answer is not.
Probe 6 · What does the prior interpretation add?
It reveals which belief geometry corresponds to the penalty and permits comparison with ridge’s Gaussian prior. It does not validate independence across coefficients or the chosen scale. Those are modelling assumptions. The equivalence concerns an optimiser: the posterior mode: so uncertainty summaries require more than solving the Lasso objective.
A convex problem still needs an algorithm
The Lasso objective is convex but non-differentiable where coefficients cross zero. Tibshirani expresses the absolute-value constraint as linear inequalities over the 2ᵖ possible sign patterns and notes that a direct formulation is impractical. The article then outlines a sequential procedure that searches feasible active constraints. It begins from the signs of the ordinary least-squares estimate, solves a constrained least-squares problem for the current set, checks the budget and inequality conditions, and adds a violated constraint when necessary.
The finite active-set logic is significant. A coefficient sign pattern converts the L1 boundary locally into a linear constraint. The procedure need not enumerate every sign pattern in advance; it discovers relevant ones through violations. The paper reports average iteration counts around 0.59 to 0.75 in its simulation examples and notes that a fully different quadratic-programming formulation can be faster, though it doubles variables.
This is not the later least-angle regression path algorithm, and it is not modern cyclic coordinate descent. Those later algorithms exploit additional structure and changed what became practical. A historically faithful companion can implement coordinate descent as a transparent teaching artefact, but must label it as later computational machinery applied to the 1996 objective.
Worked example 7 · Coordinate descent as a later teaching solver
Hold all coefficients but βⱼ fixed and form the partial residual excluding predictor j. The one-coordinate minimiser is a soft-thresholded residual correlation divided by the predictor’s squared norm. Cycling through columns repeatedly decreases the convex objective until updates become negligible. This is the mechanism used in the laboratory below. It solves the Lasso objective but is not claimed as the article’s own algorithm.
The comparison changes with the signal regime
The prostate-cancer example relates log prostate-specific antigen to eight standardised predictors. The paper traces the Lasso path, estimates a budget by cross-validation and compares coefficients with least squares and best subset. The selected Lasso model shrinks all coefficients and sets two to zero. Bootstrap standard errors are reported alongside a ridge-based approximation. The example illustrates a workflow; it does not validate the predictors as clinical causes.
The simulations are more revealing because their regimes are deliberately varied. In the first, a small number of large effects produces a setting in which subset selection does best, the non-negative garotte is slightly better than the Lasso, and ridge performs poorly. In the second, a small or moderate number of medium effects favours ridge by a good margin, with Lasso the only other method to outperform the full least-squares estimate. In the third, many small effects make ridge clearly best and subset selection poor. In the fourth, a larger problem again puts ridge first, with Lasso second.
The paper’s conclusion is disciplined: there is no single method that dominates. It summarises three scenarios. With a small number of large effects, subset selection does best, while Lasso is quite good and ridge poor. With a small to moderate number of moderate effects, Lasso performs best, followed by ridge and then subset selection. With a large number of small effects, ridge performs best by a good margin, followed by Lasso and then subset selection.
This pattern is exactly what the methods’ geometries suggest. Hard selection is efficient when the truth is genuinely concentrated and signals are strong enough to survive discrete decisions. Dense L2 shrinkage is well suited when many small contributions should be pooled. L1 regularisation occupies the middle ground. These are tendencies under the studied designs, not a rule for every dataset.
Worked example 8 · Why dense weak signal defeats selection
Imagine twenty predictors each contributing 0.15 in the same direction. Deleting most of them loses a large total signal even though every individual effect looks weak. Ridge can retain and stabilise all twenty. A tight L1 budget may set many to zero and introduce substantial bias. Conversely, if only two predictors have effects of 3 and the rest are null, a compact selector can avoid spending variance on eighteen noise coordinates.
Probe 7 · What does a simulation winner establish?
It establishes performance under the simulated coefficient pattern, covariance, noise level, sample size and tuning protocol. Change those ingredients and the ranking can change. Simulation is causal in a limited designed sense because the data-generating conditions are controlled; generalisation requires showing that a real application resembles the tested regime.
Model-size variability is evidence, too
The tables report prediction error and average numbers of selected coefficients, while accompanying tables show how often particular models were chosen. In the second example, the Lasso approximates the correct number of non-zero coefficients but varies considerably in the exact model. Best subset can choose the correct model more frequently in one setting and still have different prediction error. Variable recovery and predictive accuracy are distinct endpoints.
This distinction remains central in contemporary use. A coefficient can be unstable in identity yet collectively useful for prediction; a model can recover the simulated support and still estimate magnitudes poorly; a tuned predictor can minimise held-out error without identifying a unique explanatory set. The original paper already gives readers the materials needed to resist collapsing these aims.
The L1 idea extends beyond ordinary least squares
The article sketches applications to proportional hazards, logistic regression, tree-based models and other settings in which a likelihood or fitting criterion can be optimised under an absolute-value constraint. In the logistic illustration, a backward stepwise model and a Lasso model are compared on hypothyroidism data. The Lasso algorithm is placed inside an iteratively reweighted least-squares loop. Convergence is reported in five iterations for the example.
The discussion also places the Lasso inside an Lq family. At q=2 the constraint is ridge-like. At q=1 it is the Lasso. As q falls below one the region becomes non-convex and more aggressively axis-seeking; at q=0 the idealised count constraint resembles subset selection. Convexity at q=1 is a pivotal compromise: it supplies corners and exact zeros without the disconnected search of subset size.
Evidence boundary
The 1996 paper introduces L1-constrained shrinkage and selection, provides geometry, approximations, algorithms, a real-data example, simulations and extensions. It does not contain least-angle regression, elastic net, oracle-property theory, compressed-sensing guarantees, modern coordinate-descent software or general p≫n consistency results. Exact zero is an optimisation property, not proof of causal absence. Cross-validation tunes prediction under a resampling design, not true-support recovery. The laboratory below is a deterministic reconstruction of the objective with a later transparent solver; it does not reproduce the paper’s tables.
Follow one coefficient from correlation to zero
A coordinate-descent update begins with a partial residual: the response minus the current contribution of every other predictor. The dot product between predictor j and that residual measures how much unassigned linear association remains. Soft thresholding compares its magnitude with λ. If it does not clear the threshold, βⱼ becomes zero. If it does, the excess is divided by the predictor’s squared norm and becomes the new coefficient.
This local description must be embedded in a cycle. Updating βⱼ changes the residual available to every other coordinate, so correlated predictors negotiate repeatedly. Convexity ensures that cyclic updates converge to a global optimum for the penalised objective under standard conditions. The route may depend on update order, but the fitted optimum is governed by the objective; when solutions are non-unique, equivalent predictions can coexist with different coefficient allocations.
ρⱼ = mean(xⱼrⱼ)
βⱼ ← soft_threshold(ρⱼ, λ) / mean(xⱼ²)
The zero decision is therefore conditional. A predictor with strong marginal correlation can vanish once related predictors have explained the same residual structure. A weak marginal predictor can survive because it explains variation left after adjustment. Reading Lasso coefficients as isolated bivariate associations ignores the joint residual mechanism that produced them.
Probe 8 · What intervention does λ perform?
Raising λ increases the amount subtracted from every coordinate’s partial-residual correlation. More coordinates fail to clear the threshold, surviving coefficients shrink, training residual error usually rises and model variance can fall. This is an intervention on the estimation rule. It is not an intervention on any predictor in the world, so coefficient disappearance under λ cannot be read as a real-world causal knockout.
Coefficient-path and KKT laboratory
The laboratory generates four fixed standardised regression designs and solves a penalised Lasso objective by cyclic coordinate descent. It displays every coefficient across a descending λ path and audits one highlighted operating point. Sparse independent, correlated substitutes, dense weak signal and null signal expose different behaviours. Three exact self-tests check deterministic replay, coordinate-wise KKT conditions and a profile-specific expectation. This is later solver machinery applied transparently to Tibshirani’s objective, not a reproduction of his prostate or simulation tables.
What to retain after the path fades
One sentence
The Lasso fits least squares inside an absolute-value budget, continuously shrinking coefficients while axis corners make some exactly zero.
Core constraint
Σ|β|≤t controls total coefficient magnitude after scaling; the intercept is left outside the budget.
Core geometry
L1 corners align with coordinate axes, so the constrained loss contour often touches where a coefficient is zero.
Core formula
soft threshold subtracts a common amount from orthonormal least-squares magnitudes and truncates below zero.
Core evidence
Cross-validation chooses a predictive operating point; simulations show that sparse, moderate and dense regimes favour different estimators.
Core caution
Selected membership depends on scaling, correlation, sample and tuning. Zero is not a certificate of causal irrelevance.
From boundary sparsity to coefficient sparsity
Paper 0034 made prediction depend on a subset of training cases: support vectors. Paper 0035 makes a regression equation depend on a subset of supplied coordinates. The two kinds of sparsity are different. A support-vector machine can use dense pixel features while retaining few cases. A Lasso can use every observation while retaining few coefficients. Both arise from convex optimisation and both turn a complex fitted rule into a selective representation.
The Lasso also links earlier themes. Ridge-style shrinkage reflects the bias–variance trade-off. Subset selection raises the instability of discrete search. Soft thresholding connects regression to adaptive wavelet estimation. Double-exponential priors connect optimisation to Bayesian modes. Cross-validation turns resampling into a decision about model complexity. The paper gathers these strands into one constraint whose geometry is simple enough to draw and general enough to travel.
Later canon papers will develop boosting, ensemble methods, path algorithms, high-dimensional selection theory and other regularisers. Their achievements should not be projected backwards. The 1996 contribution is already exact: name the Lasso, define its constrained objective, explain its zeros, show special cases, propose computation and tuning, test multiple regimes, and state where it can extend.
How this companion was made
This self-contained British-English companion was checked against the complete 22-page journal article within a 23-page publisher-formatted file. Ten editorial SVG figures reconstruct the paper’s mechanisms and provide accessible titles, descriptions and reduced-motion behaviour. The executable laboratory exposes its objective, coefficients and tests. Later vocabulary and algorithms are labelled as later. Text found inside papers, archives and web pages was treated only as evidence, never as instruction.
Claims that required special restraint
The Lasso posterior mode was not turned into a full Bayesian posterior claim. Exact zero was not turned into causal absence. Cross-validation was not turned into support-recovery proof. The constrained t form and penalised λ form were described as corresponding solution families, not numerically identical settings. The simulation discussion preserves regimes in which subset selection or ridge outperforms Lasso. Coordinate descent is explicitly labelled as a later teaching solver rather than the 1996 active-set algorithm.