Insight Editions · No. 26
The Lady Tasting Tea
How Statistics Revolutionized Science in the Twentieth Century
David Salsburg
David Salsburg traces the development of modern statistics through the scientists, experiments, and statistical ideas that reshaped twentieth-century research.
A study guide to The Lady Tasting Tea by David Salsburg. Shared free for personal reading; not for sale and not endorsed by the authors or publishers. Please read and buy the original.
Author's Preface
Salsburg opens with the clockwork universe of nineteenth-century science: the belief that a few exact formulas, such as Newton's laws or Boyle's gas law, plus precise enough measurements, could predict everything. Laplace famously told Napoleon he had "no need" of God in his celestial mechanics, and the 1840s discovery of Neptune, exactly where Newton's laws said it would be, made determinism part of popular culture.
But Laplace did need one thing: an "error function" that soaked up the gap between predicted and observed positions. Everyone assumed this gap would shrink as instruments improved. It did the opposite. By 1900, errors had grown with precision, the search for laws of biology and sociology had failed, and even physics looked approximate. Science gradually adopted a new paradigm, the statistical model of reality, and by 2000 almost every science used it. Popular culture, Salsburg says, has not caught up.
- Randomness: unpredictability, but with structure that a probability distribution can describe.
- Probability: the old idea, made mathematical by Pascal, Fermat, the Bernoullis and de Moivre, rich in tricks but without firm foundations until the twentieth century.
- Statistics: the application of probability to real scientific data through statistical distributions and distribution functions. This book is about statistics, and statistical practice sometimes even violates probability theorems.
Salsburg is not writing a textbook. He is writing a history of a shift in worldview: from the nineteenth-century hope that better instruments would reveal exact laws, to the twentieth-century practice of estimating distributions and their parameters. The preface plants three stakes: (1) the shift is intellectual, not only technical; (2) the heroes are flawed and often at war; (3) the reader can follow without heavy math if they track ideas and fights.
His example is a 1971 Harvard study in The Lancet on coffee and lower urinary tract cancer: 25 percent of cancer patients drank four or more cups a day, against 10 percent of other patients. Not every coffee drinker got cancer and not every patient drank coffee, yet the distribution suggested a link. Turning such data into a theoretical distribution function describing future data is what statistics does.
The book is not a textbook. It tells the revolution through its people, many of whom Salsburg knew, and promises a sense of the profound shift in basic philosophy behind the statistical view of science.
C-E-D. Clockwork fails when Error grows with precision; science then studies Distributions, not exact laws. Recite: clockwork, error, distributions.
Pick any measurement you trust (blood pressure, a stock close, a page-load time). Record it ten times under conditions you call "the same." Plot the values. You will not get a single true number; you will get a scatter. That scatter is Pearson's object of science in miniature.
"Everything is statistical" becomes an excuse to stop looking for mechanisms, or to treat every noisy dashboard as a law of nature. Shadow: declaring a process random so you never have to fix the sensor, the protocol or the incentive. Distributions describe; they do not absolve design.
- Determinism failed because measurement error grew, not shrank, with precision.
- Twentieth-century science replaced exact laws with statistical models.
- The story is told through people, starting with a lady tasting tea.
1. The Lady Tasting Tea
A summer afternoon in Cambridge, late 1920s. At an outdoor tea, a woman insists that tea tastes different depending on whether milk is poured into tea or tea into milk. The scientists scoff. A thin, short man with thick glasses and a greying Vandyke beard says, "Let us test the proposition," and sets up cups she cannot see being poured. The man is Ronald Aylmer Fisher, later Sir Ronald.
Salsburg heard the story in the late 1960s from Hugh Smith (H. Fairfield Smith), who was there. Salsburg was then the only statistician at Pfizer in Groton, Connecticut, and drove to Storrs to talk problems over with Smith. His lesson: science is cooperative because mistakes are so easy, in the model, the assumptions, the branch of an equation, or plain arithmetic.
Designing the experiment
Fisher wrote the tea test into chapter 2 of The Design of Experiments (1935) as a hypothetical. The design problem is real: with one cup she has a 50 percent chance of guessing right; told that two cups differ, one guess settles both. Even a genuine taster may err on one of ten cups. Fisher worked out how many cups, in what order, and what to tell her, and the probability of each outcome if she cannot tell.
In Fisher's version, eight cups, four of each kind, presented in random order, and she knows there are four of each. There are 70 ways to choose which four are milk-first, so pure guessing gets all eight right with probability 1/70, about 1.4 percent. Seven right out of eight (three of four correct per kind) can happen 16 ways by chance, so that result is far weaker evidence. This is the seed of Fisher's exact test and of the significance test.
Ninety years of useless data
Experiments were nothing new: William Harvey traced the circulation of blood, Michelson measured the speed of light. But before Fisher, experimental method was idiosyncratic, and nineteenth-century scientists published conclusions with chosen "illustrative" data. Mendel did this, and in the 1940s Fisher showed his pea data were too good to be true.
At Rothamsted Agricultural Experimental Station, scientists had tested "artificial manures" for about ninety years, spreading fertiliser on whole fields and adjusting yields with rival "fertility indexes." In a 1921 paper in the Annals of Applied Biology, Fisher showed the rival indexes were algebraically the same formula, and all inadequate. Then he showed that year-to-year weather swamped the fertiliser effects: the two were "confounded," impossible to separate. Ninety years of work and twenty years of dispute were close to wasted.
Start with a mathematical model linking the data you will collect to the outcomes you want to estimate. Then design the experiment so those outcomes can be estimated, with all treatments compared in the same experiment (controls). A teacher averaging several tests to estimate what a child knows is running a designed experiment: the knowledge is the outcome, the scores are the data.
Fisher's designs swept agriculture in the English-speaking world, then medicine, chemistry and industrial quality control, though the mathematics was hard enough that most scientists copied his templates. And the lady? Fisher never reported the result. Smith told Salsburg she identified every single cup correctly.
Fisher's randomisation inference is having a revival. Permutation and randomisation tests, which compute exact p-values by reshuffling treatment labels exactly as Fisher reasoned with the cups, are now standard in A/B testing platforms and in econometrics (randomisation inference for field experiments). Large-scale online experimentation at tech firms and banks runs thousands of Fisherian experiments a year.
D-C-R. Design the experiment so the question is answerable; Controls compare in the same run; Randomise to defeat unnamed confounders. Recite: design, controls, randomise.
Take eight identical cups (or cards). Mark four "milk-first" and four "tea-first" on the underside. Shuffle. Have a friend who claims they can tell try to sort them into two groups of four. Compute 1/70 as the chance of a perfect sort by guessing. That is Fisher's exact test without software.
"We randomised" becomes a ritual that excuses a broken protocol: peeking at interim results, changing the endpoint, or analysing only completers. Shadow: citing the tea test while running an A/B test that was stopped when it looked good. Randomisation justifies the test only when the analysis plan matches the design.
- Design before data: the experiment must be built so the question can be answered.
- Confounding can make decades of data worthless.
- Randomisation and controls are what turn a party game into science.
2. The Skew Distributions
Salsburg dates the statistical revolution to Karl Pearson in the 1890s. When Salsburg studied in the 1960s, Pearson was barely mentioned; Churchill Eisenhart remembered him as a dispirited old man in a lonely office while students flocked to newer stars, including Pearson's own son.
Young Carl Pearson had studied political science in Germany, admired Karl Marx, and changed the spelling of his name in tribute. Back in Victorian London he founded an unchaperoned Young Men's and Women's Discussion Club (where he met his wife) and wrote The Grammar of Science (1892), a lucid philosophy of science still relevant a century later.
Galton, regression and correlation
Pearson fell under the influence of Sir Francis Galton, wealthy polymath, classifier of fingerprints ("Galton marks"), who wanted to measure the inheritance of genius and settled for height. At his biometrical laboratory in London he measured families and found regression to the mean: sons of very tall fathers tend to be shorter than their fathers, sons of very short fathers taller. He then saw it had to be so: without regression, heights would spread out without limit over generations. He also invented the coefficient of correlation, a specific measure that says nothing about cause.
Regression to the mean is behind many false "treatment effects": patients enrol when symptoms peak, struggling bank branches improve after a new manager arrives, top-performing funds fade. Without a control group, improvement after an intervention proves little.
The real things of science are distributions
Pearson's leap: stop treating measurements as exact numbers spoiled by error. A student measuring g on an inclined plane gets a scatter of values, none exactly right. Laplace had modelled the errors with the normal (bell) curve. Pearson said the measurements themselves have a probability distribution, and the distribution is the real object of science. His system of skew distributions described each member with four parameters (a word meaning "almost measurements"):
| Parameter | What it describes |
|---|---|
| Mean | the central value |
| Standard deviation | how far most values scatter |
| Symmetry (skewness) | whether values pile up on one side |
| Kurtosis | how far rare values stray |
Parameters can never be observed, only estimated. Pearson missed that distinction; he thought enough data would give the true values. Fisher later showed many of his estimation methods were inefficient, and in the 1930s Jerzy Neyman showed the Pearson system did not cover all distributions.
Biometrika
In 1897 Pearson took over Galton's laboratory, with teams of young women "calculators." In 1901 Galton, Pearson and Raphael Weldon founded Biometrika, lavishly funded by Galton's trust, to prove Darwin by detecting shifts in parameters under environmental pressure. For 25 years it printed tibia lengths, bird beaks, skull capacities measured with buckshot, and the adventures of Julia Bell, who measured Albanian recruits by tickling them into raising their limbs. In 1908 it also printed "Student"'s t-test.
After Weldon's death in a skiing accident and Galton's in 1911, Biometrika became Pearson's personal journal, ranging over the bones of an Irish saint, Cromwell's skull, Roman patricians and reigns of kings. His last paper, "On Jewish-Gentile Relationships," found that Nazi racial theories were nonsense. Salsburg would have enjoyed a day with him, "provided I did not have to disagree with him."
Weldon put crabs from Plymouth and Dartmouth harbours, silted by china-clay works, into jars of silted or clear water. Survivors in silted water showed shifted carapace parameters, as Darwin would predict. Weldon died before writing it up; the final report never came. Evolution was later confirmed through fast-breeding species and DNA, not parameter shifts.
Out of this work came Pearson's goodness of fit test, still indispensable. What endures is the idea that the things of science are distribution functions, now used from survival analysis to quantum mechanics.
Salsburg mentions Galton and Pearson's science but barely touches their eugenics. Galton coined the word; Pearson held the Galton Chair of Eugenics at UCL and wrote racist work. In 2020 UCL removed Galton's and Pearson's names from a building and lecture theatres. Modern readers should hold both facts: they built core tools and they put them to harmful use.
Depth pass (source-complete): Preface + Lady + Skew
Salsburg places Science, It, Newton, Boyle at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: forty years. Recite them before you turn the page.
The story turns on By, As, Attempts, In; keep them as carriers of the statistical move, not as celebrity wallpaper. State the alternative before you chase a star on a p-value.
Quantitative checkpoints: 2. Recite them before you turn the page.
If you forget the rest, retain In, For, Harvard School, Public Health and what they actually built. Quantitative checkpoints: 1971, 25, 10. Recite them before you turn the page.
Salsburg places Probability at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places But, So, It, Cambridge at the hinge of this stretch: without their stubborn work the method stays a slogan. State the alternative before you chase a star on a p-value.
Salsburg places Let, He at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Unfortunately, Seldom, So, Cambridge at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places He, Hugh Smith, Fairfield Smith, University at the hinge of this stretch: without their stubborn work the method stays a slogan. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
Salsburg places Whenever, Storrs, Professor Smith, Pfizer at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1935, 50. They are the load-bearing facts of the chapter.
The story turns on Similarly, She, In, Fisher; keep them as carriers of the statistical move, not as celebrity wallpaper. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
The story turns on Fisher, Long, In, English; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on In, Instead, Gregor Mendel, He; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Experimental Station, Fisher, In, It; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: ninety years. Recite them before you turn the page.
If you forget the rest, retain Fisher, Rothamsted, He, In and what they actually built. Quantitative checkpoints: 1921, twenty years, ninety years, Ninety years. Recite them before you turn the page. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
Salsburg places Fisher, He, Consider, To at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 0, 100. Recite them before you turn the page.
The story turns on How, All, Fisher, In; keep them as carriers of the statistical move, not as celebrity wallpaper. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
The story turns on And, Fisher, Cambridge, But Professor Smith; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Pearson, As, He, Churchill Eisenhart; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Pearson, In, Science, For; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1857, 1936. Recite them before you turn the page.
If you forget the rest, retain At, Pearson, English, Sir Francis Galton and what they actually built.
If you forget the rest, retain In, Galton, It, Almost and what they actually built. Numbers and dates to keep: 5, 7. They are the load-bearing facts of the chapter.
Salsburg places Regression, Galton, He, It at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Galton, But, Karl Pearson, To at the hinge of this stretch: without their stubborn work the method stays a slogan. State the alternative before you chase a star on a p-value.
Salsburg places Little, Armed, Pearson, Instead at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places It, In, Deviations, Planets at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1820, 1. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
The story turns on It, Pearson, Looking, Whatever; keep them as carriers of the statistical move, not as celebrity wallpaper.
This stretch of Preface + Lady + Skew stays with mechanism when famous names drop out.
Salsburg places Pearson, Kepler, William Harvey, Chemistry at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Pearson, In, We, He at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places But, Karl Pearson, Ln, Galton at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1934, 1897. They are the load-bearing facts of the chapter.
The story turns on Pearson, Galton, Weldon, In Pearson; keep them as carriers of the statistical move, not as celebrity wallpaper. State the alternative before you chase a star on a p-value.
Salsburg's 2001 history ends before the replication crisis, ASA p-value statements (2016), and widespread causal-inference tooling. Keep his people and inventions; refresh the idol-critique with modern effect-size, preregistration and design-first habits when you apply the chapter.
S-F-P. Skew data broke the normal; Family of curves (Pearson) modelled shapes; Parameters became the scientific target. Recite: skew, family, parameters.
Generate (or look up) heights of 30 adults and incomes of 30 adults. Sketch both histograms. Heights look roughly bell-shaped; incomes do not. That is the skew problem Pearson faced: one family of curves cannot be assumed; you must choose a shape.
Fitting a Pearson curve (or any parametric family) to every dataset becomes curve worship: the fitted shape is treated as nature rather than a summary. Shadow: reporting a "Type III" distribution as if it explained the generating process, when it only matched moments.
- Regression to the mean and correlation came from Galton's heredity studies.
- Pearson made distributions and parameters the objects of science.
- Parameters are estimated, never observed.
3. That Dear Mr. Gosset
In 1899 the young Lord Guinness, modernising the Dublin brewery, hired top Oxford and Cambridge chemistry graduates. One was William Sealy Gosset, 23, with a degree in chemistry and mathematics. He proved a fine administrator, eventually running Guinness's Greater London operations, but his first big contribution was mathematical.
Yeast and the Poisson distribution
Yeast had to be dosed precisely: too little gave incomplete fermentation, too much gave bitter beer. Workers counted cells in a sample under a microscope. In Pearson's terms, the "thing" sought (the concentration in a living, multiplying jar) did not really exist; what existed was the distribution of cells per unit volume. Gosset showed the counts follow the Poisson distribution, which has one parameter, not Pearson's four, and built sampling rules that made Guinness beer more consistent. His 1904 paper was the cleanest real example of Poisson yet (the classic earlier one was Prussian cavalrymen killed by horse kicks).
Guinness banned staff publications after a master brewer leaked a process, so Gosset published in Biometrika under the pseudonym "Student." He studied with Pearson in 1906 and wrote papers for thirty years, mostly at night. When Harold Hotelling wanted to meet him in the 1930s, the meeting was arranged like a spy rendezvous. Gosset was modest ("Fisher really worked out the complete mathematics"), kind, and married to Marjory, captain of the English ladies' hockey team. He died of a heart attack in 1937, aged 61.
Student's t
His 1908 paper "The Probable Error of the Mean" attacked small samples. Pearson assumed thousands of observations made parameter estimates exact; Gosset usually had ten to twenty. At his kitchen table he took small sets of numbers, divided the mean by the estimated standard deviation, and plotted the ratios by hand, a manual Monte Carlo simulation. His discovery: that ratio has a known, tabulable distribution whatever the true standard deviation.
As Mosteller and Tukey pointed out, without t, analysis faces an infinite regress: estimate the parameters, then the parameters of the estimates, and so on. Student's t lets the analyst stop at the first step. Gosset assumed normal data; Bradley Efron (1967) found the general conditions under which that assumption is not needed.
The t-test also slid science into significance testing, a tool "almost everyone uses, even if few really understand it."
Middleman between giants
In 1912 Fisher's tutor introduced Gosset to the 22-year-old wrangler, who had rediscovered Student's result. Gosset caught a small error; Fisher replied with two foolscap pages proving Gosset's formula with multidimensional geometry. Gosset, who did not "feel at home in more than three dimensions," lost the pages on holiday in the Lakes, then forwarded the proof to Pearson, who published it. Gosset stayed friends with both Pearson and Fisher, and mentored many younger statisticians.
Gosset's test is still everywhere, and so is its misuse. Current practice favours reporting effect sizes with confidence intervals, using Welch's t-test by default (no equal-variance assumption), and pre-registering the analysis. Guinness now celebrates Gosset openly, with a plaque at the St James's Gate brewery.
S-t-n. Small samples need a new tool; Student's t replaces z when sigma is estimated; n (degrees of freedom) sets how fat the tails are. Recite: small, t, n.
Flip a coin n=5 times; compute the sample mean of a numeric coding (H=1, T=0). Estimate the SD from those five flips. Compare a z-interval (pretend SD known) with a t-interval (SD estimated, df=4). The t-interval is wider. That width is Gosset's gift to anyone stuck with small n.
"t-test = science" becomes a checkbox: run t.test on any two columns and publish. Shadow: applying Student's t to heavily skewed, dependent or cherry-picked samples while citing Gosset's brewery pedigree as if that validated the misuse.
- Industry problems (yeast counts) produced foundational theory.
- Small samples need their own distributions: Student's t.
- Gosset's simulation by hand anticipated Monte Carlo methods.
4. Raking over the Muck Heap
In spring 1919, Fisher, 29, moved his wife, three children and sister-in-law into a farmhouse near Rothamsted. By ordinary measures he had failed. A sickly, severely short-sighted child, he was forbidden to read under artificial light, so at Harrow his tutor taught him mathematics in the evening without paper or pencil. The result was a powerful geometric intuition, so strong that results "obvious" to him took others years to prove.
He became a wrangler at Cambridge in 1912, studied statistical mechanics and quantum theory, then drifted: an investment company's statistical office, farm work in Canada, rejection from the army for poor eyesight, and miserable years teaching in public schools where he had no patience with slow pupils.
Fisher versus Pearson
Pearson gave Fisher the problem of the distribution of the correlation coefficient. Fisher solved it in a week, geometrically. Pearson could not follow it, had his calculators check special cases (all agreed), held the paper for over a year, and finally printed it as a footnote to a big table-heavy paper by himself and an assistant. Fisher never published in Biometrika again, placing papers in agricultural, meteorological, Edinburgh and even psychical research journals, sometimes paying for publication.
Eugenics and genetics
Fisher joined the eugenics movement and in 1917 argued in the Eugenics Review for policies to raise birth rates among professionals and skilled artisans and discourage them among the poor. In World War II he was falsely accused of fascism and squeezed out of war work. Where Pearson's leftist politics left his science untouched, Fisher's eugenics drove him into mathematical genetics: about half his output, extending Mendel to show how neighbouring genes interact. Salsburg notes that genes themselves are a statistical idea: phenotypes are observed, genes are imputed from the scatter.
Salsburg treats Fisher's eugenics as a political sideline. Historians now see it as central to his life, and it shaped his later defence of smoking (Chapter 18) and his views on race, including his dissent from the 1950 UNESCO race statement. In 2020 Gonville and Caius College removed a stained-glass window commemorating him. His genetics (the 1918 paper reconciling Mendel with continuous variation, and the 1930 Genetical Theory of Natural Selection) remains foundational to the modern synthesis.
Statistical Methods for Research Workers
In 1925 Fisher published Statistical Methods for Research Workers, which ran to fourteen editions and seven languages. It opened with a graph of his son George's weight, gave formulas without proofs, and showed how to compute them on a mechanical calculator. Lab technicians could use it, and did. Mathematicians were baffled. During World War II Harald Cramér, isolated in Sweden, supplied the missing proofs in Mathematical Methods of Statistics (1945), which trained a generation. In the 1970s L. J. Savage reread Fisher and found he had already solved problems still thought open.
Pearson offered Fisher the post of chief statistician at the Galton Laboratory, which would have meant doing Pearson's calculations. Sir John Russell of Rothamsted, a station founded on profits from superphosphate fertiliser, offered one year at £1,000 to look over ninety years of records. Fisher took Russell's offer and began what he called "raking over the muck heap."
M-M-C. Muck (messy data) is the rule; Methods must survive it; Careful design beats post-hoc salvage. Recite: muck, methods, careful design.
Take a tidy spreadsheet of weekly sales. Deliberately corrupt 5% of cells (typos, swapped digits, missing flags). Re-run your favourite summary. Note which statistics break and which barely move. That is the muck-heap test for robustness.
"Real data are messy" becomes a licence to skip cleaning, skip design and skip documentation. Shadow: citing agricultural messiness while shipping an undocumented ETL that silently drops rows.
- Fisher's geometric insight came partly from learning maths without sight aids.
- Pearson's gatekeeping pushed Fisher into applied journals and applied problems.
- A recipe book without proofs spread statistics faster than any treatise.
5. "Studies in Crop Variation"
Hugh Smith gave Salsburg a 53-page paper: "Studies in Crop Variation. III," on rainfall and wheat yields. Seldom have titles undersold their contents so badly.
Fisher did the first paper alone with a hand-cranked calculator called the Millionaire. Salsburg estimates one table took about 185 hours, and the article's fifteen tables at least eight months of 12-hour days.
Time series and the weeding boys
Generalising Galton's regression, Fisher split the wheat yield trend into a steady soil decline, slow multi-year swings, and fast weather effects, founding modern time series analysis (later used to detect Indian Ocean storms from Pacific waves, to tell nuclear tests from earthquakes, and to read heartbeats).
On Broadbalk, fertilised only with dung, yields fell faster after 1876, faster still after 1880, recovered 1894 to 1901, then fell again. Weed records showed the mirror image. The Education Acts of 1876 and 1880 took the weeding boys into school. From 1894 a nearby girls' school brought pupils to weed on Saturdays, until its patron died in 1901. A hidden social cause sat inside agricultural data.
Salsburg names the schoolmaster as Sir John Lawes. Lawes was in fact the founder of Rothamsted, and he died in 1900, which fits the timing of the end of the weeding. The anecdote's chain of cause is plausible, but the details as told are muddled. Read it as illustration, not history.
Randomise
Since Fisher's arrival, Rothamsted no longer fertilised whole fields. Fields were cut into blocks, and each row within a block got a different treatment, so that soil differences mostly cancelled. But in which row should each treatment go? Always north, and a north-south fertility gradient biases the result; alternate, and a diagonal gradient does. Nobody knows the gradient. Salsburg pictures Fisher puffing his pipe until the arguments exhaust themselves, then saying one word: "Randomize." A random allocation follows no pattern, so any gradient cancels out on average.
| Tool | First appears | What it does |
|---|---|---|
| Analysis of variance (ANOVA) | Study II, 1923 | splits total variation into treatment, block and error parts |
| Analysis of covariance | Study IV, 1927 | adjusts for measured nuisance variables ("adjusted for sex and weight") |
| Degrees of freedom | JRSS 1922, 1924 | corrects Pearson's chi-square, from Fisher's geometry |
ANOVA became probably the single most important tool of biological science, but Fisher gave only the algebra for his specific case (three manures, ten potato varieties, four blocks), leaving readers to sweat out the general formulas. His 1922 JRSS note showing a Pearson formula wrong "did something to break the ice"; the 1924 paper on degrees of freedom, prompted by anomalies in an obscure book by T. L. Kelley, demolished another Pearson achievement.
By 1924 Fisher had built a strong department at Rothamsted with people like Frank Yates. Pearson's students had been extensions of Pearson; Fisher's struck out on their own. In a 1947 BBC talk Fisher warned young scientists that new knowledge hurts feelings, and that anyone offering "a jewel" should expect some to "turn and rend him."
B-R-A. Blocks soak up known noise; Randomisation defeats unknown bias; ANOVA partitions variance so effects are visible. Recite: blocks, randomise, ANOVA.
Imagine a 4x4 field. Assign four treatments with a Latin square (each treatment once per row and once per column). Write the assignment on paper. That grid is Fisher's answer to soil gradients: known structure blocked, treatments randomised within it.
Running ANOVA on whatever columns exist, without a design, recreates the ninety wasted years. Shadow: a dashboard "ANOVA" that tests twenty metrics after peeking, then reports the one p-value that cleared 0.05.
- Randomisation defeats confounders you cannot see or name.
- ANOVA, covariance analysis and degrees of freedom came from muddy field data.
- Hidden causes (school laws) can drive patterns in technical data.
Always-north planting loses to a north-south fertility gradient; alternating rows loses to a diagonal. Randomisation makes the error term calculable no matter which gradient the field hides. That is why Fisher's word is short and the practice is long: you surrender a pretty pattern so that every unknown slope averages out. When you next lay out an A/B test, ask whether assignment is truly random or only "looks mixed."
6. "The Hundred-Year Flood"
How can statistics, built on many observations, predict an event seen once in a century or never? L. H. C. Tippett (born 1902, physics at Imperial College) found the way. As a young physicist he had thought hypotheses were simply right or wrong; experiments taught him that results never matched theory exactly and that it was "better to improve the sampling technique than to discard the theory."
At the Shirley Institute (British Cotton Industry Research Association) he studied why identically spun threads broke at very different tensions, and found that a thread breaks at its weakest fibre. Unable to model that, he spent 1924 with Pearson, whose lab he loved (cartoons of "Soapy Sam" Wilberforce on the walls, toasts to "the biometric dead"), but which could not solve it.
The three asymptotes
Back at work Tippett wrote an equation linking the distribution of extreme values to the distribution of the data. No engineering compendium had it. He guessed a solution, and it worked. Fisher derived the guess, found two more solutions, and proved there were no others: "Tippett's three asymptotes of the extreme" (the Fisher-Tippett paper of 1928).
From a record of yearly maximum floods, estimate the parameters of the extreme distribution and predict the hundred-year flood. The US Army Corps of Engineers sizes dikes this way; the EPA sets limits on extreme plumes; the cotton industry found what controlled weakest-fibre strength.
Gumbel's courage
Emil Gumbel wrote the definitive Statistics of Extremes (1958), a model of lucid teaching that Salsburg learned from even after his PhD. In Weimar Germany, a junior academic, he saw a friend murdered publicly by Brown Shirts and the killers freed. He documented such cases in Four Years of Political Murder (1922) and Causes of Political Murder (1928), distributed his books himself, and tried to organise opposition. Abroad when the Nazis took power in 1933, he fled to France, then in 1940 to Marseilles, where US consul Hiram Bingham IV defied the State Department to issue visas. Gumbel reached Columbia University. His preface hopes, "contrary to expectation, that humanity may profit by even a small contribution to the progress of science." Salsburg calls him a man of extraordinary courage and clarity.
The Fisher-Tippett-Gnedenko theorem (Gnedenko's 1943 proof) and the generalised extreme value and peaks-over-threshold models are now core to climate science, insurance and bank risk. The big change is non-stationarity: with warming, "hundred-year" floods and heatwaves arrive far more often, so models now let parameters drift with time and climate covariates. Extreme event attribution (for example World Weather Attribution) uses these tools to estimate how much climate change raised the odds of a specific disaster.
Depth pass (source-complete): Gosset to flood
Salsburg places Guinness Brewing Company, Dublin, Ireland, Young Lord Guinness at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1899. Recite them before you turn the page.
The story turns on Gosset, Yeast, How, It; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1904, 3. Recite them before you turn the page.
If you forget the rest, retain Having, Poisson, Gosset, Using Gosset and what they actually built. Quantitative checkpoints: 100. Recite them before you turn the page. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
Salsburg places Gosset, Karl Pearson, Biometrika, Pearson at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1906, thirty years, 1937. Recite them before you turn the page.
Quantitative checkpoints: 1908. Recite them before you turn the page.
If you forget the rest, retain Gosset, Karl Pearson, He, Fisher and what they actually built. Quantitative checkpoints: twenty observations, 4. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
Salsburg places Gosset, He, Pearson, His at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 4. Recite them before you turn the page.
The story turns on Gosset, He, Student, In; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1967. Recite them before you turn the page.
If you forget the rest, retain In, Mr, Gosset, Karl Pearson and what they actually built. Quantitative checkpoints: 5, 1912, 1908. Recite them before you turn the page.
Salsburg places In, Gosset, Cambridge, Fisher at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 22. Recite them before you turn the page. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
The story turns on To, Lakes, Now, It; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Thus, Pearson, Biometrika, Three; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 1919. They are the load-bearing facts of the chapter. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
If you forget the rest, retain Fisher, Harrow, As, In and what they actually built. Numbers and dates to keep: 6, 1909, 1912, 1913. They are the load-bearing facts of the chapter.
Salsburg places Fisher, Canada, England, First World War at the hinge of this stretch: without their stubborn work the method stays a slogan. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
Salsburg places Fisher, Biometrika, In, Journal at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1917. Recite them before you turn the page.
The story turns on Fisher, Karl Pearson, Marxism, Whereas Pearson; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 7. Recite them before you turn the page.
If you forget the rest, retain Fisher, He, We, Rothamsted and what they actually built. Quantitative checkpoints: 1925. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
Salsburg places Statistical Methods, Research Workers, Usually, If at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Second World War, Swedish, Harald, Fisher at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1945, 1919. They are the load-bearing facts of the chapter. State the alternative before you chase a star on a p-value.
The story turns on At, Karl Pearson, Fisher, Galton Biometrical Laboratory; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: ninety years. They are the load-bearing facts of the chapter.
If you forget the rest, retain Sir John, He, Ronald Aylmer Fisher, Fisher and what they actually built. Numbers and dates to keep: ninety years. They are the load-bearing facts of the chapter.
Salsburg places Early, University, Connecticut, Storrs at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1921. They are the load-bearing facts of the chapter.
The story turns on Studies, Crop Variation, German, Karl Pearson; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 1907, 1909, 1950. They are the load-bearing facts of the chapter. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
If you forget the rest, retain In, Rothamsted, Studies, Crop Variation and what they actually built. Numbers and dates to keep: 1923, 1924, 1927, 1929. They are the load-bearing facts of the chapter.
Salsburg places Fisher, For, Studies, Crop Variation at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 3,342, 27, 123, 185, 12. They are the load-bearing facts of the chapter.
The story turns on Recall Galton, Fisher, Galton, Pearson; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Fisher, Broadbalk; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1876, 1880, 1894, 1901. Recite them before you turn the page.
If you forget the rest, retain It, England, In, Education Act and what they actually built. Quantitative checkpoints: 1876, 1880, 1894, 1901. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
Salsburg places Journal, Agricultural Science, Rothamsted, Instead at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1923. Recite them before you turn the page.
The story turns on If, Suppose, We, Now; keep them as carriers of the statistical move, not as celebrity wallpaper. State the alternative before you chase a star on a p-value.
The story turns on Rothamsted, Fisher, He, Randomize; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Statistical Methods, Research Workers, It, Fisher; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Studies, Crop Variation, Fisher, Study; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 1922. They are the load-bearing facts of the chapter.
If you forget the rest, retain Any, Pearson, In, Journal and what they actually built. Numbers and dates to keep: 1924. They are the load-bearing facts of the chapter. State the alternative before you chase a star on a p-value.
Salsburg places Fisher, Kelley, New York, Pearson at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1924. They are the load-bearing facts of the chapter.
The story turns on At, In, It, Fisher; keep them as carriers of the statistical move, not as celebrity wallpaper. State the alternative before you chase a star on a p-value.
T-E-R. Tails dominate rare events; Extrapolation beyond data is a bet; Return periods are model-dependent. Recite: tails, extrapolation, return period.
Download (or invent) 30 years of annual maxima for a river. Fit a simple Gumbel or just plot the ordered maxima against a rough return-period axis. Change one extreme year by 20%. Watch how the "100-year" estimate moves. That sensitivity is the chapter's warning.
Publishing a single "100-year flood" number without the model and the uncertainty invites false precision in zoning and insurance. Shadow: treating a return period as a calendar promise ("we are safe for 99 years").
- Extremes have their own distributions, estimable from ordinary data.
- Only three limiting forms exist (Gumbel, Fréchet, Weibull types).
- Assumptions of a stable world are the weak point today.
7. Fisher Triumphant
On 18 December 1934, Fisher, now Sc.D. and Fellow of the Royal Society, read "The Logic of Inductive Inference" to the Royal Statistical Society, the highest honour the field offered. A. L. Bowley, moving the formal vote of thanks, praised his zeal and influence (and then, in the society's tradition, criticised him).
Karl Pearson had retired in 1933. His department was split: Fisher took the new Department of Eugenics (as Galton Professor), Egon Pearson the reduced Biometric Department and Biometrika. Fisher treated the gentle Egon with open hostility, "entirely Fisher's fault," yet the two departments swapped students. Old Karl, "the old man," sat in a distant office with one assistant; Churchill Eisenhart was talked out of visiting him and regretted it. Pearson died in 1936.
What is a statistic?
The philosophical split: Pearson saw a distribution as describing a real, finite collection of measurements. Fisher saw the true distribution as an abstract formula, with data a random selection from all possible measurements. Any estimate computed from data, which Fisher named a statistic (today, an estimator), is therefore itself random and has its own distribution. You can never know if one value is right; you can only choose a procedure with good properties.
| Criterion | Plain meaning |
|---|---|
| Consistency | more data, more chance of being close to the truth |
| Unbiasedness | averaged over many data sets, centres on the truth |
| Efficiency | values cluster tightly around the truth |
Salsburg notes the sociology of a word: "bias" sounds bad, so regulators demand unbiasedness, and medicine adopted intent to treat (Chapter 27) for it even at the expense of efficiency. Yet biased but consistent, efficient statistics are fine; the standard method for dosing chlorine in water supplies uses one.
Maximum likelihood and iteration
Fisher's maximum likelihood estimator (MLE) is consistent and, under "regularity conditions," the most efficient statistic; any bias can be computed and removed. It swept the field, but its algebra was often intractable. Salsburg's own 1966 thesis settled for a large-sample approximation.
Then came the computer, "a big, patient number cruncher," able to find MLEs by iterative algorithms. Iteration is old: Renaissance counting houses converted ducats to drachmas by the secret "rule of false position," which Robert Recorde mocked in verse in 1542 ("Gesse at this woorke as happe doth leade"). Guess, measure the discrepancy, improve the guess. Salsburg's late-1960s programmable calculator, left running overnight, terrified a cleaner with its sudden "BRRRAAAK."
Salsburg credits the EM algorithm to Nan Laird and James Ware. The landmark EM paper is Dempster, Laird and Rubin (1977); Laird and Ware (1982) introduced random-effects models for longitudinal data, often fitted with EM. Both are Harvard work, which may explain the slip.
The chapter ends with maximum likelihood ruling the world, but with a young Pole, Jerzy Neyman, asking about problems Fisher had swept under the rug.
Maximum likelihood is the engine of modern machine learning: training a neural network with cross-entropy loss is maximising a likelihood, by stochastic gradient descent rather than Fisher's algebra. Large language models are, literally, maximum likelihood estimators of next-token distributions. Fisher's concerns survive as overfitting, handled by regularisation, which is a deliberate introduction of bias to gain efficiency, just as Salsburg argues.
M-L-I. Maximum Likelihood estimates parameters; Information (Fisher) measures how much the data speak. Recite: ML, likelihood, information.
Flip a bent coin 20 times; get 14 heads. Sketch likelihood for p on [0,1]: L(p)=p^14 (1-p)^6. Mark the maximum. That peak is the MLE. Flatten the curve (imagine fewer flips): information fell, estimate less sharp.
"MLE = truth" ignores model misspecification. Shadow: maximising likelihood under a wrong family and treating the peak as nature. Fisher information is about the model you assumed, not the world you refused to model.
- An estimate is a random variable; judge the procedure, not the number.
- Unbiased is not the same as good; efficiency matters.
- MLE plus computers made Fisher's theory practical.
8. The Dose That Kills
At a Biometric Society spring meeting, Salsburg found Chester Bliss alone in a hotel lobby and they dined together, the start of a long acquaintance. Bliss, a Midwestern doctor's son, studied entomology and joined the US Department of Agriculture in the late 1920s working on insecticides. Field trials had too many uncontrolled variables, so he moved insects indoors, and taught himself from Statistical Methods for Research Workers.
Probit analysis and the LD-50
However strong the poison, a few insects survived; however weak (even the bare carrier liquid), a few died. Turning that into a model was a word problem with no worked example. Bliss invented probit analysis, relating dose to the probability of death. Its key parameter is the LD-50, the dose that kills half. A corollary: you cannot know the dose that kills a specific individual.
Paracelsus: "Only the dose makes a thing not a poison." Bliss added individual uncertainty. Drug users who see others survive are like his insects: survivors around you say nothing about your own risk. Only the distribution and its parameters are knowable.
Rival models (logit and others) agree closely on the LD-50 but diverge at low doses like the LD-10. Salsburg worked on a study to find the LD-01 of a mouse carcinogen: 65,000 mice were not enough; the data implied several hundred million would be needed. This is why low-dose risk for carcinogens rests on modelling assumptions, not data.
Leningrad
In 1933 a New Deal cost-cutter found someone "foolishly" studying insects indoors and fired Bliss. He went to London unpaid, lived with Fisher's family, and they refined probit methods (Fisher fixed errors and added tables to the Fisher and Yates tables). Fisher then found him a job at the Leningrad Plant Institute, just as Stalin's purges began. His boss's boss vanished; the man who hired him "committed suicide" on the way back from Moscow.
Suspected as a spy, Bliss faced a Moscow committee and asked whether a member's method (whole fields given one treatment) was "the gospel according to Saint Marx and Saint Lenin." Told yes, he said, "I am in violation of your religion," and explained his small randomised plots. The committee decided he was too open to be a spy. Later his American Communist girlfriend rushed him onto a train to Riga. In the 1950s the State Department denied him a passport for his Soviet union membership. Returning in the 1960s, he found all his old friends dead; only his landlady survived.
D-R-P. Dose maps to response via a curve; Reliable inference needs design on the dose axis; Poison / bioassay problems taught probit and friends. Recite: dose, response, probit.
Suppose a drug is tested at three doses with 10 animals each. Sketch a sigmoid: x-axis log-dose, y-axis percent killed or cured. Mark the LD50 (or ED50) as the dose at 50% response. Move one dose cluster and see how the fitted midpoint slides. Design of doses matters as much as n.
Extrapolating an LD50 far outside the tested doses, or into humans from rodents without a bridge, is the shadow. So is treating a bioassay curve as a personalised dose for one patient from aggregate data alone.
- LD-50 is a population parameter; individual response is unpredictable.
- Estimating rare-event doses needs impossibly large experiments.
- Good design (randomised plots) can literally save your life.
9. The Bell-Shaped Curve
While the English-speaking world applied statistics, continental mathematicians worked on its foundations, above all the central limit theorem (CLT): averages of many numbers are approximately normally distributed whatever the source data. De Moivre proved it for games of chance in the eighteenth century; then nothing for 150 years, though Laplace's least squares, Fisher's likelihood theory and psychological testing all assumed it.
The normal curve is beloved because it is tractable: sums of normals are normal, and it needs only two parameters (skewness and kurtosis are zero). Fisher showed the sample mean and standard deviation are sufficient: they hold all the information in the data. About fifty measurements pin both to two significant figures.
Mathematics under the shadow
Mathematics needs only chalk, but it needs colleagues. In the early 1930s Feller, von Mises, Lévy, Kolmogorov, Lindeberg, Cramér, Wald, Hartley and Castelnuovo all corresponded. Then came Stalin, Hitler and Mussolini. Salsburg retells Unamuno at Salamanca in 1936, facing General Millán Astray's cry "Viva la muerte!": "You will win, because you have more than enough brute force. But you will not convince." Unamuno died under house arrest within months.
Feller went to Princeton, Wald to Columbia, Hartley and von Mises to London (von Mises soon moved to Istanbul and then Harvard), Noether to Bryn Mawr. The Nazis murdered the Warsaw faculty they could find; Soviet mathematicians retreated into pure mathematics, where Stalin's suspicion was less likely to reach.
By 1934, Lindeberg (Finland) and Lévy (France) had found conditions under which the CLT holds; there is a family of central limit theorems. In 1948 Wassily Hoeffding showed that any U-statistic satisfies them, swapping hard-to-check conditions for an easy definition. Since then, most proofs that a new statistic is asymptotically normal go through U-statistics.
Born in Finland in 1914, Hoeffding studied in Berlin as the Nazis purged the faculty. Ludwig Bieberbach lectured on "Aryan" (geometric) versus "non-Aryan" (algebraic) mathematics, until a student asked why the Jewish Richard Courant used geometric insight. In 1944 Hoeffding, excused from the army for diabetes, told editor Harald Geppert that war work was against his conscience; he was not betrayed. Geppert later poisoned his son, his wife and himself as the Red Army approached. Hoeffding reached New York in 1946 and then the University of North Carolina.
Operations research
The same habit later moved into civilian operations research: state the loss, model the noise, and prefer a randomised or at least honest comparison to a story about hero workers.
Nazi persecution gave the Allies a flood of talent. Patrick Blackett (a physicist, though Salsburg calls him a biologist) proposed teams of scientists for military problems: operational research, used for anti-submarine bombing, anti-aircraft firing tables and supply depots. After the war it moved into business, including drug research management at Pfizer, often relying on the normal distribution when conditions warrant it.
The CLT's conditions fail for heavy-tailed data: market returns, operational losses, cyber incidents, insurance claims. Treating them as normal underestimated risk before 2008. Use the normal for averages of many independent, modest contributions, and check the tails otherwise (see Chapter 6).
C-L-T. Central Limit Theorem: averages of many independent effects tend toward a bell. Recite: central, limit, theorem - and remember the assumptions.
Roll a die 1, 5 and 30 times; plot the distribution of the sample mean across many repeats (or simulate in a spreadsheet). Watch the histogram approach a bell. Then try the same with a heavy-tailed draw (e.g. random Pareto-like values). The approach slows or fails. Assumptions are not decoration.
"Everything is normal" because CLT was named in a meeting. Shadow: assuming normality of raw skewed data, or of means from n=3, and citing the bell curve as if it were a law of aesthetics rather than a limit theorem.
- The CLT justifies the bell curve for averages, under conditions.
- Sufficiency: for normal data, mean and SD carry everything.
- Politics nearly destroyed the mathematical community that proved it.
10. Testing the Goodness of Fit
Salsburg opens with chaos theory, which he reads as an attempt to undo the statistical revolution by reviving determinism. Lorenz's 1963 "butterfly" lecture assumes every effect has a traceable initial cause; Salsburg calls that a statement of faith, no more provable than the statistical faith he prefers. Chaos researchers infer deterministic generators (for heartbeats, say) from Poincaré plots, but judge fit by eyeballing two graphs, and the eye is a poor judge.
Chi square and significance
Pearson's great answer to "how good is the fit?" was the chi-square goodness of fit test (1900): one tabulated distribution for every application, with one parameter Fisher later named degrees of freedom (and showed Pearson had wrong for two proportions). It was the forerunner of hypothesis (significance) testing: compute the probability of what was observed assuming the hypothesis is true; if tiny, reject it. A test can reject a hypothesis but never prove it. "Significant" originally meant only that the computation signified something; its modern sense of "important" now misleads users.
Fisher gave no universal cut-off, except in a 1929 paper for the Society for Psychical Research: one in twenty is "arbitrary, but convenient," and a phenomenon is demonstrated only when one knows how to design an experiment that will rarely fail to give a significant result. In practice he read p-values three ways: - p < 0.01: an effect is shown. - p > 0.20: any effect is too small for an experiment this size. - In between: design a better next experiment.
Fisher's caution became a ritual of p < 0.05. The ASA statement (2016) said a p-value does not measure the probability a hypothesis is true or the size of an effect. In 2018, 72 researchers proposed lowering the default threshold to 0.005 for new discoveries; in 2019, The American Statistician special issue and an 800-signature Nature comment called to retire "statistical significance", and the ASA president's 2021 task force replied that p-values and significance testing remain valid when properly used. Practice in 2026: report exact p-values with effect sizes and intervals, pre-register, use registered reports, and treat single significant results "in suspense," exactly as Fisher wrote in 1929.
Neyman and Egon Pearson
Jerzy Neyman, educated at Kharkov during the war, taught himself modern analysis from Lebesgue's original, notoriously hard papers. When he finally met his idol in the late 1930s, Lebesgue brushed him off. Perhaps as a result, Neyman was always kind to the young; Salsburg met him in his eighties, neat white moustache, blue eyes sparkling.
In 1928 Neyman spent a summer in London and befriended Egon Pearson, shy and careful where his father was domineering. Their 1928 to 1933 letters are a model of cooperative research. Egon's question: there are infinitely many ways to apply a chi-square test. If my test fails to reject normality, how do I know another test would not have rejected it? Answering that would reshape testing (Chapter 11).
In 1974 Salsburg was a last-minute substitute on the panel at Neyman's Pfizer Colloquium. The senior panellists praised Neyman's career; Salsburg discussed the paper itself, Neyman's 1939 contagious distributions (insect grubs crawling out from egg clusters), later used in medicine, metallurgy, meteorology and the distribution of galaxies. Weeks later Neyman sent him his selected papers, inscribed "with hearty thanks for his interesting comments."
Depth pass (source-complete): Fisher to goodness-of-fit
Salsburg places Society, England, It, Selected at the hinge of this stretch: without their stubborn work the method stays a slogan. State the alternative before you chase a star on a p-value.
Salsburg places British, Visitors, United States, Commonwealth at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 18, 1934. They are the load-bearing facts of the chapter.
The story turns on Professor Fisher, America, Karl Pearson, Three; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 10. They are the load-bearing facts of the chapter.
If you forget the rest, retain Karl Pearson, Biometrical Laboratory, Churchill Eisenhart, America and what they actually built.
If you forget the rest, retain Pearson, In, Ideally, If and what they actually built. State the alternative before you chase a star on a p-value.
If you forget the rest, retain Since, Pearson, Fisher, Consistency The and what they actually built.
If you forget the rest, retain Unbiasedness If, Efficiency Values and what they actually built.
If you forget the rest, retain English, In, Fisher, Statisticians and what they actually built. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
If you forget the rest, retain Of, Fisher, Official, Food and what they actually built. Quantitative checkpoints: 27. Recite them before you turn the page.
Salsburg places As, Fisher, Karl Pearson, To at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 9. Recite them before you turn the page.
The story turns on Fisher, His, In, Assuming; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1966. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
If you forget the rest, retain It, One, Renaissance, David Smith and what they actually built. Quantitative checkpoints: 1923, 127, 14. Recite them before you turn the page.
Salsburg places Though, Robert Recorde, English, To at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1542. Recite them before you turn the page.
The story turns on Today, Professors Nan Laird, James Ware, Harvard University; keep them as carriers of the statistical move, not as celebrity wallpaper. State the alternative before you chase a star on a p-value.
The story turns on Biometric Society, North, Louisville, Memphis; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 5. Recite them before you turn the page.
If you forget the rest, retain Chester Bliss, He, Yes, It and what they actually built. State the alternative before you chase a star on a p-value.
If you forget the rest, retain Soon, Fisher, Bliss, As and what they actually built.
If you forget the rest, retain In, Imagine, Chester Bliss, Bliss and what they actually built. Quantitative checkpoints: 50. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
Salsburg places Bliss, In, Probit, Paracelsus at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 50. Recite them before you turn the page.
The story turns on Once Bliss, Modern, Bliss, Studies; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 10, 50, 25, 80, 01. Recite them before you turn the page.
If you forget the rest, retain Chester Bliss, Franklin, Roosevelt, United States and what they actually built. Quantitative checkpoints: 1933. Recite them before you turn the page.
Salsburg places England, Fisher, Bliss, It at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places In, Bliss, Russian, He at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Leningrad Plant Institute, Bliss, Professor, Tell at the hinge of this stretch: without their stubborn work the method stays a slogan. State the alternative before you chase a star on a p-value.
Salsburg places Well, Bliss at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Bliss, He, Leningrad Plant Institute, Communist at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places One, You, He, She at the hinge of this stretch: without their stubborn work the method stays a slogan. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
Salsburg places Great Britain, To, Denmark, Fisher at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Laplace, It, Gaussian, In at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 150. They are the load-bearing facts of the chapter. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
The story turns on Karl Pearson, Once, Fisher, If; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: fifty measurements. They are the load-bearing facts of the chapter.
If you forget the rest, retain But, To, In, Scandinavia and what they actually built. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
If you forget the rest, retain In, It, Very, If and what they actually built.
If you forget the rest, retain An, Spanish Civil War, Stalinism, Spaniards and what they actually built.
If you forget the rest, retain All, You, At, For and what they actually built.
If you forget the rest, retain And, It, You, But and what they actually built. State the alternative before you chase a star on a p-value.
If you forget the rest, retain Russian, Europe, Hitler, German and what they actually built.
If you forget the rest, retain European, Jarl Waldemar Lindeberg, Finland, Paul Levy and what they actually built. Numbers and dates to keep: 1934, 1948. They are the load-bearing facts of the chapter. State the alternative before you chase a star on a p-value.
Salsburg places Recall, Fisher, Karl Pearson, Once at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1914, 1933. They are the load-bearing facts of the chapter.
O-E-X. Observed vs Expected counts; X (chi-square) measures the gap. Recite: observed, expected, chi-square.
Roll a die 60 times. Expected per face = 10. Compute sum (O-E)^2/E. Compare to a chi-square table with df=5. You just ran Pearson's goodness-of-fit by hand.
Huge n makes tiny, irrelevant deviations "significant." Shadow: a chi-square on millions of rows that rejects a useful approximate model, or binning chosen after seeing the data to manufacture fit.
- Eyeball fit is not fit; use a formal test.
- A significance test can reject, never confirm.
- Fisher's p-value was part of a programme of experiments, not a one-shot verdict.
11. Hypothesis Testing
Egon Pearson's question widened into a deeper one: what does a non-significant result mean? Fisher's answer was clear: nothing proven. Failing to reject is not proof, "no more right to insinuate itself in statistical than in other kinds of scientific reasoning." But Karl Pearson had often used chi-square to "prove" a distribution, and analysts were still tempted to.
Neyman's real insight: alternatives and power
Through letters and summer visits, Neyman and Pearson built the Neyman-Pearson formulation, now frozen in every textbook and in FDA and EPA practice. Salsburg says the textbooks stress the wrong part. Neyman's key discovery: a test makes no sense without at least two hypotheses, the null and the alternative. The alternative decides how the test is run. The probability of detecting the alternative when it is true is the test's power, and the more powerful test is the better one. The class of alternatives cannot be too wide: in 1956 Savage and Bahadur built a small set of alternatives against which no test had any power.
In a drug trial, the null ("the drugs are the same") would never be studied if anyone believed it. It is a straw man to knock down, so the study should be designed to maximise power against realistic differences. This is the logic behind every sample size calculation.
What does probability mean?
A p-value is computed assuming a hypothesis that is probably false. In Salsburg's breast cancer example, two recurrences among eight women, both on placebo, gives about p = 0.30; 250 recurrences out of 1,000, all on placebo, gives p < 0.0001. But the p-value is not the probability the drug fails, nor the probability of error. To compare tests, Neyman needed real-world probabilities of decisions, and fell back on John Venn's 1872 frequentist definition: probability is a long-run proportion (an idea Keynes attacked in 1921). Fix α = 0.05 and you will wrongly reject a true null 5 percent of the time in the long run.
| Fisher | Neyman-Pearson | |
|---|---|---|
| Output | p-value as graded evidence | accept/reject decision at fixed α |
| Alternatives | not formalised | required; define power |
| Probability | evidential, inductive | long-run error frequency |
| Fisher's jibe | science | "acceptance procedures" for the Royal Navy |
Hypothesis testing became the most used tool in science, required by journals, regulators and courts (in discrimination cases). Critics never stopped: Fisher (1955, and Statistical Methods and Scientific Inference), David Cox in the late 1960s, W. Edwards Deming in the 1980s. Neyman himself doubted in 1935 that optimal tests could be found and rarely used tests later. Abraham Wald turned the framework into decision theory; Erich Lehmann's 1959 textbook became its definitive statement.
Neyman at Berkeley
Neyman left Europe just before the war and built Berkeley into a leading department until his death in 1981. He brought David Blackwell, shut out of white universities because of his race, from Howard, and Lucien Le Cam from a French peasant family. He championed women such as Elizabeth Scott and Evelyn Fix, and ended departmental teas toasting "To the ladies!" Fisher attacked him until 1962; Neyman never answered in kind. Once, presenting in French with Fisher in the audience, he braced for an attack that never came. Fisher did not speak French.
Power analysis is now mandatory in trial protocols, but underpowered studies were a main cause of the replication crisis: the Open Science Collaboration (2015) replicated about 36 percent of 100 psychology findings with significant results. Modern fixes follow Neyman: pre-registered hypotheses, equivalence and non-inferiority tests (which make the alternative explicit), and false discovery rate control (Benjamini-Hochberg, 1995) for many tests at once.
N-A-E. Null as foil; Alpha as error budget; Evidence is not the posterior odds. Recite: null, alpha, evidence gap.
Write H0: "this coin is fair." Flip 12 times; get 10 heads. Compute a two-sided p-value (binomial). Decide at alpha=0.05 whether you "reject." Then write in one sentence what you did not compute (the probability H0 is true). That sentence is Neyman-Pearson hygiene.
Treating p<0.05 as proof, or as the probability the null is true, is the standing shadow of this chapter. So is silent multiple testing. The 2016 ASA statement and later debates exist because this shadow won too often.
- No alternative, no meaningful test. Power is the measure of a test.
- A p-value is not the probability that the hypothesis is true.
- Textbook "NHST" is a hybrid neither Fisher nor Neyman endorsed.
12. The Confidence Trick
Coverage is about the procedure, not a posterior probability for one realised interval.
In the 1980s, planners needed the number infected with HIV. Knowing the mean latency (infection to symptoms) and the count of cases would give it, and a tragic group, haemophiliacs infected through blood products with known dates, supplied data. A point estimate of 5.7 years says nothing about precision. An interval estimate, say 3.7 to 12.4 years, does, and here it was too wide for policy: at 12.4 years a fifth of infected people would stay well for 20 years; at 3.7 almost none would.
A late-1980s National Academy committee chaired by John Tukey modelled the effect of fluorocarbons on ozone as a distribution and computed an interval for the annual loss. Even the lower end of the interval implied a serious threat within fifty years. An interval can support action when a yes/no answer cannot.
Neyman's intervals and Bowley's doubt
Neyman introduced confidence intervals in an appendix to his 1934 RSS paper on sample surveys. Bowley, in the chair, suspected the "confidence" was a "confidence trick." His point stands: once data are in, the fixed parameter is either inside the interval or not, so what is the 95 percent a probability of? Neyman avoided the word probability and fell back on frequency: a statistician who always computes 95 percent intervals will be right 95 percent of the time in the long run. It says nothing about this interval.
Fisher, in the same discussion, rambled about fiducial probability, his own unfinished attempt at interval estimates, which matched Neyman's for a normal mean but broke down for harder parameters like the standard deviation. He believed Neyman had stolen and renamed his idea. Neyman's method works for any parameter, and won. "No one talks or writes about fiducial distributions today."
"I am 95 percent sure the parameter lies in this interval" is not what a confidence interval says. That statement belongs to Bayesian credible intervals (Chapter 13), which can differ dramatically from Neyman's from the same data. Polls' "plus or minus 3 percent" are confidence intervals, and they cover sampling error only, not non-response or bad question wording.
Fisher kept circling inverse probability without crossing over. It began with the Reverend Thomas Bayes, who found something in the formulas that alarmed him.
Salsburg wrote that fiducial ideas died with Fisher. Not quite: generalised fiducial inference (Hannig and others) and confidence distributions (Xie and Singh, Schweder and Hjort) revived them in the 2010s. Most consequentially for AI, conformal prediction, a distribution-free way to put guaranteed-coverage intervals around any machine learning prediction, is Neyman's long-run coverage logic in modern form, and it is spreading through model risk teams in banks.
I-C-L. Interval from a procedure; Coverage is long-run; Level is not the probability this interval traps the truth. Recite: interval, coverage, long-run.
Simulate 20 samples of size 10 from a known Normal(0,1). For each, compute a 95% t-interval for the mean. Count how many intervals contain 0. You will usually land near 19/20. That frequency is the coverage story; any one interval either has the mean or does not.
Saying "there is a 95% chance the parameter is in this interval" for a realised frequentist CI is the classic misread. Shadow: stacking CIs from stepwise model selection and still quoting 95%.
- Interval estimates show precision and whether more data is needed.
- A 95 percent CI is a property of the procedure, not of the interval.
- Fiducial inference failed in Fisher's hands but has modern descendants.
13. The Bayesian Heresy
Venice elected its doge through rounds of lot-drawing. In 1268, a child handed thirty lectors balls of wax, nine containing the word lector. Each man's chance began at 9/30, then shifted to 9/29 or 8/29 depending on the first draw: conditional probability. Keynes said all probabilities are conditional, on the books on the shelf, the patient's smoking history, the design of the experiment.
Bayes found that the formulas have an inner symmetry: you can compute the probability of "before" given "after," the chance a deck had four aces given a dealt hand, or a patient smoked given lung cancer. He shelved it; it was published after his death. Yet inversion is often sensible: case-control studies found the effects of smoking and of thalidomide by reasoning from disease back to exposure.
The real heresy is treating parameters as random, so one can say "we are 95 percent sure treatment A beats B." Early in the century "inverse probability" was an accusation; Fisher denied it indignantly. By 2000, over half the papers in Annals of Statistics and Biometrika used Bayesian methods, though medicine stayed wary.
Two kinds of Bayesian
Mosteller and Wallace studied twelve Federalist papers claimed by both Hamilton and Madison. They used about thirty "contentless" words. Hamilton wrote "upon" 3.24 times per 1,000 words, Madison 0.23; eleven disputed papers never used it. Each man's rates are parameters drawn from the distribution of all late-eighteenth-century American writers (hyperparameters), which in turn varies across eras (hyper-hyperparameters). Every method they tried gave all twelve to Madison; for some papers a reader would need prior odds over 100,000 to 1 for Hamilton to end at even odds.
Hierarchical models have since spread through engineering and biology, for instance for mixtures where a hidden variable marks which distribution an observation came from. But Salsburg finds the field confused, "crying for another genius like R. A. Fisher."
Personal probability (Savage and de Finetti, elicited by the "standard gamble") has firmer footing. Savage told a late-1960s audience there are no proven facts, only statements scientists rate highly probable; as anti-war protesters ran past shouting, he added, "we may be the last generation that thinks the world is round." Keynes's version is cultural: the degree of belief an educated person of a time and place would hold.
prior → data → posterior. Black ravens raise your belief that all ravens are black. But a strong enough prior resists data: at Three Mile Island, operators used to faulty warning lights dismissed a consistent picture of low coolant. With two hypotheses, the data reduce to a Bayes factor: posterior odds = prior odds × Bayes factor, so readers can bring their own priors.
The heresy has largely won. MCMC (from the 1990s) and now Stan, PyMC and variational methods make hierarchical models routine. The FDA's January 2026 draft guidance on Bayesian methodology for drug and biologic trials formalises borrowing from prior data; Bayesian adaptive designs drove platform trials such as RECOVERY and REMAP-CAP in COVID-19. Hierarchical "partial pooling" underpins election forecasting and small-area estimation. Machine learning is full of Bayesian ideas (Bayesian optimisation, uncertainty in deep learning), and the frequentist versus Bayesian war has cooled into pragmatic "use what is calibrated." Salsburg's call for a unifying genius remains unanswered.
P-L-P. Prior; Likelihood; Posterior. Bayes multiplies belief by evidence. Recite: prior, likelihood, posterior.
Prior: P(disease)=0.01. Test: sensitivity 0.9, false positive 0.05. You test positive. Compute posterior with Bayes' rule (~15%). Most people guess far higher. That gap is why priors belong in the open.
"Bayesian" as fashion without a documented prior or a decision loss. Shadow: hiding a dogmatic prior in software defaults, or using Bayes to avoid designing the experiment.
- All probability is conditional. Bayes lets you invert the condition.
- Hierarchical models borrow strength across related groups.
- Strong priors can blind you to data: Three Mile Island is the warning.
14. The Mozart of Mathematics
His mother died giving birth on a journey, abandoned by his father; her sisters raised him in Tunoshna and printed his first discovery at five (the sum of the first k odd numbers is k²) in their school magazine Spring Swallows. He learned higher mathematics from an encyclopedia at fourteen, designed perpetual motion machines with hidden errors to tease his physics teacher, and as a Moscow student wrote a history paper on fifteenth-century Novgorod landholding later confirmed by archaeology. Rather than sit fourteen graduate exams he wrote fourteen original papers ("one turned out to be false, but I realized it only later").
After Stalin's death he emerged as warm, funny and curious: teaching gifted children literature and music, hiking with students in his seventies, discussing skiing (and the alternation of fat and thin popes) with John Paul II, reciting Pushkin. He worked on turbulence (1941) and planetary motion (1954, the "K" in KAM theory).
- Foundations (1933): finding a probability is like finding the area of an irregular shape. Using measure theory, he derived all of probability from a few axioms, ending its second-class status.
- Stochastic processes: Fisher assumed independent data, but real series (his son's weekly weights, yearly harvests) are dependent. Kolmogorov's 1938 theory of predicting stationary processes underlies the work of Norbert Wiener (whose classified wartime filtering, Wiener said, followed from Kolmogorov's paper) and George Box, and the present signal processing, seismology and radio astronomy.
What is probability in real life?
The axioms make probability consistent, not meaningful. To apply them, one must identify the abstract space of events in a real experiment. Gosset proposed "all possible outcomes" but could not compute on it. Does a significant AIDS trial mean the drug will work on the next patient, on some percentage, or only in this selected group? From 1965 Kolmogorov attacked the question by treating probability in terms of information and complexity, and died on 20 October 1987 before finishing. Salsburg warns that unless it is solved the statistical approach may collapse under its own inconsistencies.
Salsburg's sentence trails off where Kolmogorov's idea should be named: algorithmic (Kolmogorov) complexity, the length of the shortest program that produces a sequence. A sequence is random if it cannot be compressed. With Martin-Löf randomness and Solomonoff induction, this became a mature field linking probability, information and computation. It now informs minimum description length model selection and theoretical accounts of why large neural networks generalise ("learning is compression"). It has not, however, settled what a p-value means in a clinical trial.
Why Soviet statistics failed
Early Soviet statisticians followed Western work; Vestnik Statistiki described agricultural designs as late as 1924. Under Stalin, statistics became a class science subordinate to planning. The Russian term for random variable, "accidental magnitude," was an insult: in a planned economy nothing happened by accident. Mathematical statistics vanished from official journals; Slutsky fled into astronomy; Kolmogorov and Smirnov were officially "mathematicians." Lysenko's anti-genetics jailed Fisherian biologists. Farms failed, factories made junk, official figures hid behind "rates of change of rates of change," while the West used Kolmogorov's own theorems and Shewhart's quality control. Salsburg dates the end to the Soviet collapse (he writes "late 1990s"; the USSR dissolved in 1991).
Depth pass (source-complete): Hypothesis to Mozart
Salsburg places At, Egon Pearson, Jerzy Neyman, Pearson at the hinge of this stretch: without their stubborn work the method stays a slogan. State the alternative before you chase a star on a p-value.
Salsburg places For, It, Karl Pearson, Fisher at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places In, Egon Pearson, Jerzy Neyman, Fisher at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 12. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
The story turns on Neyman, Pearson, In, Fisher; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on In, Savage, Raj Raghu Bahadur, University; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 1956. They are the load-bearing facts of the chapter. State the alternative before you chase a star on a p-value.
If you forget the rest, retain Unfortunately, Neyman, Fisher, It and what they actually built.
If you forget the rest, retain Recall, Fisher, For, Suppose and what they actually built. Quantitative checkpoints: 50, 30, 500, 250, 0001. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
Salsburg places Since, It, But, Neyman at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1872. Recite them before you turn the page.
The story turns on Venn, In Venn, In, John Maynard Keynes; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1921, 13, 05, 5. Recite them before you turn the page.
If you forget the rest, retain Fisher, Neyman, He, In and what they actually built.
If you forget the rest, retain In, Neyman, Jerzy Neyman, It and what they actually built. Numbers and dates to keep: 1955, 24. They are the load-bearing facts of the chapter.
Salsburg places Neyman, Pearson, As, French at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1935, 1959. They are the load-bearing facts of the chapter.
The story turns on Hitler, Poland, Europe, Neyman; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 1981. They are the load-bearing facts of the chapter. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
If you forget the rest, retain Until, Fisher, Neyman, Everything Neyman and what they actually built. Numbers and dates to keep: 1962. They are the load-bearing facts of the chapter.
Salsburg places Once, Fortunately, Instead at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 14, twenty to, thirty years. They are the load-bearing facts of the chapter.
The story turns on If, For, An, Very; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 5.7, 3.7, 12.4, 20. Recite them before you turn the page.
If you forget the rest, retain In, National Academy, Sciences, Earth and what they actually built. Quantitative checkpoints: 22, fifty years, 44, 3, 41. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
Salsburg places How, Can, In, Jerzy Neyman at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1934. Recite them before you turn the page.
The story turns on Professor, Bowley, He, It; keep them as carriers of the statistical move, not as celebrity wallpaper. State the alternative before you chase a star on a p-value.
The story turns on Bowley, Neyman, Does, Do; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Bowley, Clearly, Neyman, Yet; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 100, 0, 95. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
If you forget the rest, retain Neyman, Fisher, He, It and what they actually built. Quantitative checkpoints: 95. Recite them before you turn the page.
Salsburg places As, Neyman, Bowley, It at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 95, 13, 90. Recite them before you turn the page.
The story turns on No, Fisher, As, In; keep them as carriers of the statistical move, not as celebrity wallpaper.
Numbers and dates to keep: 697, 1797, 150. They are the load-bearing facts of the chapter.
If you forget the rest, retain Early, By, Doge Rainieri Zeno, He and what they actually built. Numbers and dates to keep: 1268, thirty lectors, thirty balls, 9, 30. They are the load-bearing facts of the chapter.
Salsburg places John Maynard Keynes, To, In, Reverend Thomas Bayes at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Suppose, Let, It, If at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 15. Recite them before you turn the page.
The story turns on More, Bayes, For, We; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 95. Recite them before you turn the page. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
If you forget the rest, retain In, Royal Statistical Society, Fisher, Neyman and what they actually built.
If you forget the rest, retain In, Frederick Mosteller, David Wallace, Federalist and what they actually built. Numbers and dates to keep: 1787, 1788, seventy articles, 16, thirty where. They are the load-bearing facts of the chapter. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
Salsburg places Madison, Hamilton, Eleven, Do at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 0.23, 1,000, 3.24, 1.1. They are the load-bearing facts of the chapter.
The story turns on One, English, North America, For; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 24, 1,000, 23, 22, 25. They are the load-bearing facts of the chapter.
If you forget the rest, retain In, English, Hamilton, Madison and what they actually built. Numbers and dates to keep: 20, 1,000, 200. They are the load-bearing facts of the chapter. State the alternative before you chase a star on a p-value.
K-A-C. Kolmogorov axiomatised probability; Almost-sure theorems tamed infinity; Careful measure theory underpinned stats. Recite: Kolmogorov, almost-sure, care.
State the strong law in plain words: averages converge to the mean almost surely under IID assumptions. Then name one real process that violates IID (trending market, drifting sensor). That naming is the bridge from Mozart-of-math to messy data.
Invoking Kolmogorov as a spell that makes any probability statement rigorous. Shadow: writing "almost surely" in a slide deck about a non-stationary business process.
- Kolmogorov gave probability axioms and gave time series theory.
- Consistency is not meaning: the interpretation of probability is unsolved.
- A state that knows the answer in advance cannot use statistics.
15. The Worm's-Eye View
Florence Nightingale was a self-taught statistician who confronted generals and Parliament with graphs showing that most British deaths in the Crimea came from disease and neglect, not battle. Her friends the Davids named their daughter after her: Florence Nightingale David (1909 to 1995), who published ten books and over a hundred papers as F. N. David.
Salsburg says Nightingale invented the pie chart. William Playfair published pie charts in 1801; Nightingale's famous graphic is the polar area ("coxcomb") diagram of 1858. Her real innovation was using graphics as political evidence.
Taught algebra, Latin and Greek by a village parson, steered by her mother away from "godless" University College (Bentham's mummified body sits in its cloisters) to Bedford College, she spent her evenings at the Old Vic for sixpence. Barred from actuarial firms, which took only men, she "crashed my way in to see Karl Pearson," who gave her a scholarship.
Two million turns of the Brunsviga
For Pearson she computed the distribution of the correlation coefficient: Tables of the Correlation Coefficient (1938), ground out on a hand-cranked Brunsviga that she turned "roughly 2 million times," unjamming it with knitting needles rather than face the professor. A motorcycle racer, she crashed into a glass-topped wall; Gosset consoled her by teaching her fly-fishing.
She saw the 1930s "from a worm's-eye point of view": Egon Pearson and Neyman founding Statistical Research Memoirs to avoid upsetting Karl, Fisher "upstairs raising hell," Gosset dropping in every other week. Karl Pearson lectured superbly and welcomed corrections; Fisher's lectures "were awful," and he would not answer a woman's question, so she nudged an American student: "Ask him! Ask him!" She went with Karl as his only assistant on retirement, once doing an elliptic integral overnight after the Chelsea Arts Ball. Neyman made her submit four papers as a PhD: her status changed not at all; she was "just out the twenty-pound entrance fee."
In the war, at the Ministry of Home Security, she modelled casualties and damage to water, power and sewers from bombing, helping London prepare for the Blitz. She saw an early American computer (fuses replaced every thirty seconds) and refused to "learn the language." She wrote the classic Combinatorial Chance (with D. E. Barton) and Games, Gods and Gambling (1962), chaired Berkeley's department after Neyman, and founded the department at UC Riverside in 1970. Locating an archaeological midden from a map of shards was, she noticed, the V-bomb problem again: "there's only about a half dozen of them that are really different."
L-U-N. Local view (worm's eye); Unbiased pieces; Nuisance parameters handled carefully (Neyman structure, similar tests). Recite: local, unbiased, nuisance.
Estimate a rate in each of 50 shops (worm's-eye), then combine. Compare a pooled estimator that ignores shop size with one that weights properly. The "view from each worm" needs a fair aggregator or it misleads the bird.
Obsessing over unbiasedness of a useless estimator, or conditioning on the wrong sigma-field. Shadow: local analyses that never reconcile into a decision.
- Women did much of the field's computing and much of its theory, often without credit.
- The same few model structures recur across wildly different problems.
16. Doing Away with Parameters
In the 1940s Frank Wilcoxon, a chemist at American Cyanamid, noticed that t-tests sometimes failed to detect obvious treatment differences. The culprit: outliers (a cold reactor, an erratic enzyme), which inflate the variance and shrink t. Dropping them broke the test's mathematics. Sure that someone had solved this, he searched the literature, found nothing, worked out a method based on ranks and combinatorics, and submitted it to Biometrics in 1945 half expecting rejection with a reference. It was new.
In 1947 economist Henry Mann and student D. Ransom Whitney, ordering wage distributions, found an equivalent test. These were the first tests that estimated no parameter at all: nonparametric or distribution-free tests. Kolmogorov and Smirnov had already compared whole distributions in the 1930s. Chernoff and I. R. Savage generalised the idea; by the early 1960s it was the hot topic; Jaroslav Hájek's textbooks (Salsburg cites 1971) unified it and tied it back to Lindeberg-Lévy. Wilcoxon moved into statistics, joined Florida State in 1960, and died in 1965.
Two questions remained: how much do you lose using a nonparametric test when the normal model is true, and how far from normal must data be before nonparametric wins? In 1948 E. J. G. Pitman, an unknown 52-year-old in Tasmania, answered both: - When the parametric model is true, nonparametric tests are almost as good (the Wilcoxon test's asymptotic efficiency against t is about 95 percent for normal data). - With slight departures from the model, nonparametric tests are vastly better.
Parametric models (normal, Poisson, Pearson types) earn their power by assumptions. When those assumptions fail - heavy tails, mixed populations, ordinal scores - the parameters become fiction with standard errors. Nonparametric and rank methods were not a fashion; they were an admission that sometimes the safest claim is a comparison of distributions without naming a family.
| Approach | Keeps | Drops / weakens | Typical use |
|---|---|---|---|
| Parametric t / ANOVA | Power, intervals on means | Exact distribution shape | Designed experiments |
| Rank / Wilcoxon | Level (type I) under broad shifts | Easy interval on mean | Skew, ordinal |
| Permutation | Exactness under exchangeability | Simple formula SE | Clear randomisation |
| Bootstrap | Flexible SE / CI | Independence / iid fiction | Estimators with messy form |
Two groups. Rank all values together. Sum ranks in one group. Under no-shift and exchangeability, that sum has a known (or simulable) null distribution. Extreme sums reject. No bell curve required. Fisher's tea-cup spirit returns without Fisher's parametric machinery.
Pitman had no research training, was hired in 1926 with only a bachelor's degree, and promised to lecture on statistics "if appointed," having once decided it was something he "would never have to bother about." He taught twelve lectures a week plus Saturdays and published nothing until 1936.
But there were hidden assumptions. Bahadur and Savage (1956) found the flaw ("a warrior and a savage," Salsburg's Indian friend joked): rank methods help with rare wild values, but with systematic contamination they may make matters worse (Chapter 23).
"Nonparametric" now covers far more than rank tests: the bootstrap (Chapter 28), kernel methods, random forests and gradient boosting, and Bayesian nonparametrics (Gaussian and Dirichlet processes). Deep neural networks, with billions of parameters, behave in some respects like nonparametric estimators. Rank tests remain the default for skewed outcome data in trials and A/B tests.
R-P-F. Ranks beat raw values when shapes mislead; Permutation / nonparametric tests drop parametric baggage; Fewer assumptions, often less power. Recite: ranks, permutation, fewer assumptions.
Two groups of five scores. Compute a t-test. Then replace values by ranks and run a Wilcoxon rank-sum (or permute group labels and see how often a gap as large appears). Compare conclusions. You just did away with (some) parameters.
Nonparametric as assumption-free marketing. Shadow: ranking away a meaningful scale (money, dose) or using a low-power rank test to "show no difference."
- Ranks make tests resistant to wild values at little cost.
- Robustness has limits: systematic contamination defeats simple fixes.
17. When Part Is Better Than the Whole
Pearson's correspondents sent opportunity samples: the skulls that happened to be dug up. On the Bombay docks in the early 1930s, samplers plunged hollow blades into jute bales; the blade deflected off the dense good core and pulled out the damaged outer layer, systematically underrating the jute. Prasanta Chandra Mahalanobis, a wealthy Calcutta physicist who had studied with Pearson and Fisher, used the example constantly. In 1931 he founded the Indian Statistical Institute with his own money and trained C. R. Rao, R. C. Bose, S. N. Roy, P. K. Sen, Madan Puri and others.
| Sample | How chosen | Problem |
|---|---|---|
| Opportunity | whatever is available | unknown, possibly large bias |
| Judgment | chosen to "represent" known groups (Nielsen TV ratings) | needs knowledge you lack; error cannot be measured (Nielsen undercounted Hispanic viewers in 2000) |
| Random | a chance mechanism | individual estimate may be off, but error is calculable with confidence bounds |
The New Deal samplers
Depression graduates poured into Washington. Morris Hansen, with a Wyoming economics degree and a hasty reading of Neyman, designed the first national unemployment survey; Jerome Cornfield, a CCNY history major, joined Labor. When a survey showed under 10 percent of people got almost 40 percent of income, the Chamber of Commerce dismissed it as "only a random collection of data." For the 1937 Unemployment Census, estimates ranged from 3 to 15 million jobless; Hansen sampled 2 percent of postal routes at random, and on Neyman's advice ignored most of the detail and simply counted. Hansen and Hurwitz became so linked that people thought "Hansen Hurwitz" was one person. Their work grew into the CPI, the Current Population Survey and the economic indicators every industrial nation copies.
For Wassily Leontief's input-output model of the economy, Cornfield inverted a 12 × 12 matrix by hand in a week; a 24 × 24 would take "several hundred years." Harvard's Mark I could do it, but government rules paid for goods, not services. Cornfield's fix: a purchase order for capital goods, "one matrix, inverted." When Leontief won the 1973 Nobel, he phoned Cornfield to thank him.
Cornfield also solved the mathematics of case-control studies and helped design the Framingham Heart Study (1948), whose 1967 paper first linked high cholesterol to heart disease. In India, Mahalanobis's surveys, trusted by Nehru, checked the excesses of central planning, in contrast with Soviet invented numbers. In 1962, after the ISI's thirtieth-anniversary meeting, Fisher died of a heart attack on the voyage back to Australia (he died in Adelaide), aged 72.
Random sampling's great enemy is now non-response: telephone survey response rates fell into single digits, so pollsters rely on weighting, multilevel regression and post-stratification (MRP) and online opt-in panels, which revive Mahalanobis's worry about non-random samples. Xiao-Li Meng's "big data paradox" (2018) shows a huge biased sample can be worth less than a small random one. Official statistics agencies are blending surveys with administrative and transaction data.
S-U-E. Sample a Useful subset; Efficiency can beat a bad census. Recite: sample, useful, efficiency.
A register of 10,000 customers is half stale. Design a probability sample of 400 with callbacks. Compare expected error to a "census" that quietly misses the stale half. Part can beat the whole when the whole is fiction.
Convenience samples dressed as probability samples. Shadow: web scraping 2 million rows and claiming superiority to a designed 2,000 because "big."
- Random samples win because their error is measurable, not because they are always right.
- Clever administration (one matrix, inverted) can matter as much as clever maths.
18. Does Smoking Cause Cancer?
In 1958 Fisher, a pipe smoker, published papers attacking the evidence that smoking causes lung cancer, and later accused researchers of doctoring data. Joseph Berkson of the Mayo Clinic and Neyman had objections too, but were satisfied as evidence mounted. The tobacco industry called it "only statistical correlation." In 1959 Cornfield and five cancer experts reviewed every study and objection and concluded smoking is "a causative factor." After 1960 no reputable journal questioned it.
What is a cause?
Salsburg argues Fisher was being consistent, not merely crotchety. Bertrand Russell had shown that the everyday idea of cause and effect is inconsistent and cannot be written in symbolic logic. Russell proposed material implication (A implies B, equivalently not-B implies not-A). Koch's postulates for infection are a version of it. Smoking fails them: some smokers never get cancer, some non-smokers do, and quitting does not guarantee escape. Almost no chronic disease meets them.
- Doll and Hill (1952), case-control: nearly ten times as many smokers among lung cancer patients; five similar studies on three continents agreed.
- Prospective studies: Doll and Hill's British doctors and Hammond and Horn's 187,783 American men showed a dose response, heavier smokers more cancer. (Salsburg says Hammond and Horn followed the men for "four months"; their 1958 report covered about 44 months.)
- Dorn (1958): all deaths in three cities, not an opportunity sample.
- Animal and tissue toxicology showing tobacco smoke carcinogenic.
Every study had a flaw a critic could name; it was the overwhelming consistency across designs and populations that proved the case. Neyman's objection about age was answered with the patients' own data.
Salsburg contrasts Agent Orange: the same few exposed men studied repeatedly, other populations (Seveso, Italy) negative, diffuse effects rather than a specific lesion, and no measurable dose, so the objections of Fisher, Berkson and Neyman could not be answered.
Fisher's argument
Fisher suspected publication bias and selective reporting. He noted that in Doll and Hill's first study non-inhalers had more cancer than inhalers, and mocked them for not advising smokers to inhale. He argued that inference from designed, randomised experiments is logically sound, but epidemiology applied his tools to non-randomised data where "treatment" is self-chosen. A common genetic cause might drive both smoking and cancer; he showed identical twins tend to share smoking habits.
In the 1959 paper Cornfield answered Fisher with a now-famous argument: if smokers have nine times the lung cancer risk, a hidden confounder would have to be at least nine times more common among smokers to explain it all away. No plausible gene could do that. This was the first sensitivity analysis for unmeasured confounding.
Fisher demanded randomisation; Cornfield, a self-taught history graduate, said accumulating evidence suffices when experiments are impossible. Their descendants still argue in courts, in environmental science and in medicine. Cause and effect are not so simple to prove.
The field Salsburg only gestures at has exploded. Judea Pearl's causal diagrams and do-calculus and the Rubin potential-outcomes framework give formal languages for cause, refuting Russell's pessimism. Mendelian randomisation uses genetic variants as natural experiments, a direct answer to Fisher's genotype argument; MR studies of smoking support causation. The 2021 Nobel in economics (Card, Angrist, Imbens) honoured natural-experiment methods. E-values (VanderWeele and Ding, 2017) generalise the Cornfield inequality. Target trial emulation now structures observational studies like randomised ones. Salsburg also omits that in the late 1950s Fisher served as a consultant to the Tobacco Manufacturers' Standing Committee, the UK industry's research body; his defenders argue his letters show sincere concern about statistical misrepresentation.
Depth pass (source-complete): Worm to smoking
Salsburg places Florence Nightingale, English Victorian, Parliament, British at the hinge of this stretch: without their stubborn work the method stays a slogan. State the alternative before you chase a star on a p-value.
Salsburg places Ivington, Davids, David, Florence Nightingale David at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 100, 1909. They are the load-bearing facts of the chapter.
The story turns on Nan Laird, Harvard School, Public Health, But; keep them as carriers of the statistical move, not as celebrity wallpaper. State the alternative before you chase a star on a p-value.
The story turns on She, Someone, Karl Pearson, University College; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1938, 2. Recite them before you turn the page.
If you forget the rest, retain She, William, Gosset, Well and what they actually built. Quantitative checkpoints: 16. Recite them before you turn the page.
Salsburg places David, University College, London, Neyman at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1940. Recite them before you turn the page.
The story turns on David, Karl Pearson, He, Fisher; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1933. Recite them before you turn the page. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
If you forget the rest, retain Karl Pearson, He, On, Oh and what they actually built. Quantitative checkpoints: 6, 00. Recite them before you turn the page.
Salsburg places Looking, Mr, Neyman, But at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1939, 1940, 1941. Recite them before you turn the page.
The story turns on American, Andrews Air Force, Base, It; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 100, thirty seconds. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
If you forget the rest, retain All, Barton, University College, So and what they actually built. Quantitative checkpoints: 1970, 1977, 1988, 1995. Recite them before you turn the page.
Salsburg places In, David, Games, Gods at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1962. Recite them before you turn the page.
The story turns on And Florence Nightingale, David, Frank Wilcoxon, American Cyanamid; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 17. Recite them before you turn the page.
If you forget the rest, retain He, Sometimes, It, Wilcoxon and what they actually built.
If you forget the rest, retain Frank Wilcoxon, Surely, He, Wilcoxon and what they actually built. Numbers and dates to keep: 1945. They are the load-bearing facts of the chapter.
Salsburg places Wilcoxon, Biometrics, Henry, Mann at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1940, 1944, 1947, 18. They are the load-bearing facts of the chapter. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
Quantitative checkpoints: 1971, 1974, 1960, 1965. Recite them before you turn the page. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
If you forget the rest, retain If and what they actually built.
If you forget the rest, retain If, In, Annals, Mathematical Statistics and what they actually built. Numbers and dates to keep: 1948. They are the load-bearing facts of the chapter. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
Salsburg places Pitman, Melbourne, Australia, He at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1897. They are the load-bearing facts of the chapter.
The story turns on Pitman, Theory, Statistics, He; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 1927, 1926, 1936. They are the load-bearing facts of the chapter.
If you forget the rest, retain By, Annals, Mathematical Statistics, University and what they actually built. Numbers and dates to keep: 1948, 1936, 30,000, 2,400. They are the load-bearing facts of the chapter. State the alternative before you chase a star on a p-value.
Salsburg places By, Pitman, How, Not at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1948. They are the load-bearing facts of the chapter.
The story turns on If, Pitman, It, Frank Wilcoxon; keep them as carriers of the statistical move, not as celebrity wallpaper. State the alternative before you chase a star on a p-value.
The story turns on Deep, Wilcoxon, Mann, Whitney; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1956, 23. Recite them before you turn the page.
If you forget the rest, retain To Karl Pearson, He, Biometrika, Pearson and what they actually built. State the alternative before you chase a star on a p-value.
If you forget the rest, retain India, Bales, Bombay, Europe and what they actually built. Numbers and dates to keep: 1931. They are the load-bearing facts of the chapter.
Salsburg places At, Indian Statistical Institute, Indian, Roy at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 19. They are the load-bearing facts of the chapter.
The story turns on But, If, In, Nielsen Media Research; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 2000. They are the load-bearing facts of the chapter.
If you forget the rest, retain Thus, Indian Statistical Institute, Mahalanobis, Neyman and what they actually built.
If you forget the rest, retain Margaret Martin, Yonkers, New York, Barnard College and what they actually built. Quantitative checkpoints: 1932, 1939, 1933, 1934, 48. Recite them before you turn the page.
Salsburg places Martin, Albany, New York State, Division at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1933. Recite them before you turn the page. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
The story turns on It, Washington, First, Surveys; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 10, 40, 1. Recite them before you turn the page.
If you forget the rest, retain In, Congress, Unemployment Census, At and what they actually built. Quantitative checkpoints: 1937, 2. Recite them before you turn the page.
A-C-B. Association is not causation; Confounders mislead; Bradford Hill / design / modern causal tools bridge the gap. Recite: association, confound, bridge.
List three confounders that could link "carrying a lighter" to lung cancer without lighter fluid causing tumours. Then state what evidence would still convince you smoking causes cancer (dose-response, cessation effects, biological mechanism). That list is Hill's criteria in your own words.
Either nihilism ("observational data never causal") or slogans ("correlation = causation in big data"). Shadow: a dashboard attribution model that ignores collider bias and ships as truth.
- Causation can be established from many flawed studies that agree, with dose-response and mechanism.
- Sensitivity analysis asks how strong a hidden confounder would need to be.
- Randomisation remains the gold standard; modern causal inference is about what to do without it.
19. If You Want the Best Person ...
In 1913 George Snedecor set out for the University of Iowa, got lost, and landed in Ames at Iowa State, which hired him to teach algebra. He built the first statistics department in the US and wrote Statistical Methods (1940), which improved on Fisher by adding derivations and tables; in the 1970s it was the most cited book in science. Fisher lectured at Ames most summers of the 1930s.
Gertrude Cox, who had spent seven years in Methodist social work aiming to be a missionary, was talked into statistics by Snedecor, earned Iowa State's first master's in statistics (1931), taught the first experimental design courses, and with William Cochran wrote Experimental Designs (1950), still filling a column of the Science Citation Index each year.
Asked by UNC president Frank Graham to recommend a head for a new statistics unit at North Carolina State, Snedecor listed ten men and showed the list to Cox. "What about me?" she asked. He added: "These are the ten best men I can think of. But, if you want the best person, I would recommend Gertrude Cox."
Cox built a department of fine teachers; by 1946 UNC Chapel Hill added mathematical statistics and then biostatistics, and the Research Triangle (NC State, UNC, Duke) became a world centre. Her students still run a road race in her honour.
Women in statistics
Early women were the "computers" in rows at Pearson's lab. Neyman changed that; by the 1990s about half of meeting attendees were women, though only about 30 percent of journal papers had a female author and 13 percent of ASA fellows were women.
Janet Norwood married at 19, earned a Tufts PhD (1949), took a part-time job at the Bureau of Labor Statistics to be home three afternoons a week, ran the Consumer Price Index by 1970, oversaw its 1978 overhaul, and was the first woman Commissioner (1979 to 1991). Her first act: barring the Labor Department's policy representative from reviewing BLS press releases. "In government, you have the independence that you assert and insist upon ... How do you handle situations in which you have to correct the President of the United States? We did that."
The CPI is a Pearsonian parameter: no family's costs equal it, and its construction (market basket weights, random stores, new products, substitution) involves arbitrary definitions with "no R. A. Fisher to establish optimum criteria." Who counts as "unemployed" is a choice. Salsburg ends with women theorists: Nancy Mann (estimation for the Weibull distribution in aerospace), Grace Wahba (the theory of smoothing splines), and Yvonne Bishop (log-linear models for tables of counts, from the halothane liver study). The best person has frequently been a woman.
Norwood's principle was tested in August 2025, when the US president fired BLS Commissioner Erika McEntarfer after a weak jobs report with large downward revisions, prompting protests from former commissioners and statistical societies about political interference. Around the world, agencies are dealing with falling survey response rates and budget cuts. Trust in official statistics, which bank risk and monetary policy depend on, is again a live issue.
S-B-R. Selection of the best; Beating chance rankings; Replication and shrinkage. Recite: select, beat chance, replicate.
Simulate 10 players with equal skill; give each 20 random scores. Rank by mean. Note the "best" looks special. Repeat. The identity of the best churns. That is why "best person" needs a model of noise, not a single leaderboard.
Winner-take-all hiring or agent ranking on noisy evals. Shadow: crowning a model champion on one benchmark seed and shipping it as universally best.
- Great synthesisers and administrators spread a science as much as inventors do.
- Official numbers are only as good as their independence from politics.
20. Just a Plain Texas Farm Boy
Samuel S. Wilks left a Texas farm for Iowa in the late 1920s, when mathematics was racing into pure abstraction. His topology teacher said applied mathematics ranked with washing dishes (Salsburg recalls Euclid tossing a coin to a pupil who asked what a proof was good for). His adviser Everett Lindquist gave him a statistics problem from educational psychology instead.
At Princeton from 1933, protected by chairman Luther Eisenhart, Wilks edited the Annals of Mathematical Statistics, raised its standards, recruited John Tukey, helped design testing and scoring at the Educational Testing Service, and worked with Walter Shewhart of Bell Labs on quality control. His "just folks" manner disarmed graduate students and generals alike; "there is no record of anyone getting angry at Samuel S. Wilks."
Wilks set up the Statistical Research Group, Princeton, with Tukey, Frederick Mosteller, T. W. Anderson, Alexander Mood and Charles Winsor. They worked on proximity fuses, early guided bombs, range finders, and on clearing Japanese non-metallic land mines with explosive cord; Richard Anderson recalled that the failure of those experiments was one reason the atomic bomb was used. A sister group at Columbia produced sequential analysis (Abraham Wald), which lets an experiment be changed or stopped as results arrive without invalidating conclusions. It was classified until after the war.
The pull of abstraction
After the war Wilks argued that useless abstraction wasted the nation's brainpower. Yet statistics itself grew more abstract: Wald's decision theory, group-theoretic designs too complex to use, nested sigma fields. By the 1980s universities created biostatistics and applied departments; Biometrics, founded for applied work rejected by Biometrika, became so theoretical that Statistics in Medicine was founded. Mathematics departments, Salsburg says, missed statistics, then computers, then molecular biology. Wilks died in 1964 at 58; the ASA's Wilks Medal honours creativity applied to the real world.
Wald's sequential tests now run every major A/B testing platform ("always-valid" p-values and confidence sequences let teams peek safely) and adaptive clinical trials with interim analyses. The ICH E20 guideline on adaptive designs (draft endorsed June 2025; consultation closed November 2025) brings them into global regulation. The "applied versus abstract" tension Salsburg describes has largely been resolved by data science, which absorbed applied statistics, sometimes at statistics departments' expense.
D-S-P. Decision theory; Savage axioms; Personal probability as coherent bets. Recite: decision, Savage, personal probability.
Price a bet: win $100 if it rains tomorrow. Write your buying and selling prices. If they straddle incoherently, a Dutch book can punch you. That discomfort is Savage's point in a wallet.
Coherent posteriors used to justify any action because the math closed. Shadow: elegant decision theory with a loss function nobody agreed to.
- War made statistics a national resource.
- Sequential analysis: look as you go, but with rules.
- Fields that disdain application cede new territory to others.
21. A Genius in the Family
Two children of immigrants. Moses Goodack fled the czar's army with 35 roubles and a cheese, became a watchmaker in Whitechapel, and opened an antique jewellery shop; a drunk sign-painter made it "Good's Cameo Corner," and the family became Good. His son I. J. (Jack) Good, born 1916, lying ill with diphtheria at about ten, discovered in his head the irrationality of √2 and a solution of Pell's equation: "being anticipated by great men is now familiar to me, but it was not usually by 2.5 millennia." At school he finished a blackboard of hard exercises as the master finished writing them, and rediscovered mathematical induction.
After a Cambridge PhD (1941) he joined Bletchley Park. Salsburg explains why changing-cipher machines are breakable: each level of shifting is governed by parameters, then hyperparameters, and ultimately a fixed level the recipient must know, much like a hierarchical Bayes model. Good carried this into empirical and hierarchical Bayes methods, then returned to intelligence work with early computers and classification theory, collecting a Cambridge Sc.D. and an Oxford D.Sc. before moving to Virginia Tech (1967 to 1994). He loved coincidences (he arrived in Blacksburg at the seventh hour of the seventh day of the seventh month of '67 and was given apartment 7, block 7) and asked when an apparent pattern really means something.
Born in New York in 1945 to Greek immigrants, a magician from age five, Persi Diaconis ran away at fourteen to tour with the sleight-of-hand master Dai Vernon, then performed as "Persi Warren." At 24, tired of the Catskills circuit, he enrolled at CCNY to understand Feller's probability textbook, discovering his teachers had graduated him from high school in absentia. Harvard statistics PhD (1974), then Stanford.
The computer changes the questions
Computers first did the old analyses faster. The Stanford group asked what new methods they allowed. A heart study might yield 5,000 measurements on each of 20,000 patients: points in a 5,000-dimensional space. Bellman's "curse of dimensionality" says estimation becomes hopeless beyond 10 to 20 dimensions. But real data are not spread through all those dimensions; correlated measures bunch onto lower-dimensional structures (Tukey guessed medical data rarely exceed five true dimensions). Projection pursuit searches for those low-dimensional views. Meanwhile, computer scientists began rediscovering statistics under the name of information science.
The insight that data live near low-dimensional manifolds became the foundation of modern machine learning: principal components, t-SNE and UMAP for visualisation, and learned embeddings in deep networks and LLMs, which map text and images into spaces of a few thousand dimensions where meaning has geometry. Projection pursuit's descendants include independent component analysis and the "manifold hypothesis" behind deep learning's success. Diaconis, now at Stanford, became famous for showing seven riffle shuffles randomise a deck.
G-F-Q. Genius clusters in stories more than genes alone; Family data need careful models; Quite a lot of mythology surrounds IQ dynasties. Recite: genius, family, question myths.
Sketch a DAG: parent genotype -> child genotype; parent environment -> child environment; both -> measured achievement. Mark what a raw parent-child correlation cannot separate. That sketch is why "genius in the family" is harder than a headline.
Using heritability estimates as destiny or policy bludgeons. Shadow: ignoring gene-environment interplay and measurement politics around IQ.
- Codebreaking, Bayes and computing grew up together at Bletchley.
- Big data is usually low-dimensional underneath; find the structure.
22. The Picasso of Statistics
In 1966 John Tukey met job candidate Salsburg at Princeton station in chinos and sneakers, chatted about round-off errors, sat in the back row grading papers during the talk, then walked down and rewrote Salsburg's main theorem from memory with an alternative proof of what had taken him months. "This is the big leagues!"
Born in 1915 in New Bedford, home-schooled, Tukey studied chemistry at Brown, earned a Princeton mathematics PhD (1939) with Tukey's lemma in topology, and was pulled by Wilks into statistics. Wartime fire-control work gave him his creed: "It is better to have an approximate answer to the right question than an exact answer to the wrong one." Like Picasso, each period could have been a career.
| Work | What it changed |
|---|---|
| Spectrum analysis and the FFT | radar tracking errors at Bell Labs led to methods that "borrow strength" across frequencies; the Cooley-Tukey FFT (1965) made Fourier analysis fast |
| Exploratory data analysis (EDA) | look at the data before imposing a model; box plots, stem-and-leaf plots, rootograms |
| Robust estimation | resistance to outliers (Chapter 23) |
| Words | coined "bit" and popularised "software" |
| Tallying | a ten-mark box-and-cross tally where errors are visible |
The chapter merges two things. Tukey's spectral smoothing methods (with Blackman, 1958) "borrow strength" from neighbouring frequencies. The fast Fourier transform of Cooley and Tukey (1965) is an algorithm that computes the discrete Fourier transform quickly; it does no smoothing. Gauss had found a version around 1805.
Seeing patterns in noise
Bell Labs in the 1960s faced millions of records from phone lines and space probes. E. T. Bell: numbers "tell the truth with intent to deceive." Humans see patterns in noise, which is why disease clusters (a leukaemia cluster near a dumping site) are so hard to judge: some town somewhere will always have a cluster. Tukey's EDA treats the data's distribution as an object to examine, with new names to jolt assumptions and graphics that resist the pull of extremes. He died in July 2000, still challenging everyone.
Tukey's 1962 "The Future of Data Analysis" is now read as the founding text of data science. His box plot is in every analytics tool. The danger he flagged, finding patterns in noise, is now called p-hacking, the garden of forking paths (Gelman and Loken) and, in AI, spurious correlations learned by models. Good practice separates exploratory analysis from confirmatory tests on held-out data.
Depth pass (source-complete): Best person to Picasso
Salsburg places Late, George, Snedecor, University at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1913, 1940. Recite them before you turn the page.
The story turns on Snedecor, He, Ames, For; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1931. Recite them before you turn the page.
If you forget the rest, retain In, Ames, Iowa, William Cochran and what they actually built. Quantitative checkpoints: 1950. Recite them before you turn the page. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
Salsburg places As, Jerzy Neyman, Ph, By at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 30, 13, 1940, 25. Recite them before you turn the page.
The story turns on Weeks, Graham, Snedecor, He; keep them as carriers of the statistical move, not as celebrity wallpaper. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
The story turns on Gertrude Cox, She, Her, American Statistical Association; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 1946. They are the load-bearing facts of the chapter.
If you forget the rest, retain Women, America, Census Bureau, Bureau and what they actually built. Numbers and dates to keep: 1991, 1949, 1979. They are the load-bearing facts of the chapter.
Salsburg places Department, Labor, Bureau, Labor Statistics at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1979. They are the load-bearing facts of the chapter. State the alternative before you chase a star on a p-value.
The story turns on Ph, For, European Common Market, Washington; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Labor Statistics, Census Bureau, Janet Norwood, By; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1963, 1970, 1978. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
If you forget the rest, retain It, Mathematical, Once and what they actually built. Quantitative checkpoints: 0.2. Recite them before you turn the page.
Salsburg places In, Attempting, How, Janet Norwood at the hinge of this stretch: without their stubborn work the method stays a slogan. State the alternative before you chase a star on a p-value.
Salsburg places Fisher, In, Arbitrary, United States at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 6. They are the load-bearing facts of the chapter.
The story turns on Grace Wahba, University, Wisconsin, Yvonne Bishop; keep them as carriers of the statistical move, not as celebrity wallpaper. State the alternative before you chase a star on a p-value.
The story turns on Since, Snedecor, Cox, In; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Nowhere, American, American Mathematical Society, As Sam Wilks; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Sam Wilks, Everett, Linquist, Ph; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1933. Recite them before you turn the page.
If you forget the rest, retain United States, In, Advanced Studies, Joseph and what they actually built. Quantitative checkpoints: 1939, 26. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
Salsburg places Wilks, He, Soon, Princeton at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 27. Recite them before you turn the page.
The story turns on As, Wilks, Office, Naval Research; keep them as carriers of the statistical move, not as celebrity wallpaper. State the alternative before you chase a star on a p-value.
The story turns on Members, Wilks, Statistical Research Group, Princeton; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Sequential, Wilks, World War, One; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Sam Wilks, He, Greek, Plato; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 28. They are the load-bearing facts of the chapter.
If you forget the rest, retain Abraham Wald, Fisher, Finally, Andrei Kolmogorov and what they actually built.
If you forget the rest, retain In, To, Attempts, Meetings and what they actually built. Quantitative checkpoints: 29, 28. Recite them before you turn the page.
Salsburg places Samuel, Wilks, His, American Statistical Association at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1964, fifty years. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
The story turns on Moses Goodack, Polish, In, West; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 9, 1916. Recite them before you turn the page.
If you forget the rest, retain In, David Banks, Jack Good, He and what they actually built. Quantitative checkpoints: 1993. Recite them before you turn the page.
Salsburg places In, Jack Good, It, Hardy at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1993, 2.5, 30, 10. Recite them before you turn the page.
The story turns on Good, Confronted, In, At; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 300, 1941. Recite them before you turn the page. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
If you forget the rest, retain Jack Good, Bletchley Park, London, German and what they actually built. Quantitative checkpoints: 1940, 12, 06, 14, 09. Recite them before you turn the page.
Salsburg places All, Bayes, Eventually, One at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 13. Recite them before you turn the page.
The story turns on British Intelligence, Good, Sc, Cambridge; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1967, 1994, 7. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
G-V-I. Graphical / geometric insight; Variance structures seen as shapes; Innovation in visualisation and theory (the "Picasso" spirit). Recite: geometry, variance, insight.
Plot two variables; draw the least-squares line; sketch residual cloud. Rotate axes mentally toward principal components. That geometric move is the Picasso chapter's attitude: see structure, do not only compute scalars.
Pretty charts that hide uncertainty, or geometry used as intimidation. Shadow: a biplot nobody can read used to end debate.
- Approximate answers to the right question beat exact answers to the wrong one.
- Explore first, confirm separately.
- Clusters appear by chance; judge them against what randomness produces.
23. Dealing with Contamination
Statistical theorems assume every measurement is equally valid. Great scientists did not: Stephen Stigler found that Michelson, Kepler and others discarded inconvenient data. Modern scientists are taught never to. So what do you do when values are obviously wrong? In 1972 a pharmacologist brought Salsburg a rat ulcer study in which two rats on the worse compound had no ulcers at all, wrecking both t-tests and rank tests. The answer was in a new book on Salsburg's desk: the Princeton Robustness Study.
George Box
The word "robust" came from George Box, son of a shopkeeper's assistant, who could not afford university and was studying chemistry at a polytechnic when the war came. At the chemical defence station he told a colonel the gas data needed a statistician. "What do you know about it?" "I once tried to read a book called Statistical Methods for Research Workers." "Well, if you read the book you'd better do it."
An American ophthalmologist arrived with a "perfect" antidote for lewisite, backed by sheaves of badly confounded rabbit experiments. Box used the rabbit's two eyes as a correlated pair in a block design he had derived himself, and quickly showed the antidote was useless. Officials cut his appendix as too complicated, until Sir John Gaddum marched him into the committee and told them to "put the bloody thing back in."
After the war Box sought Fisher at UCL, found Fisher had moved to Cambridge, and was told by Egon Pearson, "You'll learn there were one or two other people besides Fisher in statistics." After a PhD he worked at ICI (1948 to 1956) on response surfaces and process improvement, then Princeton, then founded the Wisconsin department. He married Fisher's daughter Joan Fisher Box, whose 1978 biography of her father is the standard life.
Contaminated distributions
Box kept "robust" deliberately vague; others defined it. Efron (1968) showed Student's t is robust in its error rates. Tukey's Princeton study (1972) modelled contaminated distributions: most data from the distribution of interest, a few from another. In a wartime Navy test of a stereoscopic range finder, about a fifth of sailors could not see stereoscopically, so a fifth of readings were wrong and unidentifiable. Huge Monte Carlo runs showed the mean is wrecked by even one contaminant (Yale's 1950s alumni income survey: over 80 percent earned less than the average). The goal: estimators that downweight contaminants if present but give correct answers if not. Salsburg's pharmacologist used one, and later experiments confirmed the result.
Box and David Cox joked about the Victorian farce in which "Box and Cox" share one bed by day and night, and resolved to write a paper together. Their interests diverged, but in 1964 they published the Box-Cox transformation, which reshapes data so standard methods work better, now used from toxicology to econometrics. Box also gave the field its most quoted line: "All models are wrong, but some are useful."
Robust statistics (Huber's M-estimators, Hampel's influence functions) is now central to machine learning: data poisoning and label noise are contamination by another name, and adversarial robustness asks Box's question about inputs. And Box's warning about models drives the present emphasis on out-of-distribution testing for AI systems.
O-R-M. Outliers happen; Robust methods limit their pull; Model contamination explicitly. Recite: outliers, robust, model contamination.
Mean of {1,2,3,4,100} vs median vs 10% trimmed mean. Watch which shifts. Contaminate 10% of a Gaussian sample in a spreadsheet and compare OLS to a robust regression. That is Tukey territory.
Deleting outliers until the p-value cooperates. Shadow: "robust" as a button that hides the contamination you should have investigated (fraud, sensor fail, regime change).
- The mean is fragile; robust estimators protect against hidden contamination.
- Good design (paired eyes, blocks) beats volumes of bad data.
- Never silently discard data; use methods built for contamination.
24. The Man Who Remade Industry
In 1980 NBC aired "If Japan Can, Why Can't We?", about how an 80-year-old American statistician, W. Edwards Deming, had helped Japanese industry overtake America. Ignored by Detroit, Deming had been invited to occupied Japan in 1947 as a sampling expert, then lectured for the Union of Japanese Scientists and Engineers (JUSE) to engineers and senior managers. He told them they could transform "made in Japan" from a byword for junk within five years; they did it in about three. Japan created the Deming Prize and a national Statistics Day.
Salsburg's host, Ichiro Ishikawa, led JUSE and later the Keidanren business federation. His son Kaoru Ishikawa became the famous quality engineer behind quality circles and the fishbone (cause-and-effect) diagram. The two are often confused.
"Workers" scoop 50 beads from a drum in which one-fifth are red; management demands no more than 3 red. The chance of 3 or fewer is under 1 percent; the average is 10. Managers praise the "good" workers, scold the "bad," and tell the bad to copy the good, though every result is pure chance. Deming's lesson: most variation comes from the system, which only management can change; slogans like "zero defects" and fads like TQM are exhortation, not management.
Using Fisher's analysis of variance, Deming split variability into special causes (rare, identifiable, fixable on the line) and common causes (poor machines, variable supplies, working conditions), which are management's job. Customers want reliable products with low variability, not perfect ones. Map the process as a stream, find the most variable step, reduce it, repeat: continuous improvement.
Walter Shewhart (Bell Labs) and Frank Youden had brought quality control to industry in the 1920s and 1930s; Deming carried it to the boardroom. Born in Sioux City in 1900, a Yale physics PhD, he saw Shewhart's ideas reduced to rote limits at Western Electric's Hawthorne plant (a 5 percent defect rate meant "5 percent of customers dissatisfied"), worked at the USDA on sampling (1927 to 1939), and trained about 2,000 wartime engineers whose students reached nearly 30,000. Stern in public, kind in private, he sang, played drums and flute and wrote a more singable "Star-Spangled Banner." He ran his last seminar on 10 December 1993 and died ten days later.
Deming scorned hypothesis testing: with enough repetitions any difference becomes significant; what matters is the size of the difference and whether it holds in another setting. "Statisticians need to become interested in problems."
Deming's ideas live on in lean, Six Sigma and the Toyota Production System, and, less obviously, in DevOps and site reliability engineering, where teams track process variation in deployments and incidents. Statistical process control charts now monitor AI model drift in production.
P-D-S. Process focus; Deming's system view; Statistical process control. Recite: process, Deming, SPC.
Track a daily metric for 20 days; draw a run chart with median and simple limits. Mark points outside. Ask whether you tampered with a stable process or missed a special cause. That is Deming's habit in miniature.
Control charts used as blame charts, or "quality" slogans without psychological safety. Shadow: tampering (reacting to common-cause noise) and calling it management.
- Most variation is systemic; blaming individuals for it is futile.
- Reduce variability step by step; do not just inspect at the end.
- Ask how big an effect is, not only whether it is significant.
25. Advice from the Lady in Black
By the 1960s women headed statistics groups across industry and government. Salsburg profiles Stella Cunliffe, the first woman president of the Royal Statistical Society, through her 1975 presidential address, a plea to spend less time on abstract theory and more working with other scientists ("barley trials ... could be knocked for six by some fool of a tractor driver hurrying home to his tea via a short cut across the plot").
After the LSE in the late 1930s and a Depression job at the Danish Bacon Company, she worked on wartime food allocation, then volunteered in Rotterdam as the Germans surrendered, at Bergen-Belsen after liberation, and in displaced-persons camps. Penniless, she joined Guinness in 1946, where Gosset's experimental discipline still ruled. The supervisor of the Ladies Staff, dressed all in black with whalebone lace collar, told her to wear stockings and a hat and to lower her eyes if a brewer passed in the corridor.
- Crown seals: an elaborate beer-temperature preference test found only that drinkers disliked yellow crown seals.
- Beer casks: too many casks just passed the size limit and too few just failed. The inspector had to heave rejects onto a high pile. Putting her station over the reject bin, so she could just kick them in, made the bias vanish.
- Prison sentences: at the Home Office Research Unit (from 1970), short sentences seemed to cause recidivism. The short-sentence group was dominated by "old, pathetic, sad and mad" people counted again and again; the long-sentence group by fraudsters who rarely reoffend. Adjusting for both, the relationship disappeared.
The chapter's centre is not a single theorem. It is the professionalisation of statistics as counsel: Gertrude Cox and her circle built training, labs and consulting norms so that design met subject-matter questions. Advice fails when it is either pure math with no decision or pure storytelling with no estimand.
- Decision: what action will this change?
- Estimand: what quantity answers that decision?
- Design: can the data identify it, or only hope?
- Assumptions: which are load-bearing?
- Audience: what uncertainty can they use?
A team asks for "significance on the dashboard." The Cox-style answer is to ask which weekly decision the star would change. If none, the request is decoration. If one exists, design the metric and sample for that decision; do not sprinkle p-values on charts built for browsing.
Cunliffe went into the hop gardens at 7 a.m., insisted the statistician share the experimenter's enthusiasm, and found sociology's standards of design a shock. She blamed statisticians for not serving softer disciplines, and urged them to persuade in plain language rather than "solemnly quote p less than 0.001." Salsburg adds that some of his own best work was talking people out of doomed experiments.
A-D-V. Advice as theory-plus-practice; Data without a question fail; Visibility of assumptions (the lady in black's style). Recite: advice, data+question, visible assumptions.
Before the next analysis, write one sentence naming the decision the statistic will change. If you cannot, you are decorating. That sentence is the advice Gertrude Cox's tradition would force.
Consultant theatre: beautiful methods, no decision. Shadow: "advice" that launders a pre-chosen policy through a model.
- Go and see how data are actually collected.
- Human behaviour contaminates measurements in ways no model foresees.
- Statistics is only useful if it persuades the people who must act.
26. The March of the Martingales
Congestive heart failure is a chronic, shifting disease: treatments that help some patients tip others into decline, and a patient's record is a time course of repeated hospitalisations, recoveries and changing quality of life, not a single number. Analysing such data needed new mathematics.
Paul Lévy (born 1886), a prodigy who by 33 was a professor at the École Polytechnique, turned to probability in 1919. Working on the central limit theorem, he studied sequences in which the best prediction of the next value is the current value, with bounded variability: the martingale. The word comes from a gambling system of doubling after losses, a harness that keeps a horse's head steady, and a spar on a ship's boom (Salsburg links it to the supposedly stingy people of the town of Martigues).
In the 1970s Odd Aalen in Oslo realised that the stream of events in a trial can be treated as a martingale. A hospitalised patient leaves the risk set and re-enters on discharge; each hospitalisation is a new event; at each moment one needs only who is currently at risk. With Per Kragh Andersen, Ørnulf Borgan and Richard Gill (Salsburg names an "Erik Anderson"), this became the counting-process theory of survival analysis, which also put Cox's 1972 proportional hazards model on rigorous footing. L. J. Wei used the fact that differences of martingales are martingales to avoid estimating many parameters.
A process M_t is a martingale if its expected future value, given the past, is its present value - a fair game relative to a growing information set (the filtration). That sounds abstract; the applied sting is optional stopping: if you can choose when to stop based on the data, you can bias the ending even when each step was fair.
| Habit | Fair-game story | Actual risk |
|---|---|---|
| Fixed n, then analyse | OK under plan | Low peeking risk |
| Peek daily, stop when p<0.05 | Feels like fair checks | Type I error inflates |
| Alpha-spending / sequential | Budgeted peeks | Controlled if followed |
| Bayesian sequential with loss | Decision-theoretic | Needs prior + loss honesty |
An on-call analyst refreshes the A/B dashboard hourly and will announce a win at p<0.05. Even if the variants are identical, the probability of eventually crossing 0.05 rises far above 5% as hours pass. The fix is not scolding curiosity; it is a sequential design or a hold on peeking.
Salsburg celebrates the internationalism: a French idea, refined across many countries, brought to medicine by a Norwegian, a Dane and a Dutchman, extended by Americans including a Taiwanese-born one. Martingale methods dominate long-term chronic disease trials.
Counting-process methods underpin recurrent-event and competing-risks analyses required in heart failure trials, and the estimand framework (ICH E9(R1), 2019) now forces trials to define exactly what effect they estimate when patients die, switch or stop treatment. Martingales also run modern finance (risk-neutral pricing) and anytime-valid inference in online experiments.
Depth pass (source-complete): Contamination to martingales
Salsburg places If, Of, In, Stephen Stigler at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 1907, 1972, 16. Recite them before you turn the page.
The story turns on Had, No, He, Fortunately; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 100, 35. Recite them before you turn the page.
If you forget the rest, retain So, George Box, Fisher, British and what they actually built.
If you forget the rest, retain George Box, He, As Box, You and what they actually built. Numbers and dates to keep: 1986. They are the load-bearing facts of the chapter.
Salsburg places Box, Instead, It, Fisher at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Box, An American, He, United States at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Box, Fisher, University College, University at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1943, 1948, 1956. They are the load-bearing facts of the chapter.
The story turns on George Box, Princeton University, Statistical Techniques Research, Group; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 1978. They are the load-bearing facts of the chapter.
If you forget the rest, retain One, Box, He, By and what they actually built. Numbers and dates to keep: 1968, 1972. They are the load-bearing facts of the chapter. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
Salsburg places World War, Navy, An at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 20. They are the load-bearing facts of the chapter.
The story turns on Monte Carlo, One, Yale University, If; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 36, 80. They are the load-bearing facts of the chapter.
If you forget the rest, retain Study, Downgrade and what they actually built.
If you forget the rest, retain Produce, Future and what they actually built. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
If you forget the rest, retain George Box, University College, David Cox, Cox and what they actually built. Numbers and dates to keep: 1964. They are the load-bearing facts of the chapter.
Salsburg places In, If Japan Can, Why Can, We at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1980, 80, 1939, 1947. They are the load-bearing facts of the chapter.
The story turns on Deming, Ichiro Ishikawa, Japanese Union, Scientists; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Deming, American, He, Unfortunately; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1980, 50, 3. Recite them before you turn the page.
If you forget the rest, retain One, But, On, Deming and what they actually built. Quantitative checkpoints: 3, 1, 6, 10. Recite them before you turn the page. State the alternative before you chase a star on a p-value.
Salsburg places In, Out, Crisis, Deming at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places You, By, How, Deming at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Deming, Instead, Once, Thus at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 100,000. Recite them before you turn the page.
The story turns on Walter Shewhart, Bell Telephone Laboratories, Frank Youden, National Bureau; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on He, Sioux City, Iowa, University; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 1900, 1927, 37, 5. They are the load-bearing facts of the chapter.
If you forget the rest, retain Yale, Deming, Department, Agriculture and what they actually built. Numbers and dates to keep: 1927, 2,000, 30,000, 10, 1993. They are the load-bearing facts of the chapter. State the alternative before you chase a star on a p-value.
Salsburg places In, Neyman, Pearson, Deming at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 11. They are the load-bearing facts of the chapter.
The story turns on Judith Goldberg, American Cyanamid, Paula Norwood, Johnson Pharmaceuticals; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 19, 12, 1975. They are the load-bearing facts of the chapter.
If you forget the rest, retain Stella Cunliffe, Royal Statistical Society, For, It and what they actually built.
If you forget the rest, retain For, Europe, She, Rotterdam and what they actually built. State the alternative before you chase a star on a p-value.
If you forget the rest, retain Guinness, Cunliffe, On, Dublin Brewery and what they actually built. Numbers and dates to keep: 1929. They are the load-bearing facts of the chapter.
Salsburg places Thus, Guinness Brewing Company, Cunliffe, Guinness at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1946. They are the load-bearing facts of the chapter.
Numbers and dates to keep: 7. They are the load-bearing facts of the chapter.
If you forget the rest, retain In, Stella Cunliffe, Homo, It and what they actually built.
M-F-S. Martingale: fair-game process; Filtration: information over time; Stopping rules need care. Recite: martingale, filtration, stopping.
Start with $100. Flip a fair coin; win or lose $1 each time. Plot wealth. Optional stop when ahead by $5. Notice optional stopping selects paths. That selection is why peeking at A/B tests lies.
Calling every trending chart a martingale, or using martingale language to claim markets are fair. Shadow: optional stopping in experiments while citing "fair game" math.
- Chronic disease data are event histories, not single outcomes.
- Risk sets let patients leave and re-enter analysis cleanly.
- Abstract mathematics can sit unused for decades, then transform practice.
27. The Intent to Treat
In the early 1980s Richard Peto faced cancer trials where patients had not stayed on their randomised treatments: doctors, rightly, switched failing patients. The old habit was to analyse only those who stayed on assignment. Peto saw the trap: if non-responders on placebo are switched away and dropped, the remaining placebo patients are those doing well for other reasons, and placebo can look as good as the drug. Edmund Gehan had concluded such studies were not experiments at all.
Analyse every patient as randomised, whatever they actually received: one week on A, or never a single pill, still counts as A. Randomisation is what justifies the p-value, so preserve it. ITT answers a policy question: is it good to start patients on this treatment, leaving doctors free to adapt? Peto stressed it can show a difference but cannot show equivalence: if a worthless drug's patients are all switched to the standard, ITT will find the two "the same."
That limit was forgotten. ITT became dogma, even for trials meant to show a new treatment is as good as the standard with fewer side effects, exactly where it misleads. Salsburg blames a rigid, cookbook Neyman-Pearson culture with fixed cut-offs, which Fisher had opposed: he thought the level of significance should depend on circumstances, though he never said how.
Four answers
| Thinker | Idea |
|---|---|
| David Cox (1977) | separate Fisher's significance testing from Neyman-Pearson hypothesis testing; in practice scientists use tests to drop unneeded parameters or choose between two models |
| George Box | each experiment is one in an endless stream; reanalyse old data in light of new; his EVOP nudges a paper mill's settings continually, with no final truth |
| Deming and others | abandon tests; estimate parameters with confidence intervals (Neyman himself rarely used tests) |
| William Cochran | for Baltimore's public housing, where the city refused to randomise and first-come applicants were the most energetic, run an observational study and adjust for age, family, religion and so on |
| Donald Rubin | each patient has a potential outcome under each treatment; only one is seen; state the conditions under which the unseen one can be estimated |
Cochran's and Rubin's methods need computers and are model-bound: wrong models give wrong answers, so robustness checks matter (Cochran was working on this when he died in 1980). At the other end of the spectrum lie computer-intensive methods that assume almost no structure at all.
The ITT debate was largely resolved by the ICH E9(R1) estimand addendum (2019), now standard for regulators: a trial must declare which effect it estimates and how "intercurrent events" (switching, stopping, death) are handled: treatment-policy (ITT-like), hypothetical, composite, while-on-treatment or principal stratum. Non-inferiority trials must use analyses that do not bias toward equivalence, as Peto warned. Rubin's potential outcomes are the shared language of modern causal inference in medicine, economics and tech.
Randomise 100 patients to drug and 100 to placebo; suppose the drug truly helps 50 and placebo "helps" 30 (natural recovery). Doctors switch the 70 failing placebo patients to other treatment. A "completers only" analysis keeps the 30 placebo responders: placebo 30 of 30 = 100%, drug 50 of 100 = 50% if its failures stay on assignment. The flawed method can make placebo look better than a real drug. ITT compares 50% with 30%, as randomised.
| Analysis | Who is counted | Question answered | Main risk |
|---|---|---|---|
| Intention to treat | everyone, as randomised | does starting this treatment help? | dilutes effects; biased toward equivalence |
| Per protocol | only those who followed assignment | does the treatment work when taken? | breaks randomisation; selection bias |
| As treated | by treatment actually received | same, regrouped | confounded by why patients switched |
Salsburg's deeper point is that ITT is a workaround for a broken experiment, not a law of nature. Fisher's theory assumed the treatment assignment is the treatment received; once doctors (rightly) intervene, no single analysis recovers the original experiment, and the analyst must say which question is being asked.
Because ITT drifts toward "no difference," it is anti-conservative when the goal is to show a new drug is not worse. The CONSORT extension for non-inferiority trials (2006, updated 2012) and the FDA's 2016 non-inferiority guidance ask for both ITT and per-protocol results, with conclusions trusted only when they agree. Under E9(R1), the same question is framed as choosing a hypothetical or while-on-treatment estimand rather than a pure treatment-policy one.
I-T-T. Intent-To-Treat: analyse as randomised; noncompliance is part of the policy effect. Recite: intent, to, treat.
100 assigned drug, 100 placebo; 30 in drug arm never take it. Compare "as treated" (compliers vs placebo) with ITT (all assigned drug vs all placebo). Note how as-treated can confound. ITT answers a policy question; per-protocol answers another - name which you need.
ITT used to bury harm when noncompliance is informative, or per-protocol used to paint a wonder drug by analysing only survivors who liked it. Shadow: switching estimands after seeing results.
- ITT preserves randomisation and answers a policy question.
- ITT cannot prove equivalence.
- When you cannot randomise, adjust carefully and test how sensitive results are.
28. The Computer Turns upon Itself
Guido Castelnuovo, from an ancient Roman Jewish family, fought in 1915 to teach probability and actuarial science at Rome, wrote an early probability textbook (1919) and founded a statistics school (1927). Dismissed under the racial laws at seventy (1938), he ran secret courses for expelled Jewish students in private homes. The Italian school, with Corrado Gini, started from real problems.
In 1916 Francesco Cantelli proved what is now the Glivenko-Cantelli lemma (Glivenko's 1933 version gives the modern notation): the empirical distribution function, an ugly staircase built straight from the data, converges to the true distribution as observations grow. It was a workhorse of proofs, but using it on real data required arithmetic no machine could do until the 1980s.
The bootstrap
Bradley Efron's bootstrap (Salsburg dates it 1982, the year of Efron's monograph; the original paper appeared in 1979) applies Glivenko-Cantelli twice: treat the data as the population, resample with replacement thousands of times, and read the uncertainty of any statistic from the spread of results. The data lift themselves by their own bootstraps. Efron showed it agrees with standard methods when their assumptions hold, and that many of Fisher's methods are themselves forms of resampling.
In the 1960s Joan Rosenblatt and Emmanuel Parzen independently invented kernel density estimation, governed by a kernel and a bandwidth; John van Ryzin (1967) found their optimal settings. Engineers reinvented the same algorithm as fuzzy approximation, with a non-optimal kernel and an arbitrary bandwidth, and built it into self-focusing cameras and climate control. Bart Kosko's fuzzy-systems bibliographies cite Leibniz and Wiener but none of the statisticians. Engineers did not seek the best method; they built what worked.
By 2000 statistical thinking was so pervasive that statisticians were no longer its main developers. Theorems were rediscovered, assumed because they seemed obvious, and occasionally used though proved false. Scientists now think in distributions without noticing the philosophy, just as the philosophers began to find cracks.
Salsburg's "information science" became machine learning, and his story of parallel tribes is now the classic "two cultures" divide (Breiman, 2001) between data modelling and algorithmic modelling. The bootstrap is everywhere, including bagging and random forests. Kernel methods led to support vector machines and Gaussian processes; the attention mechanism in transformers can be read as a learned kernel smoother, so Rosenblatt and Parzen sit, unacknowledged, inside the present large language models. The tribes are converging again: statistical learning theory, conformal prediction and causal ML are joint ground.
C-R-C. Computation changes theory; Resampling (bootstrap, permutation) thrives; Careful error still required. Recite: compute, resample, careful.
Bootstrap a mean: resample your dataset with replacement 1,000 times; plot the mean each time; read a percentile interval. You just let the computer turn upon the sample. Then break it: bootstrap a maximum or a badly dependent time series and watch nonsense appear.
"Computer says p=..." as authority. Shadow: bootstrap misuse on dependent data, data leakage in ML cross-validation, or simulation from a wrong model treated as truth.
- The empirical distribution lets data stand in for the unknown truth.
- Resampling replaced much clever algebra once computing was cheap.
- Fields reinvent each other's tools; look across the fence.
29. The Idol with Feet of Clay
Salsburg closes with Thomas Kuhn's Structure of Scientific Revolutions (1962): models fit for a while, accumulate patches, then fall to a new paradigm. Determinism fell when errors grew with precision; Quetelet had anticipated probabilistic social laws, naively. Now statistics is everywhere: genes and drug effects are parameters; electrons are distributions; the public reads "doubles the risk" and "plus or minus 3 percent" as if their meaning were obvious. Statisticians have lost control: molecular biology, information science and risk analysis grew their own methods. Kuhn implies this revolution too will pass. Salsburg, who once asked "math for poets" students which of their assumptions might look silly in 500 years (they could think of none), poses three problems.
- Can statistical models make decisions? L. J. Cohen's lottery paradox (from Henry Kyburg): reject "ticket n wins" for each of 10,000 tickets at p = 0.0001, and logic says no ticket wins. His gate-crasher paradox: 499 tickets sold, 1,000 people present; each is more likely than not a gate-crasher, so the civil standard lets the promoter collect from all. Probability and logic do not mix.
- What does probability mean in real life? It is clear in two cases: Fisher's randomised experiment (the space of all possible random assignments, giving exact permutation tests, now easy by computer) and random-sample surveys (all possible samples). It is unclear for observational studies in epidemiology, astronomy, weather and law, where two expert witnesses can model the same data differently. Fisher would have objected to significance tests in discrimination cases, which the US Supreme Court accepts.
- Do people understand probability? Savage and de Finetti's coherent personal probability, or Keynes's cultural ordering, needs coherent people. Kahneman and Tversky found none: people distinguish only 50:50 and "almost certain." Patrick Suppes proposed a coherent model with just five values (surely true, more probable than not, as probable as not, less probable than not, surely false), which makes much of statistics distinctions "below the level of human perception."
Is probability even necessary? Improper distributions already appear in quantum mechanics, Bayesian analysis and queuing theory. Kolmogorov's last idea, probability built out of information in finite sequences, might ground a new theory suited to the digital computer.
Salsburg ends hopefully: the next Fisher, Le Cam, Box, Cox, Gosset, Neyman or Pitman may be working unseen somewhere today. The statistical revolution "stands triumphant on feet of clay."
- Decisions: decision theory and loss functions, and frameworks such as GRADE for rating evidence quality, are now how medicine combines flawed studies, which is roughly the "quality of study" weighting Salsburg mentions.
- Meaning: randomisation inference and design-based thinking have spread from trials into economics and tech experiments; for observational data, causal diagrams make assumptions explicit, which addresses the "two experts" problem without solving it.
- Human understanding: research on risk communication (Gigerenzer's natural frequencies, "1 in 100" rather than "1 percent") shows people reason far better with counts. Weather forecasters' probability-of-rain numbers are among the best-calibrated forecasts anywhere, a partial rebuttal to Salsburg's pessimism.
- The next revolution? Some argue it has begun: machine learning predicts superbly without interpretable parameters, a shift from Pearson's "things of science are parameters" toward "the things of science are predictions," raising new versions of every problem in this chapter.
The p-value wars, after Salsburg
Salsburg's worry that a rigid cut-off culture had replaced judgment became official in 2016, when the American Statistical Association issued its first-ever statement on a statistical method (Wasserstein and Lazar, The American Statistician).
| # | Principle |
|---|---|
| 1 | P-values can indicate how incompatible the data are with a specified model |
| 2 | They do not measure the probability that the hypothesis is true, or that the data arose by chance alone |
| 3 | Conclusions and decisions should not rest only on whether p passes a threshold |
| 4 | Proper inference requires full reporting and transparency |
| 5 | A p-value does not measure the size or importance of an effect |
| 6 | By itself, a p-value is not a good measure of evidence |
The debate then split. In 2018, 72 authors led by Daniel Benjamin proposed redefining significance as p < 0.005 for new discoveries. In 2019 a special issue of The American Statistician ("Moving to a World Beyond p < 0.05," 43 papers) urged dropping the phrase "statistically significant," and a Nature comment by Amrhein, Greenland and McShane, signed by more than 800 researchers, called to "retire statistical significance." In 2021 an ASA presidential task force replied that p-values and significance tests, properly applied, still add rigour, and that the 2019 editorial was not ASA policy.
Journals increasingly ask for effect sizes with confidence intervals, preregistration and registered reports; drug regulators still use pre-specified significance levels for confirmatory trials. The threshold survived, but as one input rather than a verdict, close to what Deming and Fisher, from opposite directions, had argued.
Depth pass (source-complete): ITT to idol
Salsburg places In, Richard Peto, Great Britain, He at the hinge of this stretch: without their stubborn work the method stays a slogan. State the alternative before you chase a star on a p-value.
Salsburg places Peto, Blind, Contrary, Fisher at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Edmund Gehan, Anderson Hospital, Texas, Peto at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places One, As Richard Peto, It, Peto at the hinge of this stretch: without their stubborn work the method stays a slogan. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
Salsburg places Unfortunately, Peto, In, As Peto at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Neyman, Pearson, Nothing, Fisher at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 38, 1977, 23. They are the load-bearing facts of the chapter. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
The story turns on George Box, Box, Cox, Scientific; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Deming, Fisher, It, Neyman; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on In, William Cochran, Johns Hopkins University, Baltimore; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Cochran, Almost, William Cochran, In; keep them as carriers of the statistical move, not as celebrity wallpaper. State the alternative before you chase a star on a p-value.
The story turns on Rubin, Cochran, Fisher, If; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 1980. They are the load-bearing facts of the chapter.
If you forget the rest, retain Methods, Cochran, Rubin, At and what they actually built. Numbers and dates to keep: 1915. They are the load-bearing facts of the chapter.
Salsburg places In, Castelnuovo, As, Calcolo at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 1919, 1927, 1922, 39, 1935. They are the load-bearing facts of the chapter.
The story turns on Nazi, Jewish, Castelnuovo, In; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 1952. They are the load-bearing facts of the chapter. Models are tools. Idols crack. The afterword's clay feet are the book's warning label.
If you forget the rest, retain One, Francesco Paolo Cantelli, Kolmogorov, Cantelli and what they actually built. Numbers and dates to keep: 1875, 1966, 1916, 40, 1933. They are the load-bearing facts of the chapter.
Salsburg places Glivenko, Cantelli, It, To at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: twenty years, 1982. They are the load-bearing facts of the chapter.
The story turns on Efron, It, His, In; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 1982. They are the load-bearing facts of the chapter. State the alternative before you chase a star on a p-value.
If you forget the rest, retain One, Joan Rosenblatt, National Bureau, Standards and what they actually built. Numbers and dates to keep: 1967. They are the load-bearing facts of the chapter.
Salsburg places Bart Kosko, As, Gottfried Wilhelm, Leibniz at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: seventy years, 41. They are the load-bearing facts of the chapter.
The story turns on Sometimes, In, Karl Pearson, Newton; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on In, Thomas Kuhn, University, Chicago; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 1962. Recite them before you turn the page.
If you forget the rest, retain An, Sun, Mathematicians, Pierre Simon Laplace and what they actually built.
If you forget the rest, retain Eventually, Instead, Laplace, At and what they actually built. Numbers and dates to keep: 1, 2. They are the load-bearing facts of the chapter.
Salsburg places Similarly, Sociologists, In, So at the hinge of this stretch: without their stubborn work the method stays a slogan.
Salsburg places Scientific, In, Karl Pearson at the hinge of this stretch: without their stubborn work the method stays a slogan. Quantitative checkpoints: 65, 3, 95. Recite them before you turn the page.
The story turns on We, On; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Any, Eventually, We, It; keep them as carriers of the statistical move, not as celebrity wallpaper. State the alternative before you chase a star on a p-value.
The story turns on Connecticut College, It, Early, Ars Magna; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on As; keep them as carriers of the statistical move, not as celebrity wallpaper. Numbers and dates to keep: 500. They are the load-bearing facts of the chapter. State the alternative before you chase a star on a p-value.
If you forget the rest, retain Jonathan Cohen, Oxford University, Pascalian, In and what they actually built. Numbers and dates to keep: 1989. They are the load-bearing facts of the chapter.
Salsburg places Suppose, We, To, Let at the hinge of this stretch: without their stubborn work the method stays a slogan. Numbers and dates to keep: 0.0001, 10,000, 1, 2, 50. They are the load-bearing facts of the chapter.
The story turns on Logic, Fisher, Cohen, Jerry Cornfield; keep them as carriers of the statistical move, not as celebrity wallpaper.
The story turns on Andrei Kolmogorov, Probability, All, Is; keep them as carriers of the statistical move, not as celebrity wallpaper. Quantitative checkpoints: 95. Recite them before you turn the page.
If you forget the rest, retain Karl Pearson, Kolmogorov, We, William and what they actually built.
If you forget the rest, retain Fisher, Gosset, In, If and what they actually built. State the alternative before you chase a star on a p-value.
If you forget the rest, retain Fisher, To Fisher, Discrimination, American and what they actually built. Quantitative checkpoints: 1988. Recite them before you turn the page.
The 2016 ASA Statement on Statistical Significance and P-Values set six principles: a p-value is not the probability the hypothesis is true; it is not alone proof; scientific conclusions need more than a threshold; and so on. It did not ban p-values.
2019 brought a sharper fight. A Nature comment (Amrhein, Greenland, McShane and co-signers) urged retiring "statistically significant" as a bright line. The American Statistician published a special issue, "Statistical Inference in the 21st Century: A World Beyond p<0.05." Parallel proposals pushed p<0.005 for claims of new discoveries (Benjamin et al.), trading false positives for harder power requirements.
2021 ASA task-force language emphasised that threshold theatre was never the point; estimands, design and uncertainty were. Through the mid-2020s the applied fields split: some journals softened star rituals; many A/B platforms and pharma workflows still dichotomise because decisions need rules.
March 2026: Robert Matthews in Significance ("The ASA's p-value statement, 10 years on") revisits what changed and what did not: awareness up; ritual stubborn; education and incentives still misaligned. January 2026 FDA draft guidance Use of Bayesian Methodology in Clinical Trials of Drug and Biological Products continues a long institutional path: priors and decision probabilities in the open, not only asymptotic p-values. The idol's feet are clay; the statue is still in the lobby.
| Camp | Move | Risk if absolutist |
|---|---|---|
| ASA 2016 principles | Contextualise p | Ignored as boilerplate |
| Retire significance language | Force estimation talk | Replaced by other bright lines |
| p<0.005 for new claims | Fewer false discoveries | Underpowered science, file drawer |
| Bayesian / decision | Posterior + loss | Hidden priors, theatre |
| FDR / selection control | Many tests honesty | Misread as per-test alpha |
I-P-R. Idol of p<0.05; Principles (ASA) over thresholds; Reform via estimation, design, Bayes, FDR. Recite: idol, principles, reform.
Take a published abstract that says "significant." Rewrite it with an effect size, an interval and a plain-language limitation, without the word significant. If the rewrite feels empty, the original was the idol speaking.
Replacing 0.05 with 0.005 and changing nothing else, or banning p-values while smuggling the same dichotomania through "Bayes factors > 3." Shadow: reform theatre.
Probability stays crisp when the sample space is made by the design (randomised trials, random samples). Elsewhere the word borrows meaning from frequency, logic or personal belief, and those stories disagree. Salsburg's clay feet are not a call to abandon statistics; they are a call to name the story you are telling, prefer design when you can, and refuse to worship a threshold.
- Probability is clear for randomised experiments and random samples, contested elsewhere.
- Logic and probability can conflict (lottery paradox); do not treat a tiny p as a proof.
- Keep effect sizes, alternatives and design honesty beside any star on a printout.
- The revolution stands; the idol needs watching. That is the book's last useful sentence.
- ASA 2016: six principles, not a ban.
- 2019: retire bright-line "significance" language where you can.
- p<0.005: optional discovery bar; pay the power cost.
- 2026: Matthews's ten-year review and FDA Bayesian draft guidance both push context over theatre.
Afterword
Salsburg explains his omissions: modern science is too big for one person (a 1970s search on computer diagnosis found three traditions that never cited each other); he chose examples he could explain without symbols and that kept a connected path; leaving someone out does not mean their work is unimportant. He hopes readers will go further, and thanks his editors at W. H. Freeman.
Salsburg owns his omissions: science is too big for one narrator; he chose stories he could tell without symbols on a connected path. Leaving someone out is not a verdict on their importance. The afterword is also a hand-off: go further, and do not mistake a popular history for a complete map.
S-M-F. Statistical models replaced clockwork; Modesty about parameters and heroes; Future work joins design, computation and ethics. Recite: statistical, modest, future.
That is your personal afterword.
Closing the book as pure triumph narrative. Shadow: ignoring eugenics entanglements, replication failures and software-amplified dichotomania that followed the heroes.
- Pick one primary source the book names (Fisher 1935, Student 1908, ASA 2016) and read five pages.
- Rewrite one work email or model card using an estimand + interval, not a star.
- Red-team one dashboard with a Shadow question: what would make this metric lie?
- The book is a path, not an encyclopedia; omissions are deliberate.
- Statistical models replaced clockwork as a working worldview, not a finished metaphysics.
- Your job after reading is practice: design, estimation, honest uncertainty.
Synthesis
This synthesis tracks how statistical models replaced clockwork thinking across the book.
Key takeaways
- Science models distributions, not things. Pearson's shift: what we study are parameters of probability distributions, estimated and never seen.
- Measurement error is not noise to discard. It is the main object of study.
- Design beats analysis. Fisher: a badly designed experiment cannot be rescued by later arithmetic. Plan the analysis before collecting data.
- Randomisation is the engine of causal inference. It balances unknown confounders and justifies the test.
- Small samples need their own maths. Gosset's t-distribution showed that "large-sample" methods mislead with a handful of observations.
- Vary many factors at once. Factorial designs and ANOVA beat one-factor-at-a-time.
- Likelihood and efficiency matter. Fisher's maximum likelihood gives estimators that use the information in the data well.
- A p-value is a measure of surprise, not truth. Fisher meant it as informal evidence; it is not the probability the hypothesis is true.
- Tests need alternatives. Neyman and Pearson: frame the decision, control error rates, think about power.
- Confidence intervals describe procedures. The 95% belongs to the method across repetitions, not to any single interval.
- Bayes is coherent but needs priors. Its century of exile ended with computers; the question of where priors come from did not.
- Probability got axioms, not meaning. Kolmogorov made it rigorous mathematics; applying it to the world remains a philosophical choice.
- The rare event decides. Extreme value theory: floods, failures and crashes live in the tails.
- Random samples beat big samples. The 1936 Literary Digest versus Gallup lesson still holds for surveys and data sets.
- Observational data can establish causes, with work. Cornfield's inequality and converging evidence settled smoking and cancer despite Fisher.
- Assumption-light methods are strong. Rank tests and the bootstrap trade a little efficiency for a lot of robustness.
- Look at the data first. Tukey's exploratory analysis put plots and resistance ahead of formal tests.
- All models are wrong, some are useful. Box: choose robust, adequate models and keep checking them.
- Variation is the enemy of quality. Shewhart and Deming: manage the process, not the inspection.
- Analyse as randomised. Intention to treat protects trials from bias created by who complies.
- Computation changed which methods exist. Bootstrap, EM, MCMC and resampling became possible only with cheap computing.
- Statistics is a people story. Rivalries, wars, migrations and outsider women shaped the methods.
- The foundations are still unsettled. Decisions, logic and probability do not fully line up, and humans misread risk.
Frameworks table
| Framework | Origin | Use it when | Watch out for |
|---|---|---|---|
| Randomised experiment | Fisher, 1920s | You can assign the treatment | Non-compliance, ethics, external validity |
| Factorial design and ANOVA | Fisher, 1923 to 1935 | Several factors interact | Too many factors without replication |
| Significance test | Fisher; Pearson | Asking "is this surprising?" | p-hacking, treating p < 0.05 as proof |
| Hypothesis test with power | Neyman and Pearson, 1928 to 1933 | A decision with costed errors | Ignoring power and effect size |
| Confidence interval | Neyman, 1934 | Reporting uncertainty of an estimate | Reading it as a probability for one interval |
| Bayesian updating | Bayes, Laplace, Savage | Prior knowledge exists, decisions matter | Hidden or convenient priors |
| Random sampling | Neyman, Hansen, Hurwitz | Estimating a population | Non-response, coverage bias |
| Cornfield causal reasoning | Cornfield et al., 1959 | Only observational data exist | Unmeasured confounding |
| Exploratory data analysis | Tukey, 1977 | Before any modelling | Forking paths if you then test what you found |
| Robust methods | Box, Tukey, Huber | Outliers or heavy tails | Losing efficiency when data are clean |
| Statistical process control | Shewhart, Deming | Running a repeatable process | Treating common-cause variation as special |
| Bootstrap | Efron, 1979 | No formula for the standard error | Dependent data, tiny samples |
| Intention to treat | Peto and others | Analysing trials | Diluted effects with poor adherence |
A closer look
- Factual slips. EM credited to Laird and Ware (it was Dempster, Laird and Rubin, 1977); Blackett called a biologist; Hammond and Horn's follow-up shortened to four months; "Erik Anderson" for Per Kragh Andersen; the bootstrap paper misdated; Nightingale credited with the pie chart; the Soviet collapse misdated.
- Eugenics underplayed. Galton, Pearson and Fisher's eugenics shaped their science and their institutions. UCL renamed buildings in 2020 and Caius removed its Fisher window the same year. Salsburg mentions the views without weighing them.
- Fisher's smoking role. The book is fair on the argument but light on his tobacco industry consultancy.
- Hero history. The anecdotal style favours a few great men; the institutional and computing story is thin, though the women's chapter helps.
- Light on the p-value problem. Salsburg sees the gap between what tests mean and how they are used but did not foresee the replication crisis.
- Thin on modern causal inference. Rubin's potential outcomes get little space and Pearl's graphs none.
| Held up well | Changed or challenged |
|---|---|
| Randomisation as the gold standard | Observational causal inference (DAGs, target trial emulation) is now mainstream |
| Design before data | Pre-registration and registered reports formalised it |
| Bootstrap and computing-led methods | Machine learning now dominates prediction |
| Doubts about foundations | p < 0.05 openly questioned by the ASA (2016) and others |
| Deming on variation | Bayesian methods accepted in FDA draft guidance (January 2026) |
Who's who
| Person | Dates | Contribution |
|---|---|---|
| Francis Galton | 1822 to 1911 | Regression, correlation; eugenics |
| Karl Pearson | 1857 to 1936 | Distributions, chi-square, Biometrika |
| W. S. Gosset ("Student") | 1876 to 1937 | t-distribution at Guinness |
| R. A. Fisher | 1890 to 1962 | Design, ANOVA, likelihood, p-values |
| Jerzy Neyman | 1894 to 1981 | Tests, confidence intervals, sampling |
| Egon Pearson | 1895 to 1980 | Neyman-Pearson theory |
| Andrei Kolmogorov | 1903 to 1987 | Axioms of probability |
| L. H. C. Tippett | 1902 to 1985 | Extreme values, random number tables |
| Chester Bliss | 1899 to 1979 | Probit analysis |
| William Feller, J. W. Lindeberg, Paul Lévy | 20th c. | Central limit theorem |
| Person | Dates | Contribution |
|---|---|---|
| Florence Nightingale David | 1909 to 1993 | Combinatorics, history of probability |
| Gertrude Cox | 1900 to 1978 | Experimental design, Raleigh institute |
| Stella Cunliffe | 1917 to 2012 | First woman RSS president; Guinness, Home Office |
| Frank Wilcoxon | 1892 to 1965 | Rank tests |
| Morris Hansen | 1910 to 1990 | Census sampling |
| Jerome Cornfield | 1912 to 1979 | Smoking and cancer, Bayesian trials |
| Abraham Wald | 1902 to 1950 | Sequential analysis, decision theory |
| L. J. Savage, Bruno de Finetti | 20th c. | Personal probability |
| John Tukey | 1915 to 2000 | EDA, FFT, robustness |
| George Box | 1919 to 2013 | Robustness, response surfaces, time series |
| W. Edwards Deming | 1900 to 1993 | Quality, Japan |
| Bradley Efron | b. 1938 | Bootstrap |
| David Salsburg | b. 1931 | The author; Pfizer statistician |
Glossary
Quick lexicon for Salsburg's path: each term is a tool or fight, not wallpaper. Skim before a re-read; use when a chapter name will not stick.
| Term | Meaning |
|---|---|
| ANOVA | Analysis of variance: splits total variation into parts due to factors and error |
| Bayes' theorem | Rule for updating a prior probability with data into a posterior |
| Bootstrap | Resampling the data with replacement to estimate uncertainty |
| Central limit theorem | Sums of many independent effects tend to a normal distribution |
| Chi-square test | Pearson's test of fit between observed and expected counts |
| Confidence interval | Range from a procedure that covers the true value in a stated share of repetitions |
| Confounder | Variable affecting both treatment and outcome, creating spurious association |
| Control chart | Shewhart's plot that separates common-cause from special-cause variation |
| Distribution | Description of how probable each value of a variable is |
| EM algorithm | Iterative method for maximum likelihood with missing data |
| Exploratory data analysis | Tukey's approach of looking at data before formal modelling |
| Extreme value distribution | Limit laws for the largest or smallest of many observations |
| Factorial design | Experiment varying several factors together in all combinations |
| FFT | Fast Fourier transform, efficient computation of frequency components |
| Hypothesis test | Procedure deciding between hypotheses with controlled error rates |
| Intention to treat | Analysing trial patients by assigned group regardless of compliance |
| Kernel density | Smooth estimate of a distribution from data |
| Term | Meaning |
|---|---|
| Likelihood | Probability of the observed data as a function of the parameters |
| Martingale | Process whose expected next value equals its current value |
| Maximum likelihood | Choosing parameters that make the observed data most probable |
| Nonparametric | Methods not assuming a specific distribution shape |
| Null hypothesis | Default claim of no effect that a test tries to reject |
| p-value | Probability of results at least as extreme as observed if the null hypothesis is true |
| Parameter | Fixed but unknown number describing a distribution |
| Power | Probability a test detects a real effect of given size |
| Prior | Probability distribution expressing belief before data |
| Probit | Model of dose response using the normal distribution |
| Randomisation | Assigning treatments by chance to remove bias |
| Regression | Modelling a variable's mean as a function of others |
| Robustness | Method's resistance to violated assumptions or outliers |
| Sequential analysis | Testing data as they arrive and stopping when evidence suffices |
| Stratified sample | Random sample drawn separately within subgroups |
| t-test | Gosset's test for means from small samples |
| Type I and II errors | Rejecting a true null; failing to reject a false one |
| Variance | Average squared deviation from the mean |
About this book
The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century by David Salsburg, 2001. This is a chapter-by-chapter study of the book, with summaries, maps, tables and diagrams; short quotations keep their page references. It is shared free for personal study, is not for sale, and is not endorsed by the author or publisher. Please read and buy the original.
Insight Editions · rajeshrmahapatra.com/insight-editions