Insight Editions · No. 37
Hands-On Machine Learning with Scikit-Learn and PyTorch
Concepts, Tools, and Techniques to Build Intelligent Systems
Aurélien Géron
Aurélien Géron presents practical machine-learning methods, from data preparation and classical models to neural networks, transformers, generative systems, and reinforcement learning.
A study guide to Hands-On Machine Learning with Scikit-Learn and PyTorch by Aurélien Géron. Shared free for personal reading; not for sale and not endorsed by the authors or publishers. Please read and buy the original.
Overview
Hands-On Machine Learning with Scikit-Learn and PyTorch (Aurélien Géron) is a full-stack ML course in book form: sklearn foundations through deep learning, transformers, generative models, and RL.
At a glance
Preface folded
Géron positions the book for practitioners who learn by doing. The PyTorch edition replaces the older TensorFlow-centric path. Prerequisites: Python and enough maths to read a gradient.
The book on one page
1. Frame the problem and pick a metric (Ch1-2). 2. Classification metrics and thresholds are policy (Ch3). 3. Train linear models with geometry you can draw (Ch4). 4. Trees and ensembles dominate tabular (Ch5-6). 5. Reduce dimensions and cluster with eyes open (Ch7-8). 6. MLPs then PyTorch loops (Ch9-10). 7. Make deep nets trainable (Ch11). 8. CNNs, sequences, NLP, transformers (Ch12-16). 9. Speed large transformers (Ch17 authored). 10. Generative and RL (Ch18-19). 11. Autodiff and precision (App A-B).
Chapter map
| Ch | Title | Gist | |
|---|---|---|---|
| 1 | ML overview | Problem types, challenges | 2 |
| End-to-End Project | Housing pipeline discipline | 3 | Classification |
| PR, ROC, thresholds | 4 | Training Models | GD, regularisation, softmax |
| 5 | Decision Trees | CART, Gini, limits | 6 |
| Ensembles | Bagging, RF, boosting, stacking | 7 | Dimensionality Reduction |
| PCA and nonlinear projections | 8 | Unsupervised | Clustering, GMM, anomaly |
| 9 | Intro ANNs | Perceptron to MLP | 10 |
| PyTorch NNs | Tensors, loop, Optuna | 11 | Training Deep NNs |
| Init, norm, schedules | 12 | CNNs | Conv, detect, segment |
| 13 | Sequences | RNN/LSTM/temporal CNN | 14 |
| NLP + Attention | Embeddings to attention | 15 | Transformers NLP |
| Blocks, chat, fine-tune | 16 | Vision/Multimodal | ViT, CLIP-style |
| 17 | Speeding Transformers | KV, spec, FA, MoE, LoRA (authored) | 18 |
| AE/GAN/Diffusion | Generative stack | 19 | RL |
| MDP to deep RL | A | Autodiff | Forward/reverse AD |
A closer look on the edition
Sklearn-versus-deep advice still holds for tabular banking work. This is a study edition, not a substitute for running the book's notebooks.
1. The Machine Learning Overview
Géron opens by fixing what machine learning is, why it beats long rule lists, and how to map the field before writing serious code. The chapter defines supervised, unsupervised, semi-supervised, self-supervised and reinforcement learning; contrasts batch versus online and instance-based versus model-based systems; walks a tiny life-satisfaction linear model in Scikit-Learn; then catalogues failure modes of bad data and bad models and how to test without fooling yourself.
What machine learning is
Machine learning is the science and art of programming computers so they learn from data. Géron pairs a short definition with two classics. Arthur Samuel (1959): the field that gives computers the ability to learn without being explicitly programmed. Tom Mitchell (1997): a program learns from experience E with respect to task T and performance measure P if its performance on T, as measured by P, improves with experience E.
Mitchell 1997: learning means P on T improves with E.
The spam filter is the worked example that made ML mainstream in the 1990s. Users flag spam and ham; those examples form the training set; each example is a training instance (or sample). The component that learns and predicts is the model. Neural networks and random forests are example model families. Here T is flagging new mail, E is the training data, and P might be accuracy, the share of correctly classified emails. Accuracy is common in classification; later chapters add precision, recall, ROC and more.
Downloading Wikipedia is not machine learning. The computer holds more text but is not suddenly better at any defined task. Learning requires a task, a performance measure and experience that improves that measure. ChatGPT, Gemini, Claude, Perplexity and peers mark a shift: AI is no longer only a quiet backend feature; it is often the product itself. OCR and spam were decades earlier; voice prompts, translation, image search and recommendations filled the gap before generative chatbots.
Why use machine learning
Traditional programming for spam becomes an unmaintainable rule list. You examine spam, notice phrases such as "4U", "credit card", "free" and "amazing", write detectors for sender, subject and body patterns, test, and repeat until launch. Because the problem is hard, the program grows into complex rules that are expensive to maintain (book Figure 1-1).
Why ML replaces brittle spam rule lists.
An ML spam filter learns predictors from data. It detects words and phrases that are unusually frequent in spam versus ham (Figure 1-2). The program is shorter, easier to maintain and usually more accurate. When spammers switch from "4U" to "For U", a rule-based filter needs a new rule; an ML filter notices the new pattern in freshly flagged spam and adapts after retrain (Figure 1-3).
ML shines when no good classical algorithm exists. Distinguishing the spoken words "one" and "two" by hardcoding a high-pitch detector for "T" does not scale to thousands of words, millions of speakers, noise and dozens of languages. The workable approach is an algorithm that learns from many example recordings.
ML can also teach humans. Inspecting a trained spam model can reveal word combinations that predict spam, sometimes exposing unsuspected correlations. Digging into large data for hidden patterns is data mining; ML is often the tool. Figure 1-4 closes the loop from data to model to human insight.
Four situations favour ML over hand-built logic.
Four situations where Géron prefers ML to hand-built logic.
- Maintenance burden. Existing solutions need long rule lists; a model can simplify code and beat the traditional approach.
- No known algorithm. Complex problems where classical methods fail; strong ML techniques may find a workable solution.
- Fluctuating environments. Systems can be retrained on new data and stay current.
- Insight. Models and mining help humans understand complex problems and large datasets.
Examples of applications
Géron walks a deliberately wide catalogue so readers map tasks to technique families before coding.
Vision on the line and in the clinic. Analysing product images for classification is typically CNNs (Chapter 12) or vision transformers (Chapter 16). Detecting tumours in brain scans is semantic image segmentation: every pixel gets a class so shape and location are recovered, again with CNNs or ViTs.
Task families routed to later chapters.
Language tasks. Classifying news articles is text classification; RNNs and CNNs work, transformers work better (Chapter 15). Flagging offensive forum comments is the same stack. Summarising long documents is text summarisation with the same NLP tools. Building a chatbot or personal assistant combines NLU and question-answering modules.
Long sequences and speech. Estimating genetic disease risk from very long DNA is a spread-out pattern problem where state-space models (SSMs) shine (online note at homl.info). App voice commands are speech recognition over long audio, typically RNNs, CNNs or transformers (Chapters 13-15).
Tabular forecasting and risk. Next-year revenue from many metrics is regression: linear or polynomial models (Chapter 4), regression SVMs (online SVM appendix), regression random forests (Chapter 6) or neural nets (Chapter 9). Past metric sequences push toward RNNs, CNNs or transformers. Credit-card fraud is anomaly detection via isolation forests, Gaussian mixtures (Chapter 8) or autoencoders (Chapter 18).
Structure discovery and recommenders. Segmenting clients by purchases for distinct marketing strategies is clustering (k-means, DBSCAN and more, Chapter 8). Representing a high-dimensional dataset in a readable diagram is visualisation via dimensionality reduction (Chapter 7). Recommending a product from past purchases can feed client history into a neural net trained on sequences across clients (Chapter 9).
Games and control. Intelligent game bots often use reinforcement learning (Chapter 19): an agent picks actions to maximise rewards over time in an environment. AlphaGo beating the Go world champion is the celebrity case.
Reading the catalogue as a routing table. When a stakeholder asks for "some AI on our data", map the request onto this list before picking a library. Image in, class out → Chapter 12 or 16. Text in, class or summary out → Chapter 14-15. Numbers in, number out → Chapter 4 or 6. Numbers in, segment out → Chapter 8. Clickstream in, next item out → recommender pattern in Chapter 9. Agent in a simulated world → Chapter 19. Mis-routing (for example treating fraud as ordinary supervised accuracy when positives are 0.1%) is how projects burn quarters.
Types of machine learning systems
Three orthogonal axes organise the zoo. (1) How training is guided: supervised, unsupervised, semi-supervised, self-supervised, reinforcement. (2) Whether the system learns incrementally: online versus batch. (3) How it generalises: instance-based versus model-based. Axes combine. A state-of-the-art spam filter may be an online, model-based, supervised deep net.
Supervised learning
The training set includes desired solutions called labels (Figure 1-5). Classification assigns classes; the spam filter is the running example. Regression predicts a numeric target such as car price from mileage, age and brand (Figure 1-6). Some regression models serve classification: logistic regression outputs a probability of class membership (for example, 20% chance of spam).
The two headline supervised tasks.
Vocabulary note. Target and label are often synonyms; target is more common in regression, label in classification. Features are also called predictors or attributes, and the word may refer to one sample's value or to the column across samples.
Unsupervised learning
Training data has no labels; the system learns without a teacher.
Unsupervised task ladder in Chapter 1.
Clustering groups similar blog visitors by age, region, interests and session length without being told the groups. Hierarchical clustering can split groups into subgroups for targeted posts (Figure 1-7).
Visualisation algorithms project complex unlabeled data to 2D or 3D while preserving structure so clusters stay separated when possible (Figure 1-8, t-SNE style semantic clusters).
Dimensionality reduction simplifies data without losing too much information, often by merging correlated features (mileage and age into "wear"). That merge is feature extraction. Tip in the book: reduce dimensions before a supervised algorithm to gain speed, storage and sometimes accuracy.
Anomaly detection trains mostly on normal instances then flags odd credit-card transactions, manufacturing defects or outliers before another learner runs (Figure 1-9). Features may include distance from home, time, amount, merchant category and frequency.
Novelty detection is related but needs a very clean training set: it flags instances that look different from everything seen. If 1% of dog photos are Chihuahuas, novelty detection should not treat new Chihuahuas as novel; anomaly detection might still call them anomalies because they are rare.
Association rule learning finds relations among attributes in sales logs (barbecue sauce and chips with steak) and suggests placement.
Semi-supervised learning
Labelling is costly, so many datasets are mostly unlabeled. Semi-supervised algorithms handle partial labels (Figure 1-10). Google Photos-style face grouping clusters person A across photos 1, 5, 11 and person B across 2, 5, 7; one human label per person then names everyone. Typical pattern: cluster, assign the majority label inside each cluster to unlabeled points, then train any supervised model on the completed set.
Self-supervised learning
Generate a fully labelled set from unlabeled data, then use supervised methods. Example: randomly mask part of each image and train a model to recover the original (Figure 1-11). Inputs are masked images; labels are originals. The repair model can be useful alone (restore damage, erase objects) but more often is a stepping stone.
Self-supervised pretrain then transfer.
Pet classifier path. Train image repair on unlabeled pet photos; a good repairer already "knows" cats from dogs. Tweak the architecture to predict species, then fine-tune on a smaller labelled set so the model maps known species to your label schema. Transferring skill across tasks is transfer learning, central in Part II. LLMs train similarly by masking words in huge corpora then fine-tuning for sentiment, chat and more.
Category debate. Some call self-supervision a branch of unsupervised learning because the source data is unlabeled; Géron separates it because training uses generated labels and the tasks look like classification and regression. Best practice: treat it as its own category.
Reinforcement learning
An agent observes an environment, acts, and receives rewards or penalties (Figure 1-12). It must learn a policy: which action to take in each situation to maximise reward over time. Robots learning to walk and DeepMind's AlphaGo (beat Ke Jie in May 2017 after millions of games and self-play) are the book examples. Learning was off during the championship games; AlphaGo applied a fixed learned policy. That freeze is a form of offline use of a policy trained earlier.
Reinforcement loop: agent, environment, reward, policy.
- Supervised: labels given; classification or regression.
- Unsupervised: no labels; cluster, reduce dimensions, detect anomalies, mine association rules.
- Semi-supervised: few labels plus many unlabeled; cluster then propagate labels.
- Self-supervised: labels generated from the data (masking, next-token prediction).
- Reinforcement: agent, environment, reward signal, learned policy.
Batch versus online learning
The question is whether the system can learn incrementally from a stream. Random forests typically need a full retrain (batch). Gradient-descent models can take one mini-batch at a time (online).
Batch learning
Train on all available data, usually offline, then deploy without further learning (offline learning). Performance decays as the world moves: data drift or model rot. Cat-versus-dog classifiers drift slowly; financial predictors can rot quickly. Even pet classifiers need retrain when cameras, formats, sharpness and popular breeds change, or when people put tiny hats on pets.
Batch versus online learning.
Updating a batch system means training a new version from scratch on old plus new data, evaluating, and swapping. Automation (Figure 1-3's spirit) can run daily or weekly. Full retrain is expensive in CPU, memory, disk and network. Huge data plus a need for freshness can make pure batch learning impossible. Constrained devices (phones, Mars rovers) cannot haul full datasets and train for hours each day.
Online learning
Train incrementally on individuals or mini-batches. Each step is fast and cheap, so the system can adapt as data arrives (Figure 1-13). Gradient descent dominates; a few other incremental algorithms exist. Use cases: rapid market pattern shifts; training on-device with limited resources; out-of-core learning when data does not fit in memory (load a chunk, train a step, repeat; Figure 1-14).
Learning rate is the central knob. High rate adapts fast but forgets old patterns (catastrophic forgetting / catastrophic interference): a spam filter that only catches today's spam is a failure. Low rate means more inertia: slower learning, less sensitivity to noise and outlier bursts.
Learning-rate trade-off and catastrophic forgetting.
Naming trap. Out-of-core "online" training is often run offline, not on the live path. Think incremental learning. Deep-learning "batch" usually means mini-batch, not full-dataset batch learning.
Operational risk. Bad stream data can degrade a live model quickly: buggy sensors, or adversaries stuffing a search engine to climb rankings. Monitor performance and inputs; be ready to turn learning off and revert. Anomaly detection on inputs (Chapter 8) is part of that defence.
Instance-based versus model-based learning
Generalisation is the goal: perform well on instances never seen in training. Strong training metrics are not enough.
Instance-based learning
Learn by heart, then compare. A naive spam filter flags only exact duplicates of known spam. A better version flags mail similar to known spam, using a similarity measure such as shared word count. Figure 1-15 classifies a new point by the majority class among its nearest neighbours.
How the two generalisation styles differ.
Trade-offs. Instance-based methods can work on small, shifting datasets. They scale poorly: production needs a copy of the training set; prediction searches neighbours; high-dimensional inputs such as images are hostile.
Model-based learning and the first workflow
Build a model of the examples, then predict with the model (Figure 1-16). Géron's demo asks whether money buys happiness. He joins OECD Better Life Index life-satisfaction scores with World Bank GDP per capita and sorts (Table 1-1 excerpt):
| Country | GDP per capita (USD) | Life satisfaction | |
|---|---|---|---|
| Turkey | 28,384 | 5.5 | Hungary |
| 31,008 | 5.6 | France | 42,026 |
| 6.5 | United States | 60,236 | 6.9 |
| New Zealand | 42,404 | 7.3 | Australia |
| 48,698 | 7.3 | Denmark | 55,938 |
The scatter looks roughly linear though noisy (Figure 1-17). Model selection picks a linear function of one attribute (Equation 1-1):
life_satisfaction = θ₀ + θ₁ × GDP_per_capita
θ₀ and θ₁ are model parameters. Tweaking them yields any line (Figure 1-18). To choose values, define a utility/fitness function (higher better) or a cost/loss function (lower better). Linear regression typically minimises distance between predictions and training examples. Training finds roughly θ₀ = 3.75 and θ₁ = 6.78×10⁻⁵ (Figure 1-19).
Inference example. Puerto Rico GDP per capita $33,442 gives 3.75 + 33,442 × 6.78×10⁻⁵ ≈ 6.02.
Example 1-1 (Scikit-Learn pattern). Import matplotlib, numpy, pandas and `LinearRegression`. Load `lifesat.csv` from `https://github.com/ageron/data/raw/main/`. Build `X` from GDP and `y` from life satisfaction. Scatterplot with axis limits. `model.fit(X, y)`, then `model.predict([[33_442.8]])` prints about [[6.01610329]].
Instance-based alternative in two lines. Replace `LinearRegression` with `KNeighborsRegressor(n_neighbors=3)`. Poland ($32,238, satisfaction 6.1) is closest; with Portugal 5.4 and Estonia 5.7 the average is 5.73, a bit below the linear prediction.
If predictions disappoint: add attributes (employment, health, pollution), gather more or cleaner data, or pick a richer model (polynomial regression).
Workflow summary. Study the data → select a model → train (search parameters that minimise cost) → inference on new cases, hoping for generalisation. Chapter 2 repeats this end to end on housing data.
Beyond the main taxonomy. Ensemble learning combines models (Chapter 6). Federated learning trains across devices without pooling raw data. Meta-learning learns how to learn new tasks from few examples. Figure 1-20 in the book is the overview diagram of categories covered so far.
Worked Scikit-Learn pattern in Example 1-1
The listing is the book's first full training loop. It is short on purpose: download, visualise, select, fit, predict. `data_root` points at `https://github.com/ageron/data/raw/main/`; `lifesat.csv` is read into a DataFrame; `X` is the GDP column as a NumPy array shaped for sklearn (2D); `y` is life satisfaction. The scatter uses `kind='scatter'` with grid and fixed axis limits so the eye sees the band of interest. `LinearRegression()` is instantiated with defaults, `fit` runs ordinary least squares, and `predict` on `[[33_442.8]]` returns the Puerto Rico estimate.
Example 1-1 workflow in Scikit-Learn.
Why the double brackets. Scikit-Learn predictors expect a 2D feature matrix even for one sample and one feature. Passing a flat list is a common first-hour bug. The printed output `[[6.01610329]]` keeps that 2D shape.
Swapping learners is meant to feel cheap. Two import and constructor lines turn the same `X, y` into k-NN regression. That is the pedagogical point of model-based versus instance-based learning in code form: the workflow shell stays; the generalisation mechanism changes. When predictions disappoint, the book does not tell you to tune endlessly first; it tells you to add attributes, improve data quality or pick a richer hypothesis class.
Theta convention. Greek θ is the book's ongoing symbol for model parameters. Remember the three senses of "model" when you read later chapters that say "the model": family, architecture or trained weights.
How the three axes combine in practice
A production spam filter can sit at one point in the cube. Supervised (human flags), online (adapts to new campaigns), model-based (neural net or linear classifier on bag-of-words or embeddings). A nightly credit-risk batch job may be supervised, batch and model-based. A fraud score that updates from a stream of authorisations may be semi-supervised or anomaly-based, online and model-based. Customer segmentation for a campaign is unsupervised, usually batch, and may be instance-influenced if you assign new clients by nearest centroid.
Three axes combine into real system designs.
Self-supervised pretraining plus supervised fine-tuning is now the default for language and vision foundation models. Chapter 1's pet-repair story is the small picture of that pattern; Chapters 15-16 are the large one. Reinforcement learning remains a different beast: the "label" is a reward that may be sparse and delayed, and the data distribution depends on the policy being learned (Chapters 19).
Online learning's operational contract differs from batch. Batch systems need a retrain cadence, a champion/challenger swap and a data freeze for audit. Online systems need stream validation, drift alarms, rate limits on how fast parameters can move, and a kill switch when bad data arrives. Banks often prefer batch or carefully gated incremental updates for regulated scorecards even when pure online learning is technically available.
Operational contracts differ by learning mode.
Main challenges of machine learning
Your job is to select a model and train it on data, so failures are bad data or bad models. Data first, then algorithms, then deployment.
Insufficient quantity of training data
Humans generalise from tiny samples; most ML algorithms do not. Point at an apple a few times and a toddler recognises apples in many colours and shapes. Even simple ML problems typically need thousands of examples; image and speech systems may need millions unless you reuse parts of an existing model (transfer learning).
The unreasonable effectiveness of data. Banko and Brill (2001) showed that very different algorithms, including simple ones, performed almost identically on natural-language disambiguation once given enough data (Figure 1-21). Their line: reconsider the trade-off between spending on algorithm development versus corpus development. Norvig et al. (2009) popularised the slogan in "The Unreasonable Effectiveness of Data". Géron immediately qualifies it: small and medium datasets remain common, extra labels are often expensive, so do not abandon algorithms.
Illustrative: Banko and Brill style result that data volume can dominate algorithm choice.
Nonrepresentative training data
Training data must represent the cases you will see in production, for both instance-based and model-based learners. The happiness demo's country set omitted GDP below $23,500 and above $62,500. Adding those countries (Figure 1-22) changes the fitted line sharply: very rich countries are not happier than moderately rich ones, and some poor countries look happier than many rich ones. A linear model trained on the narrow band is unlikely to work at the extremes.
Sampling noise versus sampling bias. Small samples miss the population by chance (noise). Large samples can still be nonrepresentative if the sampling method is flawed (sampling bias).
Literary Digest 1936. About 10 million mail ballots, 2.4 million replies, high confidence that Landon would take 57%. Roosevelt won with 62%. Two flaws: address sources (phone books, magazine subscribers, clubs) favoured wealthier Republican-leaning voters; reply rate under 25% introduced nonresponse bias (people who dislike the magazine or ignore politics drop out).
Sampling bias: Literary Digest 1936.
YouTube funk example. Searching "funk music" assumes the ranking is representative of all funk on the platform. Results skew to popular artists; in Brazil you get funk carioca, not James Brown. Building large sets often forces imperfect proxies; document the skew.
Poor-quality data
Errors, outliers and noise hide patterns. Most data scientists spend a large share of time cleaning. If instances are clear outliers, discard or fix manually. If features are missing (5% of customers omit age), choose among: drop the attribute, drop the rows, impute (for example median age), or train twin models with and without the feature.
Irrelevant features
Garbage in, garbage out. The system learns only if training data holds enough relevant features and not too many irrelevant ones. Feature engineering has three moves:
- Feature selection. Keep the most useful existing columns.
- Feature extraction. Combine columns into better signals; dimensionality reduction helps.
- Create new features by gathering new measurements.
Overfitting the training data
The taxi-driver fallacy: one bad ride and you conclude every driver in the country is a thief. Machines do the same when capacity exceeds signal. Overfitting means strong training performance and weak generalisation.
Overfitting path and the regularisation fix.
Figure 1-23 shows a high-degree polynomial on life satisfaction that hugs training points and should not be trusted. Deep nets can fit noise when data are small or dirty. Feed country names as features and a complex model may "learn" that training countries with a w in the name have satisfaction above 7 (New Zealand 7.3, Norway 7.6, Sweden 7.3, Switzerland 7.5). That pattern is chance; it will not save you on Rwanda or Zimbabwe.
Remedies. Simplify (fewer parameters, fewer attributes, stronger constraints). Gather more data. Reduce noise (fix errors, remove outliers). Regularisation constrains the model to reduce overfitting risk. With θ₀ and θ₁ free you have two degrees of freedom (height and slope). Force θ₁ = 0 and you can only move a horizontal line toward the mean. Allow a small θ₁ and you sit between one and two degrees of freedom. Figure 1-24 shows an unregularised fit on circles, a refit including squares, and a regularised dashed line with smaller slope that generalises better to the squares. The regularisation strength is a hyperparameter: set before training, constant during training, not learned as a model parameter. Too large and the model is almost flat and underfits.
Underfitting the training data
The model is too simple to capture the structure. A linear happiness model is prone to underfit because reality is richer than a line; predictions can be inaccurate even on training examples. Fixes: select a more powerful model with more parameters; feed better features (feature engineering); reduce constraints (lower the regularisation hyperparameter).
Deployment issues
Even a well-fit model can fail operationally: too complex to maintain, too large for memory, too slow, poor scaling, security holes, or stale without updates. Modelling skill and MLOps skill differ; many firms staff a dedicated ML operations team for the second set.
Stepping back
Géron's own recap before evaluation:
- Definition. ML makes machines better at a task by learning from data instead of coding rules.
- Types. Supervised or not; batch or online; instance-based or model-based.
- Project shape. Gather a training set and feed a learning algorithm. Model-based methods tune parameters to fit then hope to generalise; instance-based methods memorise and compare by similarity.
- Failure modes. Too little data; nonrepresentative, noisy or irrelevant features; models that underfit or overfit; ignored deployment constraints.
Testing and validating
Hoping is not a strategy. Putting a bad model straight into production makes users complain. Prefer a held-out test set. Error on new cases is generalisation error (out-of-sample error). Low training error with high generalisation error means overfitting.
Split sizes. A common default is 80/20 train/test. On 10 million rows, holding out 1% still yields 100,000 test rows, often enough. Size the holdout to the precision you need, not to a slogan.
Hyperparameter tuning and model selection
Comparing a linear model to a polynomial on the test set, then tuning regularisation across 100 values on that same test set, burns the holdout. You may see 5% "generalisation" error in the lab and 15% in production because you adapted to that particular test sample.
Holdout validation (Figure 1-25): carve a validation set (dev set) from training data. Train candidates on the reduced training set, pick the winner on validation, retrain the winner on the full training set (including validation), then estimate generalisation once on the test set.
Holdout validation without burning the test set.
Validation set size trade-off. Too small: noisy model rankings. Too large: candidates train on much less data than the final model (sprinter selected for a marathon). Repeated cross-validation with many small validation folds averages evaluations for a stabler ranking at the cost of training time multiplied by the number of folds.
Data mismatch
Abundant training data may not look like production. Flower-species app: millions of web photos are easy; only 1,000 true mobile-app photos exist. Rule: validation and test must be composed of representative production-like pictures (shuffle, split half/half, avoid near-duplicates across sets).
Train-dev set (Andrew Ng's term, Figure 1-26). Hold out some web training images as train-dev. After training on the remaining web images: poor train-dev means overfitting (simplify, regularise, add or clean data). Good train-dev but poor production-like dev means mismatch; preprocess web images to look more like phone photos and retrain. When train-dev and dev both look good, read the test set once for the production estimate.
Flower-app mismatch diagnosis with train-dev.
No Free Lunch theorem
A model is a simplified representation that discards detail unlikely to generalise. Choosing a linear model assumes the signal is fundamentally linear and residuals are ignorable noise. Wolpert (1996): with no assumptions about the data, there is no reason to prefer one learner over another (No Free Lunch). For some datasets linear wins; for others, neural nets. You cannot evaluate every model, so you make reasonable assumptions and test a short list: regularised linear models for simple tasks; assorted neural nets for complex ones.
Wolpert: no universally best learner.
Exercise themes and compact answers
The chapter's oral exam (solutions in the Colab notebook at homl.info/colab-p):
- Define ML. Programs that improve at task T measured by P as experience E grows; or Samuel's learn-without-explicit-programming line.
- Four places it shines. Brittle rule lists; no known algorithm; shifting environments; insight and mining.
- Labelled training set. Examples paired with desired solutions.
- Two common supervised tasks. Classification and regression.
- Four unsupervised tasks. Clustering, visualisation or dimensionality reduction, anomaly or novelty detection, association rules.
- Robot walking unknown terrain. Reinforcement learning.
- Customer segments. Clustering.
- Spam detection. Supervised (labels from user flags), sometimes with anomaly components.
- Online learning. Incremental updates from a stream or mini-batches.
- Out-of-core. Incremental training on data larger than memory.
- Similarity-based predictions. Instance-based learning.
- Parameter versus hyperparameter. Learned during training versus set before training.
- Model-based search. Parameters that minimise a cost (or maximise utility); predict by applying the fitted model.
- Four challenges. Insufficient data; nonrepresentative data; poor quality; irrelevant features (also overfitting, underfitting, deployment).
- Great train, poor new data. Overfitting; simplify or regularise, add data, reduce noise.
- Test set. Estimate generalisation error on unseen cases.
- Validation set. Model selection and hyperparameter tuning without touching test.
- Train-dev. Diagnose overfitting versus data mismatch when abundant training data differs from production.
- Tune on test. Optimistic estimates; production disappointment.
LinearRegression API is stable; Chapter 2 prefers root_mean_squared_error (older squared-error helpers removed after 1.6). Align Colab pins before debugging "wrong" metrics.Footnotes the chapter expects you to absorb
Regression's odd name. Francis Galton studied heights and noticed children of tall parents tend to be shorter than their parents: regression to the mean. The methods he used to analyse correlations kept the name. When Chapter 4 says regression, it means predict a numeric target, not Galton's biological claim.
t-SNE figure animals. Figure 1-8 (Socher et al., zero-shot cross-modal transfer) shows animals separating from vehicles and horses near deer but far from birds. The point is that unsupervised visualisation can surface semantic structure without labels.
Photo clustering is messy in practice. The Google Photos story assumes one clean cluster per person. Real systems often create several clusters per person and occasionally merge lookalikes, so you may need multiple labels and manual cleanup.
Theta is convention, not magic. Footnotes remind readers that θ is the habitual Greek letter for parameters. Later chapters keep that notation through linear models, neural nets and beyond.
Disambiguation task in Banko and Brill. Their NLP problem was choosing among confusable words such as to/two/too from context. That is why "enough data flattens algorithm differences" is not a claim about every task; it is a claim about that family of problems at scale.
Practical checklist before Chapter 2
Before you open the housing notebook, confirm you can do all of the following without looking them up. State Mitchell's E/T/P definition in one sentence. Give one example each of supervised classification, supervised regression, clustering, anomaly detection and reinforcement learning. Explain why a spam filter should be framed as supervised learning even though production mail is unlabeled at score time (because training used labels). Draw the difference between batch retrain-and-swap and online incremental updates. Write the life-satisfaction linear equation and say which symbols are parameters. Explain why tuning a regularisation strength on the test set invalidates the published error. Say when you would invent a train-dev set.
If any item fails, reread that subsection. Chapter 2 assumes the vocabulary and will not redefine holdout validation or overfitting from scratch. The California housing project is long because it layers preparation, pipelines and evaluation on top of this map; it is not a substitute for it.
One more caution on chatbots in the opening. Géron lists ChatGPT, Gemini, Claude and Perplexity as proof that AI is now the product. That is a framing claim for 2025 readers, not a technical chapter on LLMs. The technical path to those systems runs through Chapters 14-17. Do not skip the classical ML core because the preface name-drops chatbots; the housing project and the training-model chapters still pay the bills in tabular bank and ops settings.
Metric preview toward Chapter 3
Chapter 1 promised that classification and regression share the need for held-out data. Chapter 3 shows that the choice of metric is itself a modelling decision: accuracy can bless a useless majority classifier. When you reread Chapter 1's spam-filter example, replace every casual "accuracy" with precision and recall. The overview map still holds; the scoreboard gets stricter.
Field notes for the ML map
Map supervised, unsupervised, semi-supervised, and reinforcement learning to problem statements before picking tools. Batch versus online; instance-based versus model-based; linear versus nonlinear. Main challenges: data scarcity, noise, irrelevant features, overfitting, underfitting, and misaligned metrics. Oct 2026: trees and linear models still ship in banks; deep nets win on unstructured data.
Closing depth note
Chapter 1 is the map: supervised, unsupervised, semisupervised, and reinforcement learning; batch versus online; instance-based versus model-based; the main challenges (data, features, overfitting, underfitting, and offline evaluation traps); and the test-set discipline that every later chapter assumes. Keep the train/validation/test story sacred when you reach deep models that invite endless fiddling.
Closing
The machine learning overview is not optional philosophy. It is the contract for every notebook that follows: define the problem, respect the split, measure the right metric, and distrust a beautiful fit that never saw fresh data.
2. End-to-End Machine Learning Project
Géron puts you in a fictitious Machine Learning Housing Corporation and walks a full project on 1990 California census districts: frame the business question, pick RMSE, fetch and stratify the data, explore geography and correlations, build a Scikit-Learn preparation pipeline, train linear, tree and forest models, fine-tune with grid and randomised search, then sketch launch, monitoring and MLOps. The chapter is the template every later modelling chapter assumes.
Working with real data
Study loop.
Learn on real datasets, not toys. Géron lists Google Dataset Search, Hugging Face Datasets, OpenML, Kaggle, UCI, Stanford SNAP, AWS Open Data, data.gov, DataPortals.org and Wikipedia's dataset list. The project uses the California Housing Prices set from StatLib (Pace and Barry), based on the 1990 California census. It is old on purpose for teaching: a Bay Area house was still "affordable" then. He adds a categorical attribute and drops a few features (Figure 2-1).
From open data list to the housing teaching set.
Look at the big picture
Task: predict median housing price for each district (census block group, typically 600-3,000 people) from population, median income and related metrics. Pull the project checklist at homl.info/checklist and adapt it; this chapter hits many items and skips some for later.
Frame the problem
Ask how the company will use the model. Here, predictions feed a larger investment system: good districts versus bad, along with other signals. That frames algorithms, metrics and how hard you tune. The current workflow uses experts' manual estimates; those are costly and their RMSE is about $100,000. Your model should beat that baseline.
Framing cascade before any fit.
Pipeline thinking. A pipeline is a sequence of data-processing components, often asynchronous, with data flowing store to store. Components are typically self-contained; if one breaks, downstream can often run on last-known outputs (for a while). Monitoring is still required. Figure 2-2 in the book sketches the housing investment pipeline with your price model as one stage.
Supervision and learning mode. Labeled median prices → supervised. Predict a value → regression. Multiple input features → multiple regression. One output per district → univariate (multiple outputs would be multivariate). No rapid stream and data fit in memory → batch learning. If data were huge, split with MapReduce-style batch or switch to online learning.
Select a performance measure
Default regression metric: RMSE (Equation 2-1): square root of mean squared prediction error. It penalises large errors heavily.
Choosing the regression norm.
RMSE(X, y, h) = √( (1/m) Σᵢ (h(x⁽ⁱ⁾) − y⁽ⁱ⁾)² )
Notation used for the rest of the book. m = number of instances in the evaluation set. x⁽ⁱ⁾ = feature vector of instance i; y⁽ⁱ⁾ = label. y = vector of all labels. X = matrix of all feature rows (row i is (x⁽ⁱ⁾)⊺). h = hypothesis (prediction function); ŷ⁽ⁱ⁾ = h(x⁽ⁱ⁾). Example district: longitude −118.29, latitude 33.91, population 1,416, median income $38,372, median house value $156,400.
When MAE is better. If many outliers make squared errors dominate, prefer mean absolute error (Equation 2-2): average of |h(x⁽ⁱ⁾) − y⁽ⁱ⁾|. Both are ℓk norms of the error vector: RMSE is ℓ2, MAE is ℓ1. Higher k focuses more on large errors; ℓ∞ cares only about the max. RMSE remains the chapter default because large pricing mistakes are costly.
Check the assumptions
List and verify assumptions with the downstream team. Example: if the investment system converts your prices into categories (cheap / medium / expensive) and only needs ranks, then classification metrics or ordinal framing might fit better than RMSE regression. Confirming early avoids building the wrong system.
Get the data
Ops contrast.
Colab path. The book notebooks run on Google Colab; save work to Drive; download data into the runtime. Interactivity is powerful and dangerous: out-of-order cell runs create hidden state. Prefer restart-and-run-all before trusting numbers. Book listings may differ slightly from notebooks; trust the notebook when following along.
Fetch helper. A `load_housing_data` style function downloads a tarball from Géron's GitHub data repo when missing, then returns a DataFrame. Working directory and caching conventions keep reruns offline.
Take a quick look at the data structure
Practice hub.
Shape and columns. 20,640 districts. Columns include longitude, latitude, housing_median_age, total_rooms, total_bedrooms, population, households, median_income, median_house_value, ocean_proximity. `info()` shows total_bedrooms at 20,433 non-null (207 missing). All numeric except `ocean_proximity` (object / categorical).
Category counts: `<1H OCEAN` 9136, `INLAND` 6551, `NEAR OCEAN` 2658, `NEAR BAY` 2290, `ISLAND` 5. `describe()` and histograms (Figures 2-7, 2-8) show scales, medians and tails. Median income is scaled (roughly tens of thousands of dollars as coded). Cap effects appear on age and house value (the target is capped near $500,000), which matters for metrics and for whether the team cares about distinguishing ultra-expensive districts.
Create a test set
Hold out before heavy exploration to avoid data snooping bias. A pure random split works if the dataset is large enough relative to the number of attributes. Géron still demonstrates stratified sampling on income, because median income strongly predicts median house value and a lucky random split can skew the income distribution.
Stratified holdout on income.
Method. Create an `income_cat` by binning median income (for example five categories with `pd.cut`). Use `StratifiedShuffleSplit` or `train_test_split(..., stratify=income_cat)` to build train/test. Compare income category proportions on the full set versus random versus stratified test (table in the book): stratification matches the full distribution; random drifts. Drop `income_cat` after splitting so it does not leak into features.
Explore and visualise
Work on a copy of the training set only.
EDA toolkit on the training copy.
Geography. Scatter longitude versus latitude; set alpha to see density. Overlay median house value as colour and population as size (Figure 2-11 style): coastal and urban clusters show high prices. The pattern suggests location features will matter; raw lat/lon may be weaker than distances to clusters or ocean proximity.
Correlations. Compute the correlation matrix; focus on the median_house_value column. median_income is the strongest positive correlate. Plot the scatter (Figure 2-13): a clear upward trend with some banding from capping and discrete value encoding.
Attribute combinations. Derive rooms_per_household, bedrooms_ratio (bedrooms/rooms), people_per_household. Recheck correlations: bedrooms_ratio often correlates more negatively with price than total rooms alone (more bedrooms per room can mean denser, cheaper stock). These ratios become pipeline features later.
Prepare the data for ML algorithms
Separate predictors and labels (`median_house_value`). Fit preparation only on the training set; apply the same transforms to test via stored statistics.
Clean the data
Missing total_bedrooms. Options: drop districts, drop the attribute, or impute. The book uses `SimpleImputer(strategy="median")` on numeric columns. Median is robust to outliers versus mean. Fit the imputer on training numerics; `statistics_` holds per-column medians for reuse at serve time.
Median imputation path.
Scikit-Learn design principles
Consistency. Estimators (`fit`), transformers (`transform` / `fit_transform`), predictors (`predict`). Inspection of hyperparameters and learned attributes (`strategy_`, `statistics_`). Nonproliferation of classes: NumPy arrays and SciPy sparse matrices, not custom wrappers. Composition: Pipelines and ColumnTransformers stack building blocks. Sensible defaults so baseline code stays short.
Categorical attributes
OrdinalEncoder maps categories to integers; bad for ocean proximity because INLAND is not "greater than" NEAR OCEAN in a numeric sense. OneHotEncoder creates one binary column per category (sparse optional). Handle unknown categories at transform time with `handle_unknown="ignore"` (or error during development). `categories_` and `get_feature_names_out()` keep columns auditable.
Encoding choice for ocean_proximity.
Feature scaling and transformation
Models sensitive to scale need transformers. Min-max scaling maps to a range; standardisation subtracts mean and divides by standard deviation (less sensitive to outliers). Long-tailed features often improve after log transforms. The chapter builds custom transformers and sklearn utilities (`FunctionTransformer`, later `sklearn.metrics.pairwise.rbf_kernel` for similarity features) to add cluster similarity to latitude/longitude: fit a clustering model on coordinates, then emit similarity-to-cluster-center features that beat raw lat/lon for price.
Numeric preparation stack.
Custom transformers. Implement `fit`/`transform` or use `FunctionTransformer` for stateless maps. For learned geo clusters, wrap a clustering estimator so the pipeline fits cluster centers on training districts only.
Transformation pipelines
`Pipeline` chains impute → other numeric steps → scaler. `ColumnTransformer` (or `make_column_transformer`) applies numeric and categorical pipelines side by side. Final prep goals the book lists:
Prep composition with ColumnTransformer.
- Impute numeric medians; impute categorical most-frequent if needed.
- One-hot encode categories.
- Add ratio features: bedrooms_ratio, rooms_per_house, people_per_house.
- Add geo cluster similarity features.
- Log-transform long-tailed numeric features.
- Standardise numeric features.
`fit_transform` on training housing yields `housing_prepared`. Mixed sparse/dense outputs: ColumnTransformer returns sparse if density is below `sparse_threshold` (default 0.3); this example stays dense.
- Impute missing numerics (median) and categoricals (mode).
- Encode categories with one-hot, not ordinal, for ocean proximity.
- Engineer ratio and geo-similarity features.
- Transform long tails (log) and standardise scales.
- Fit on train only; reuse statistics at evaluate and serve.
Select and train a model
Linear regression baseline. Pipe prep + `LinearRegression`. Predict on some training districts; compare to labels. Measure RMSE with `root_mean_squared_error`. Underfitting shows up as high training RMSE (still maybe better than $100k experts, but weak).
Illustrative RMSE index vs experts (lower better; tree train overfits).
Decision tree. `DecisionTreeRegressor` can drive training RMSE near zero and still generalise poorly: classic overfitting. Never trust training error alone.
Cross-validation. `cross_val_score` with `scoring="neg_root_mean_squared_error"` and k-fold (for example 10). Negate scores to read RMSE. Trees often show high mean CV RMSE and high variance across folds; linear models may underfit with tighter variance. RandomForestRegressor averages many trees and typically lands between: much stronger CV RMSE than a single tree, better than linear on this dataset. Inspect `pd.Series(rmses).describe()` for mean and spread.
Fine-tune your model
Grid search
`GridSearchCV` explores a parameter grid on the full pipeline (preprocessing hyperparameters included, for example `n_clusters` in geo features, `max_features` on the forest). Use negative RMSE scoring and CV. `best_params_` and `best_estimator_` recover the winner. Nested pipelines address parameters with double-`__` names (`preprocessing__geo__n_clusters`).
Search strategy tradeoff.
Randomised search
`RandomizedSearchCV` draws from distributions (`scipy.stats.randint` for cluster count and `max_features`) for `n_iter` draws. Better when the space is large: grid with six hyperparameters at 10 values each is a million fits; random search runs the budget you choose. HalvingGridSearchCV / HalvingRandomSearchCV allocate cheap partial resources in early rounds and promote survivors.
Ensembles and inspection
Blend strong diverse models (Chapter 6 preview): average a forest with a tuned k-NN regressor when errors differ. Inspect forest feature_importances_ aligned with `get_feature_names_out()`: log median income and INLAND indicator typically dominate; geo cluster features often outrank raw coordinates. Drop near-zero features if you need a leaner serve path.
Evaluate on the test set
Run the final pipeline once on the test set. Compute RMSE. Optionally compute a confidence interval for the generalisation error (t-based interval on squared errors in the book). Do not tweak after seeing the number; that burns the holdout. If you launched a previous model, compare old versus new on the same test design before swapping.
Honest final evaluation.
Launch, monitor and maintain
Production is more than a pickle file. Save the full pipeline (prep + model) with joblib. Deploy behind an API or batch scorer. Monitor prediction distribution and performance versus labels when labels arrive late (sale prices). Segment metrics (poor vs rich districts, rural vs urban). Monitor inputs: rising missingness, mean/std drift, new categorical levels. Keep model and dataset backups for rollback and forensics. Automate periodic retrain on fresh data and reject deploys that lose to the previous champion.
Launch and monitoring hub.
MLOps is the name for that infrastructure. First projects feel heavy; later ones reuse the rails. Géron's advice: prefer mastery of the process plus three or four algorithms over endless algorithm tourism. Try a full Kaggle-style project end to end next.
root_mean_squared_error, StratifiedShuffleSplit, ColumnTransformer, GridSearchCV, RandomizedSearchCV and the Halving* search classes used here. Prefer root_mean_squared_error over removed mean-squared-error helpers. Pipeline parameter addresses with __ remain the tuning interface. PyTorch is not required until Part II; keep the housing environment lean (pandas, sklearn, matplotlib).Exercise themes
SVR with linear and RBF kernels on a 5,000-row subset and 3-fold CV (SVMs scale poorly). Try adding a full-pipeline transformer. Explore `GridSearchCV` versus `RandomizedSearchCV` budgets. Inspect whether best models still rely on median income and INLAND. Save and reload the pipeline; score three new districts. Each exercise drills preparation-plus-model as one object, which is the deployment unit.
Deeper: framing questions you must answer
Business objective first. The model output is an input to a larger investment decision system. That means your RMSE competes with alternative signals, not with academic leaderboards. Ask who consumes the score, what decision threshold they apply, how often labels arrive (sale prices lag listings) and what a false high price costs versus a false low. Those answers pick the metric and the monitoring plan.
Current solution baseline. Experts estimate prices manually with roughly $100,000 RMSE. Document that number before you train. If your first linear model lands at $70k RMSE on CV, you have a story; if it lands at $110k, you do not ship. Baselines prevent celebrating useless wins.
Supervision checklist for this project. Labeled median prices → supervised. Continuous target → regression. Many input features → multiple. One target → univariate. Static CSV → batch. Write those five answers in the project brief so a reviewer can challenge them.
Deeper: RMSE, MAE and notation drills
Expand Equation 2-1. For each of m districts, predict ŷ⁽ⁱ⁾, subtract the label, square, average, square-root. Units stay in dollars, which stakeholders understand. Squaring means a $200k miss costs four times a $100k miss inside the sum (before the outer root). That is intentional when large errors wreck investment theses.
Equation 2-2 MAE. Average absolute miss. More robust when a few luxury districts would dominate RMSE. Géron frames both as ℓk norms of the error vector: RMSE ↔ ℓ2, MAE ↔ ℓ1. As k grows, the norm cares more about the worst misses; ℓ∞ is the max miss. Pick RMSE unless the product owner explicitly prefers robustness to outliers.
Notation flashcards. m = evaluation count. x⁽ⁱ⁾ = features of row i. y⁽ⁱ⁾ = label. X = design matrix. h = hypothesis. ŷ = prediction. Keep (i) in superscripts when you write by hand so sums stay unambiguous. The book reuses this notation through deep learning chapters.
Deeper: fetching and directory hygiene
Download once, cache forever. A small helper that checks for a local tarball, fetches from the book's data URL if missing, extracts and returns `pd.read_csv` keeps notebooks re-entrant. Pin the URL and expected SHA when you industrialise; teaching notebooks often skip checksums.
Colab-specific traps. Runtime disks vanish. Save to Drive or re-fetch. `!wget` in one cell and read in another breaks after restart unless you sequence them. Prefer a single `load_*` function at the top. Restart-and-run-all before you quote any RMSE in Slack.
Deeper: EDA numbers that matter
Missingness. 207 missing `total_bedrooms` out of 20,640 is about 1%. Dropping rows is tempting and wrong if missingness correlates with old rural stock; median impute is the book's default and is fine for the teaching target. In production, add a missing indicator feature so the model can learn "unknown bedrooms" as a signal.
ocean_proximity imbalance. ISLAND has five districts. One-hot still works; beware that a stratified split might put all five in train. For rare categories, consider grouping or targeted evaluation later.
Capped labels and features. Histograms show housing_median_age and median_house_value piled at the right edge. The target cap near $500,000 means two districts priced at $501k and $900k may share the same label. If the business cares about the top end, you need uncapped labels or a separate luxury model. Record the cap in the model card.
Median income scale. The feature is not raw dollars; it is a scaled census construct. Do not interpret a coefficient as "dollars of income" without reading the codebook. Correlation with price is still the strongest single signal in the matrix.
Deeper: stratified splitting procedure
1. Create `income_cat` with `pd.cut` on `median_income` into five bins (edges such as 0, 1.5, 3.0, 4.5, 6.0, ∞). 2. Run `StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)` or `train_test_split(..., stratify=income_cat)`. 3. Compare category proportions: full set versus stratified test versus random test. Stratified should match within tenths of a percent. 4. Drop `income_cat` from both splits. 5. Persist the split indices or a hash of row IDs so later notebook versions do not reshuffle silently.
Why income? Because it is the strongest predictor; a skewed test set makes your final RMSE lucky or unlucky. Stratification is not a substitute for a locked test set; it makes the lock fair.
Deeper: visualisation and correlation practice
Geo plot recipe. `plt.scatter(longitude, latitude, s=population/100, c=median_house_value, cmap=0.4, cmap='jet')`. High prices hug the coast and urban belts. That visual justifies geo cluster features and ocean proximity encoding.
Correlation matrix. `corr_matrix = housing.corr(numeric_only=True)` then sort `corr_matrix["median_house_value"]`. Expect median_income near the top (around 0.68 in the book), then weak positives for rooms and age, negatives for latitude (more northern inland cold spots in this state-level cut).
Scatter matrix. Pair plots of the top attributes reveal banding from caps and discrete encoding. Use them to pitch feature engineering, not to p-hack thresholds on the full training set.
Combination attributes.
- `rooms_per_household = total_rooms / households`
- `bedrooms_ratio = total_bedrooms / total_rooms`
- `people_per_household = population / households`
Recompute correlations: bedrooms_ratio often moves up the usefulness ladder. These become `FunctionTransformer` steps inside the numeric pipeline.
Deeper: imputer and sklearn API muscle memory
SimpleImputer. `strategy="median"` for numerics. `fit` on training DataFrame of numeric cols; `transform` train and test. `statistics_` is the vector you must version with the model. Mean is acceptable for symmetric features; median is safer here.
Design principles in practice.
- Estimator: anything with `fit`.
- Transformer: `transform` and usually `fit_transform`.
- Predictor: `predict`, sometimes `predict_proba`.
- Hyperparameters public in `__init__`; learned values end with `_`.
- Compose with `Pipeline` and `ColumnTransformer` instead of custom glue scripts.
Deeper: encoding and scaling choices
OrdinalEncoder vs OneHotEncoder. Ordinal implies order. Ocean proximity categories are nominal → one-hot. For high-cardinality categoricals later in your career, look at target encoding with care for leakage; not needed in Chapter 2.
handle_unknown. Set `"ignore"` for production so new categories become zeros; set `"error"` while debugging so you notice schema drift.
MinMaxScaler vs StandardScaler. Min-max packs into [0, 1] (or a custom range) and is sensitive to outliers. StandardScaler centres at zero with unit variance and is the book's default for the housing numerics after log tweaks. Trees and forests are less scale-sensitive; linear models and SVMs are not. Still scale inside a shared pipeline so model swaps stay fair.
Log tails. Population, total_rooms and friends are right-skewed. `np.log1p` before scaling often helps linear models. Keep the transform inside the pipeline so test rows get the same function (no refitting).
Deeper: geo cluster similarity features
Idea. Raw latitude and longitude are awkward for linear models. Fit a clustering model (k-means style) on training coordinates; emit a vector of similarities (RBF kernel) from each district to each cluster centre. Hyperparameter `n_clusters` becomes tunable in GridSearchCV via `preprocessing__geo__n_clusters`.
ClusterSimilarity feature path.
Why it works. Prices vary smoothly over local labour markets; cluster similarities act like soft region indicators without hard county boundaries. Feature importances later often rank some cluster columns above raw coordinates.
Deeper: full preparation pipeline code pattern
```python from sklearn.pipeline import Pipeline from sklearn.compose import ColumnTransformer from sklearn.impute import SimpleImputer from sklearn.preprocessing import OneHotEncoder, StandardScaler, FunctionTransformer
num_pipeline = Pipeline([ ("impute", SimpleImputer(strategy="median")), ("log", FunctionTransformer(np.log1p, feature_names_out="one-to-one")), ("scale", StandardScaler()), ]) # plus custom ratio and geo steps as in the notebook
cat_pipeline = Pipeline([ ("impute", SimpleImputer(strategy="most_frequent")), ("onehot", OneHotEncoder(handle_unknown="ignore")), ])
preprocessing = ColumnTransformer([ ("num", num_pipeline, num_cols), ("cat", cat_pipeline, cat_cols), ]) ```
Fit once on training predictors. Persist the whole object. Never call `fit` on test. `get_feature_names_out()` is mandatory when you audit importances.
Deeper: model bake-off details
LinearRegression. Fast underfitter. Training RMSE might sit near the high tens of thousands of dollars. Useful as a floor on how much signal linear structure captures after prep.
DecisionTreeRegressor. Can drive training RMSE toward zero. CV RMSE stays large → overfit. Lesson: training error is not a ship metric.
cross_val_score. Use `scoring="neg_root_mean_squared_error"`, `cv=10`. Negate for reporting. Publish mean and std of folds. Wide std means instability; dig into fold contents (coastal vs inland heavy folds).
RandomForestRegressor. Default hyperparameters already beat the tree and usually the linear model on CV. It is the chapter's practical winner before heavy tuning. Note: forests shuffle randomness; set `random_state` for apples-to-apples comparisons.
Deeper: grid versus random versus halving search
GridSearchCV. Dictionary of lists. Explodes combinatorially. Good for two or three hyperparameters with short lists. Include preprocessing knobs, not only model knobs.
RandomizedSearchCV. Dictionary of distributions; `n_iter` controls budget. Prefer when many hyperparameters or wide ranges. The book samples `n_clusters` and `max_features` with `randint`.
Halving searches. Successive rounds with growing resource (more rows or more iterations). Early cheap knockout for bad candidates. Use when wall-clock hurts.
Nested parameter names. `random_forest__max_features` addresses a step named `random_forest` inside a pipeline that also contains `preprocessing`. Typos fail silently as unused keys in some versions; verify with `full_pipeline.get_params().keys()`.
Deeper: feature importances and error analysis
Forest importances. Sort `feature_importances_` with names from the prep step. Expect log__median_income near the top and cat__ocean_proximity_INLAND high. Geo cluster columns should outrank raw lat/lon if the custom transformer worked. Near-zero importances are candidates for pruning if latency matters.
Typical importance order after tuning.
Residual review. Plot predicted versus actual; slice RMSE by ocean proximity and by income band. If inland errors dominate, improve inland features or collect better data. If the cap band sticks out, revisit the label policy.
Deeper: final test evaluation etiquette
1. Freeze hyperparameters from CV winners. 2. Fit the winning pipeline on the full training set. 3. Transform and predict the test set once. 4. Report RMSE and optionally a 95% confidence interval on the error. 5. Compare to the expert $100k baseline and to the previous production model if any. 6. Archive metrics, hashes of data and code commit IDs.
Do not go back to tuning after this glance. If you must, split a fresh holdout from future data.
Deeper: launch and MLOps checklist
Artefacts. joblib dump of the full pipeline; schema file for input columns; training data snapshot ID; metrics JSON; model card with caps and known failure modes.
Serving. Batch score nightly or API score on demand. Identical prep path; no reimplementation in Java "for speed" unless you have parity tests.
Monitoring.
- Label latency: sales close weeks later; use delayed evaluation windows.
- Segment metrics: rich vs poor, rural vs urban, each ocean category.
- Input drift: missing rate, mean/std vs training, unknown category rate.
- Output drift: prediction distribution shifts.
- Alert → page on-call → rollback to last champion.
Retrain cadence. Fresh census-like extracts or market feeds trigger scheduled training. Champion/challenger compare on a locked recent holdout. Human review for large swings in importances (regulatory expectation in bank twins of this project).
Bank and EA take: housing to credit
Swap labels. Probability of default, expected loss, or prepayment. Stratify on a risk driver (credit band, LTV bucket) the way this chapter stratifies on income. Impute with train-only medians; never peek at the book test. One-hot product codes with `handle_unknown="ignore"`. Forests give quick importances for challenger discussions; linear models give coefficient narratives for credit committees. RMSE becomes MAE of log-loss or calibration ECE as appropriate, but the pipeline discipline is identical.
Same pipeline discipline, new label.
EA ops angle. District population and rooms map to site load and capacity features when you predict demand for a field office. Same stratified holdout and drift monitors apply.
Exercise walkthrough notes
Exercise: SVR. `SVR(kernel="linear")` and `SVR(kernel="rbf")` with grids over `C` and `gamma`. Subsample 5,000 training rows and `cv=3` or you wait hours. Expect painful scaling versus forests; the lesson is computational cost as a product constraint.
Exercise: full-pipeline transformer. Add a custom transformer to the prep chain; prove `get_feature_names_out` still works; retune.
Exercise: model persistence. `joblib.dump` / `load`; score three hand-built districts; verify identical predictions across machines.
Exercise: feature ablation. Remove geo clusters; measure CV RMSE delta; decide if the complexity is worth it.
Chapter 2 figure map
| Figure | Meaning | |
|---|---|---|
| 2-1 | California housing prices teaching map | 2-2 |
| Investment pipeline with ML component | 2-7 | Numeric `describe` summary |
| 2-8 | Histograms of numeric attributes | 2-11 |
| Geo scatter by price and population | 2-13 | median_income vs median_house_value |
Closing expansion
What to carry forward. (1) Frame supervision and metric from business use. (2) Stratify when a driver dominates. (3) Fit prep on train only. (4) Prefer pipelines over notebook glue. (5) Trust CV and one locked test, not training error. (6) Ship monitoring with the model. Chapter 3 reuses the discipline on classification metrics where accuracy lies.
Stable train/test splits that survive data refreshes
Seeded shuffle is not enough. `random_state=42` keeps one CSV stable, then breaks when rows are inserted. Géron shows a hash-of-id rule: put instance i in the test set if `crc32(id) < test_ratio * 2**32`. New rows can join train or test by the same rule; old training rows never migrate into test.
Stable test membership strategies.
Implementation sketch. ```python from zlib import crc32 def is_id_in_test_set(identifier, test_ratio): return crc32(np.int64(identifier)) < test_ratio * 232 ``` Apply via an id column. Housing has no natural id: use `reset_index()` and treat the index as id only if** new data always appends and nothing deletes. Safer composite: `id = longitude * 1000 + latitude` (stable for geographic districts). Still watch collisions.
train_test_split. Scikit-Learn's helper adds `random_state` and multi-array alignment (features and labels split with the same permutation). Prefer it for the simple case; prefer hash ids when the dataset is a growing warehouse extract.
Income stratification in numbers
Why five bins. Median income drives price; a random 20% test can under-represent high-income districts and make final RMSE look lucky. `pd.cut` into bands (0-1.5, 1.5-3, 3-4.5, 4.5-6, 6+) mirrors the book's histogram shoulders. After `StratifiedShuffleSplit`, the test category fractions match the full set within fractions of a percent; a pure random split drifts by whole percentage points on the rare top bin.
Drop the helper column. `income_cat` is only for splitting. Leaving it in features leaks the stratification key and invents a feature you will not have at serve time in the same form.
Correlation and combination numbers from the book
After engineering ratios on the training copy, typical sorted correlations with `median_house_value` look like:
| Attribute | Corr (approx.) | |
|---|---|---|
| median_income | +0.69 | rooms_per_house |
| +0.14 | total_rooms | +0.14 |
| housing_median_age | +0.10 | bedrooms_ratio |
| −0.26 | latitude | −0.14 |
bedrooms_ratio beats raw bedroom and room counts: lower bedrooms-per-room associates with more expensive stock (think fewer cramped units). Warning from the book: avoid combination features that are near-linear clones of existing columns; collinearity hurts linear regression and interpretation.
OneHotEncoder versus pandas.get_dummies
get_dummies looks simpler and fails in production. It emits columns only for categories present in the current frame. A batch with two ocean values yields two columns; the training model expected five. Unknown categories silently create new column names.
Why production prefers OneHotEncoder.
OneHotEncoder remembers `categories_` from `fit`. `transform` always emits the same width and order. Unknowns raise by default; set `handle_unknown="ignore"` to emit an all-zero row for that feature group. Use `sparse_output=False` when you want dense NumPy immediately. For high-cardinality categoricals (country, merchant id), one-hot explodes; consider target encoding with leakage controls or embeddings (later chapters / bank feature stores).
Custom transformers: StandardScalerClone and ClusterSimilarity
Duck typing. A transformer needs `fit` (returns self), `transform`, and `fit_transform` (free via `TransformerMixin`). `BaseEstimator` without `*args/**kwargs` in `__init__` yields `get_params`/`set_params` for search.
StandardScalerClone lesson. Validate with `check_array` / `check_is_fitted`; store `mean_`, `scale_`, `n_features_in_`; `fit` signature includes `y=None` even when unused so pipelines can pass labels through.
ClusterSimilarity idea. Fit k-means (or similar) on training latitude/longitude; in `transform`, compute RBF kernel similarity from each row to each cluster centre; emit n_clusters features. Hyperparameter `n_clusters` (and RBF `gamma`) become searchable. This is the chapter's flagship custom transformer: skip once if rushed, return when GridSearchCV needs `preprocessing__geo__n_clusters`.
FunctionTransformer covers stateless maps: log1p, column ratios (`X[:, [0]] / X[:, [1]]`), and name functions for `get_feature_names_out`. Prefer it before writing a class.
Cross-validation mechanics
Figure 2-20. k-fold with k=10: each fold is validation once; train on the other nine. Ten RMSE scores → mean and std. Trees that post 0.0 training RMSE still post large CV RMSE: the smoking gun for overfit.
Scoring sign flip. Scikit-Learn scorers are utility functions (higher is better). Use `neg_root_mean_squared_error` and negate the array when reporting RMSE. Forgetting the negation produces mysterious negative "errors" in dashboards.
make_pipeline(preprocessing, model) keeps prep inside each fold so imputation statistics do not leak from the validation fold into training. Hand-prepping outside CV is a silent leak.
Linear versus tree versus forest on housing
LinearRegression underfits after rich prep: RMSE still high tens of thousands of dollars. Fixes: better features, more flexible model (regularisation is not the issue; the model is not regularised).
DecisionTreeRegressor memorizes: training RMSE 0.0, weak CV.
RandomForestRegressor averages trees on feature/sample randomness; CV RMSE drops sharply versus a single tree. It becomes the default challenger entering fine-tune. Set `random_state` for stable bake-offs.
Fine-tune search patterns in code
```python from sklearn.model_selection import GridSearchCV, RandomizedSearchCV from scipy.stats import randint
param_grid = [ {"preprocessing__geo__n_clusters": [5, 8, 10], "random_forest__max_features": [4, 6, 8]}, {"preprocessing__geo__n_clusters": [10, 15], "random_forest__max_features": [6, 8, 10]}, ] grid = GridSearchCV(full_pipeline, param_grid, cv=3, scoring="neg_root_mean_squared_error") grid.fit(housing, housing_labels)
param_distribs = { "preprocessing__geo__n_clusters": randint(3, 50), "random_forest__max_features": randint(2, 20), } rnd = RandomizedSearchCV(full_pipeline, param_distribs, n_iter=10, cv=3, scoring="neg_root_mean_squared_error", random_state=42) ```
Tips. List-of-dicts grids explore separate regions. Random search wins when dimensions grow. Halving* variants spend partial data early. Always put prep hyperparameters in the same search as model knobs when prep is inside the pipeline.
Presenting the solution
Stakeholders want limits, not only RMSE. Report: beat experts or not; which features dominate; failure modes (capped labels, island rarity, inland bias); reproducibility pack (GitHub, README, requirements.txt or environment.yml, seeds, Docker optional). Even if final RMSE only matches noisy expert ±30% band, shipping can free experts for harder cases. That is a product argument, not a leaderboard argument.
joblib deploy path
```python import joblib joblib.dump(final_model, "my_california_housing_model.pkl") # production: # import custom ClusterSimilarity, ratio helpers, etc. first final_model_reloaded = joblib.load("my_california_housing_model.pkl") predictions = final_model_reloaded.predict(new_data) ```
Custom classes must be importable in the serve process. Saving only the pickle without the transformer code fails at load. Prefer a small `housing_model` package with pinned versions.
Monitoring scenarios the book names
- Fresh data arrives; compare new-model RMSE to previous on an updated test window; deploy only if not worse.
- Subset metrics: poor/rich, rural/urban.
- Input quality: missingness spikes, mean/std drift, new categorical levels.
- Rollback: keep prior model and dataset versions.
More exercise detail
1. SVR grid on 5,000 rows, 3-fold, linear and RBF kernels, vary `C` and `gamma`. Note wall-clock versus RandomForest on full data. 2. Add a transformer (for example, a clipped income feature) inside the pipeline; confirm names and search still work. 3. Grid vs random on the same budget; compare best CV RMSE and winning params. 4. joblib round-trip and score three synthetic districts. 5. Feature importance ablation: remove low-importance columns; measure CV delta and serve latency.
Pitfalls checklist (print and tape)
Highest-cost mistakes in Chapter 2.
1. Fit imputer/scaler/encoder on the full dataset before splitting. 2. Trust training RMSE from an unregularised tree. 3. Use `get_dummies` in the serve path. 4. Ordinal-encode ocean proximity. 5. Tune on the test set. 6. Forget to import custom transformers before `joblib.load`. 7. Ignore the $500k label cap in the model card. 8. Skip input-drift monitors because offline RMSE looked fine.
Colab and notebook workflow in practice
Open homl.info/colab-p to see the chapter notebooks plus NumPy, Matplotlib, Pandas, linear algebra and calculus extras (Figure 2-3). Chapter 2's notebook is `02_end_to_end_machine_learning_project.ipynb` (Figure 2-4). Cells are Markdown or code; Shift-Enter runs. Create code cells from the Insert menu or the hover + Code control.
State discipline. Running cells out of order is the most common student failure mode. A variable from a deleted cell still lives in the kernel. Before you quote an RMSE to a colleague, Kernel → Restart and run all. Save notebooks to Drive; Colab VMs are disposable. Local installs follow the GitHub README if you prefer.
Code versus book drift. Géron warns that printed listings can lag the notebooks. When they disagree, trust the notebook and the library versions pinned there.
What `info`, `value_counts` and `describe` teach
`head()` shows ocean_proximity strings and numeric scales. `info()` flags the 207 missing bedrooms and the object dtype. `value_counts()` on ocean_proximity exposes the ISLAND rarity (five rows): any metric on that slice is noise. `describe()` gives count, mean, std, min, quartiles, max; nulls are skipped in count. Histograms (Figure 2-8) make the income shape, age cap and price cap obvious in one glance.
Std versus percentiles. Std summarises dispersion assuming a roughly symmetric shape; quartiles stay meaningful when tails are long. Housing ages and prices are not Gaussian; report both.
Hash split edge cases
Row-index ids fail if upstream reorders or deletes. Geo composite ids fail if two districts share rounded lat/lon (collision → both treated as one id). For bank twins, use a durable surrogate key from the warehouse (application_id) and hash that. Unit-test that a row's membership never flips when you append a month of data.
Visual EDA that changes the model
Alpha on geo scatters reveals overplotted urban cores. Colour by median_house_value and size by population (Figure 2-11) sells the need for location features to non-ML stakeholders faster than a correlation table. The income-price scatter (Figure 2-13) shows the main trend plus horizontal banding from capped labels and discrete value encoding; those bands are data artefacts, not model targets to fit.
Wikipedia-style correlation ellipses in the book remind you that correlation is linear. A strong U-shaped relationship can show near-zero Pearson correlation; plots catch what the matrix misses.
Imputation alternatives and when to use them
Drop rows if missingness is tiny and MCAR; here 1% is small, but dropping can still bias geography. Drop the column if it is mostly empty; total_bedrooms is not. Impute with median (robust) or mean (symmetric). Add a missing indicator column in production so the model can learn "unknown" as a signal (common in credit applications where blank fields correlate with risk). Fit indicators on train; freeze them for serve.
Scaling maths in one paragraph
Min-max: x' = (x − min) / (max − min) maps to [0, 1]. Sensitive to extreme min/max. Standardise: x' = (x − μ) / σ centres at 0 with unit variance. Robust alternatives (median and IQR) appear in later sklearn docs when outliers dominate; Chapter 2 sticks to StandardScaler after log1p on heavy tails. Always fit μ, σ, min, max on train only.
RBF similarity intuition
rbf_kernel(X, Y, gamma) decays with squared distance. Districts near a cluster centre get similarity near 1; far districts near 0. gamma controls the width. Too large gamma → every district looks unique; too small → all similarities blur. Cross-validate gamma with n_clusters rather than hand-waving.
Pipeline debugging habits
Name every step (`"impute"`, `"scale"`) so GridSearchCV addresses are readable. `set_output(transform="pandas")` (modern sklearn) keeps column names through transforms when you need auditability. `get_feature_names_out()` after fit is mandatory before you publish importances. If ColumnTransformer yields sparse and a downstream step needs dense, set `sparse_threshold=0` or explicitly.
Reading CV scoreboards
Publish a table: model, mean RMSE, std RMSE, wall time. Linear / Tree / Forest is the minimum bake-off. If forest std is high, check whether folds unevenly sample coasts. If all models sit near the expert $100k, the features may be weak or the label too noisy; spend the next week on data, not on another library.
Ensemble preview for this chapter
Average two diverse predictors when their errors anti-correlate. Example: forest + k-NN regressor. Simple mean often beats either alone. Stacking and blending wait for Chapter 6; here the point is to stop hunting one perfect model when a committee is cheaper.
Confidence interval on test RMSE
The book shows a t-based interval around the generalisation error using squared errors on the test set. Wide intervals mean the test set is small relative to noise; do not overclaim a 1% win over the previous champion. For regulated twins, pre-declare the non-inferiority margin before you see the number.
Documentation pack for launch
README: how to install, how to score one CSV, how to retrain. requirements.txt with pinned versions (sklearn==1.9.1, pandas, numpy, joblib). Model card: training window, label definition, cap at $500k, known weak slices (ISLAND), fairness notes if used for people-affecting decisions. Changelog: every deploy with metric deltas. Without this pack, a pickle in S3 is not a system.
End-to-end dry run script
A maintenance script should: pull fresh data; rebuild train/test with the same hash or stratification rules; retrain the champion configuration; evaluate on the locked recent window; run subset metrics; check input drift statistics; write a go/no-go JSON; if go, publish artefact and bump a model registry pointer; if no-go, alert. Chapter 2 sketches this; MLOps platforms implement it.
Mapping every checklist item
Géron points at homl.info/checklist. Map this chapter onto it: business framing [ok], data acquisition [ok], EDA [ok], prep [ok], shortlist models [ok], fine-tune [ok], present [ok], launch/monitor [ok]. Items skipped (for example, elaborate human labelling workflows) appear in later applied settings. Keep a personal checklist fork for bank projects with fair-lending review gates.
Worked numeric example for RMSE
Suppose three districts with labels 100, 200, 300 (thousands of dollars) and predictions 110, 180, 330. Errors +10, −20, +30. Squared: 100, 400, 900. Mean 466.67. RMSE √466.67 ≈ 21.6. MAE = (10+20+30)/3 = 20. RMSE > MAE because of the larger misses. Stakeholders feel RMSE when big misses hurt more than many small ones.
Why batch learning fits this dataset
20,640 rows fit in memory on any laptop. No streaming census. Retraining monthly from a warehouse dump is enough. If you later ingest nightly listing prices, revisit online learning or frequent batch refreshes. Do not invent streaming infrastructure for a static teaching CSV.
Inland versus coastal error slices
After you have a final model, compute RMSE for INLAND only versus `<1H OCEAN` only. Inland often harder: more heterogeneous markets, weaker ocean proxy. If inland RMSE dominates the headline number, propose more inland features (employment, commute, water rights) rather than another hyperparameter sweep on the global forest.
Random forest max_features intuition
max_features caps how many features each split considers. Smaller values increase tree diversity and sometimes generalisation; larger values let strong features (income) dominate every tree. Random search over 2..20 is the book's way of letting data choose. Do not confuse max_features with n_estimators; more trees usually help until returns flatten.
What "fine-tune" does not mean here
Not neural network fine-tuning. It means hyperparameter search and modest architecture choices (which prep steps, which model class). Keep the vocabulary straight when you jump to Part II, where fine-tuning means updating pretrained weights.
Synthesis: the eight steps as a weekly plan
Week 1: frame, metric, baseline expert number, fetch data, stratified split. Week 2: EDA and feature ideas, build prep pipeline with tests. Week 3: linear/tree/forest bake-off with CV tables. Week 4: random search, importance audit, test once, write model card, joblib + monitor hooks. That cadence matches how housing (and credit) projects actually move when the data already exist.
End-to-end walkthrough: from brief to registry
This section retells the chapter as a single project narrative so the pieces lock together. Read it after the topical sections; it is deliberately redundant on purpose. Redundancy is how procedure becomes habit.
Frame in writing. Supervised, multiple, univariate regression, batch learning, metric RMSE, secondary MAE if outliers dominate, success = RMSE meaningfully under $100k on a locked stratified test set plus a monitoring plan. You confirm with the downstream team that they need dollar prices, not cheap/medium/expensive labels. If they had wanted categories, you would have stopped and reframed as classification. That conversation is the cheapest save in the book.
Fetch and glance. `load_housing_data` pulls the tarball once and caches it. `head`, `info`, `value_counts`, `describe`, histograms. You note 20,640 rows, 207 missing bedrooms, five ISLAND districts, capped age and value, scaled income. You create `income_cat` and run a stratified 80/20 split. You verify category proportions against a random split table. You drop `income_cat`. You hash-backup the train and test indices to disk. Only then do you explore.
Explore on a training copy. Geo scatter with price colour and population size shows coastal wealth. Correlation matrix ranks median_income first. Scatter of income versus price shows the trend and the cap banding. You build rooms_per_house, bedrooms_ratio, people_per_house and recompute correlations; bedrooms_ratio jumps in usefulness with a negative sign. You write a short EDA note: "Location and income dominate; engineer ratios and geo similarities; respect label cap in the model card."
Prepare with a pipeline, not a script. Numeric branch: median impute, ratio features, geo ClusterSimilarity, log1p on long tails, StandardScaler. Categorical branch: most_frequent impute if needed, OneHotEncoder with handle_unknown ignore. ColumnTransformer concatenates. You unit-test that transforming the first five train rows twice is deterministic, and that test transform does not call fit. You print `get_feature_names_out()` and paste it into the design doc.
Bake-off. `make_pipeline(preprocessing, LinearRegression)` → CV RMSE high tens of thousands; underfit. Decision tree → training RMSE 0.0, CV RMSE poor; overfit. Random forest → CV RMSE clearly better; proceed. You record mean and std for each in a markdown table. You do not touch the test set.
Fine-tune. RandomizedSearchCV over n_clusters and max_features with n_iter=10 or more, cv=3, neg RMSE scoring. You also try a modest grid to sanity-check. Best estimator stays a full pipeline. Feature importances put log median income and INLAND on top; cluster features beat raw coordinates. You optionally average the forest with a tuned k-NN for a small ensemble gain.
Final test. Fit on full training data with winners. Predict test once. RMSE lands near your CV mean; confidence interval overlaps the CV band. You compare to $100k experts. You write the presentation: what worked, what did not, limits (cap, ISLAND, inland hardness), reproducibility pack.
Launch. joblib dump of the pipeline; package custom transformers; API or batch scorer; input drift monitors; segment metrics; registry entry with hashes; rollback pointer to the previous null model (experts). Schedule a monthly retrain dry run.
Why this narrative matters for banks. Replace price with expected loss, districts with accounts, ocean with channel, income with risk band. The Monday brief, stratified lock, pipeline prep, CV bake-off, one-shot test and monitors are the same play. The algorithm chapter changes; the procedure chapter does not.
Hyperparameter field notes
n_clusters. Too few → coarse regions; too many → sparse similarities and overfit risk. Plot CV score versus n_clusters after the search to see plateaus.
max_features for forests. Low values randomise splits more; high values lean on income every time. The sweet spot is data-dependent; that is why it is in the search, not hard-coded to sqrt(n_features).
n_estimators. More trees usually reduce variance until the curve flattens. Cost scales roughly linearly. For the teaching set, hundreds of trees are fine; for millions of rows, you budget explicitly.
CV folds. 10-fold is the chapter default for model comparison; 3-fold is acceptable inside expensive searches. Stratified CV variants matter more for classification (Chapter 3) than for this continuous target.
Data leakage museum (housing edition)
1. Fit StandardScaler on train+test, then split. 2. Use test districts in the k-means that builds geo clusters. 3. Choose income bin edges by peeking at test income distribution. 4. Engineer bedrooms_ratio using medians computed on the full CSV. 5. Tune max_features until test RMSE peaks, then report that test RMSE as unbiased. 6. Drop missing bedrooms using a rule learned with test rows included.
Each item has caused a real production embarrassment somewhere. The pipeline-plus-CV discipline exists to make these mistakes harder.
Scikit-Learn object lifecycle
Construction sets hyperparameters, not data. fit learns parameters (trailing `_` suffix). transform/predict uses learned parameters. clone copies an unfitted estimator with the same hyperparameters; GridSearchCV clones for each fold. If you manually reuse a fitted encoder on a new project without refitting, you are serving the old categories. Be explicit.
ColumnTransformer selectors
make_column_selector(dtype_include=np.number) and `dtype_include=object` keep the prep robust when column order changes. Named column lists are clearer for audits. Pick one style per project and stick to it. Mixing both without documentation confuses the next owner.
Sparse versus dense outputs
OneHotEncoder may emit sparse matrices. Numeric pipes emit dense. ColumnTransformer estimates output density and may return sparse when below `sparse_threshold` (default 0.3). Forests accept both; some custom transformers do not. When debugging a mysterious type error, print `type(housing_prepared)` and density.
Presenting numbers without lying
Say: "CV RMSE 48k ± 3k; test RMSE 49k; expert baseline 100k; label capped at 500k; inland RMSE 55k; ocean RMSE 42k." Do not say: "Our model is 50% more accurate than experts" without defining accuracy. Do not hide the cap. Do not average train and test RMSE. Committees forgive honest limits; they do not forgive silent test tuning.
Maintenance story after launch
Month 1: monitors quiet; predictions stable. Month 4: missing bedrooms rate doubles because a census field is delayed; input alarm fires; you impute as designed and page the data owner. Month 7: a challenger model with new employment features wins a pre-registered comparison; you swap the registry pointer and keep the old artefact for 90 days. Month 12: regulator asks how you avoid test leakage; you show the frozen split hashes and the pipeline code. That is what Chapter 2 is training you to be ready for.
Ties forward to Chapter 3
Classification will swap RMSE for precision, recall, ROC AUC and confusion matrices. The split hygiene, pipeline discipline and launch monitors remain. If you only skimmed prep, return here before MNIST metrics; otherwise you will tune thresholds on a contaminated holdout and ship a misleading precision number.
Glossary for Chapter 2
| Term | Meaning in this chapter | |
|---|---|---|
| District | Census block group (~600-3,000 people) | Hypothesis h |
| Prediction function | RMSE | Root mean squared error |
| Stratify | Preserve key distribution in splits | Pipeline |
| Ordered estimators with fit/transform | ColumnTransformer | Parallel column-wise prep |
| CV | Cross-validation | MLOps |
| Production ML operations | Champion | Currently deployed model |
Annotated bibliography inside the chapter
StatLib California housing (Pace and Barry) is the substrate. Scikit-Learn user guide sections on compose, model_selection and ensemble are the reference manuals. homl.info/checklist is the process spine.
Extended exercise solutions (approach, not spoilers)
SVR exercise. Build `make_pipeline(preprocessing, SVR())`. Grid over kernel, C, gamma on a 5,000-row subset with cv=3. Expect long fit times on RBF. Compare best SVR CV RMSE to the untuned forest; discuss whether the extra compute is worth it for this data size.
Transformer exercise. Write a `FeatureSlot` that clips median_income at a hyperparameter threshold; expose the threshold to GridSearchCV; prove names flow with `get_feature_names_out`.
Search budget exercise. Fix random_state and wall-clock budget; run GridSearchCV and RandomizedSearchCV; chart best score versus evaluations; note that random often wins early in wide spaces.
Persistence exercise. Dump, restart the kernel, reload, score three rows; fail the test if any prediction changes.
Ablation exercise. Remove geo features; re-run CV; if delta RMSE is small, consider dropping them for serve simplicity; if large, keep and document.
A closer look: what the chapter under-teaches
Fairness and geography. Housing prices entangle with segregation. A model that faithfully predicts prices can still be harmful if used to deny services. Géron's fictitious investment firm sidesteps that; your bank twin must not. Add slice metrics by sensitive proxies only with legal guidance.
Label delay. The chapter's monitoring section mentions delayed labels; it does not build a full delayed-evaluation store. Production needs one.
Feature store semantics. Pipelines in a notebook do not replace point-in-time correct feature joins. When you leave the CSV world, study leakage from future aggregates.
Uncertainty. A point RMSE does not give prediction intervals per district.
Since publicationenvironment pin (housing project)
```text python>=3.11 scikit-learn==1.9.1 pandas>=2.2 numpy>=2.0 matplotlib>=3.9 joblib>=1.4 scipy>=1.13 ```
PyTorch is unnecessary for Chapter 2. Keep the virtualenv lean. If Colab preinstalls differ, install the pins in the first notebook cell and restart once.
Bank case vignette: district prices to branch targets
An EA-oriented retail bank wants to predict next-quarter mortgage origination volume by micro-market. Features look like housing's: income, density, distance-to-coast proxies, product mix (one-hot). Labels arrive with a quarter lag. The team copies Chapter 2's stratified holdout on a volume band, median imputes, OneHotEncoder with ignore, forest bake-off, and a champion/challenger registry. They add calibration plots because lending committees ask for them. The first production bug is `get_dummies` in a shadow Excel path; they delete it after reading this chapter's warning. That vignette is why the housing story earns its pages.
Compression of the eight main steps
1. Big picture: use, frame, metric, assumptions. 2. Get data: fetch, cache, glance. 3. Explore: train-only EDA, correlations, ratios. 4. Prepare: impute, encode, engineer, scale, pipeline. 5. Shortlist models: linear, tree, forest with CV. 6. Fine-tune: grid/random/halving, importances, ensembles. 7. Present: limits, reproducibility, decision. 8. Launch: joblib, monitors, retrain, rollback.
Miss step 1 and you build the wrong thing. Miss step 8 and you built a demo.
Final reminders before you close the notebook
Fit on train. Validate with CV. Touch test once. Persist the pipeline. Monitor inputs. Document the cap. Everything else in Chapter 2 is detail in service of those sentences. When a future chapter introduces a glamorous algorithm, return to those sentences before you ship it.
Full procedure card (printable)
Purpose. Predict district median house value for investment screening. Metric. RMSE (MAE secondary). Baseline. Expert RMSE ≈ $100,000. Data. California housing teaching CSV, 20,640 districts. Split. Stratified 20% test on income_cat; persist indices; drop income_cat. Prep. ColumnTransformer: numeric (median impute, ratios, ClusterSimilarity, log1p, StandardScaler) + categorical (OneHotEncoder, handle_unknown=ignore). Models. LinearRegression (underfit baseline), DecisionTreeRegressor (overfit lesson), RandomForestRegressor (primary). Validation. 10-fold CV with neg_root_mean_squared_error; search with RandomizedSearchCV on n_clusters and max_features. Test. One shot; archive hashes. Ship. joblib full pipeline + custom classes + model card (label cap $500k, ISLAND rarity, inland slice). Monitor. Missingness, drift, segment RMSE, champion/challenger monthly.
Why each prep step exists (one line each)
Median impute keeps rows and resists outlier income spikes. Ratios express intensity (rooms per household) instead of raw totals that scale with district size. ClusterSimilarity turns lat/lon into soft region indicators linear models can use. log1p pulls long tails toward bell shapes that linear models prefer. StandardScaler equalises feature influence for scale-sensitive learners. One-hot avoids fake order on ocean proximity. handle_unknown=ignore stops serve crashes when a new category appears; you still monitor the unknown rate.
Decision tree zero-error autopsy
When training RMSE hits 0.0, the tree has grown until each leaf is pure on the training labels (or close enough with defaults). That is memorisation. CV RMSE exposes it because each fold validates on districts the tree did not see. The fix is not "more data in the training score"; it is constraints (max_depth, min_samples_leaf) or an ensemble. Chapter 5 details the knobs; Chapter 2 only needs you to stop trusting training error.
Random search versus human intuition
Humans guess that n_clusters=10 and max_features=sqrt are fine. Random search often finds n_clusters in the mid-twenties and max_features off the default that shave a few thousand dollars of RMSE. Those shavings matter when you are fighting to undercut a $100k expert baseline. Budget at least enough iterations to visit each hyperparameter across a wide range once; ten iterations is a demo, not a production search.
Reproducibility minimum viable pack
1. Git commit hash of training code. 2. Data snapshot id or file checksum. 3. `requirements.txt` pins. 4. `random_state` values. 5. Saved CV table and final test metric. 6. Model card markdown. 7. joblib artefact plus custom module version.
Without (1)-(3), you cannot rebuild. Without (5)-(6), you cannot defend. Without (7), you cannot serve.
Slice evaluation template
Compute RMSE for: INLAND, `<1H OCEAN`, NEAR OCEAN, NEAR BAY, income_cat bottom bin, income_cat top bin, high population districts, low population districts. Paste into a table in the launch review. Any slice worse than 1.3× overall RMSE needs a sentence in the model card. ISLAND is too small to judge; say so.
What you should be able to explain aloud
In ten minutes to a manager: the business frame, why RMSE, why stratified split, what the pipeline does, why the tree's zero training error was a trap, what the forest importances say, what the test RMSE was versus experts, what will page you after launch. If you cannot, rerun the notebook from scratch and narrate as you go.
Operator's appendix for Chapter 2
Command order that avoids leakage. split → explore train copy → fit prep on train → transform train/test → CV on train pipelines → search → fit winner on full train → test once → dump. Any other order is a smell.
Minimum tests before ship. (1) Transform determinism on five rows. (2) Unknown category does not crash. (3) Reloaded joblib matches live predictions. (4) CV function never sees test indices. (5) Model card lists the $500k cap.
When to stop tuning. When CV gains fall inside the fold noise (std across folds), stop. Further search burns time and invites accidental test peeks. Pocket the gain for a better feature instead.
Hand-off language. "Full pipeline artefact, stratified split hashes H, test RMSE R, inland RMSE R_i, monitors M enabled, rollback pointer P." If a sentence is missing, the hand-off is incomplete.
Link to Part II. Nothing here requires gradients. When you reach PyTorch chapters, keep the same split and monitoring ethics; only the model object inside the pipeline changes (and often the pipeline becomes a training script plus a TorchServe or ONNX path). The housing project remains the procedural North Star for tabular work.
Numbers to memorize from Chapter 2
- 20,640 districts in the teaching set.
- 207 missing `total_bedrooms`.
- 5 ISLAND districts.
- ≈$100,000 expert RMSE baseline.
- ≈$500,000 label cap on `median_house_value`.
- 0.69 approximate correlation of median_income with price.
- −0.26 approximate correlation of bedrooms_ratio with price after engineering.
- 20% stratified test share.
- 10 folds in the model-comparison CV.
- 0.3 default `sparse_threshold` on ColumnTransformer.
Carry these into interviews and design reviews; they show you actually ran the notebook.
Bridge into classification metrics
Housing used RMSE because the label was continuous. The moment your label becomes a class - default / non-default, spam / ham, digit 5 / not-5 - Chapter 3's toolkit replaces RMSE. Keep the pipeline, the stratified split and the one-shot test rule; swap only the scoreboard. Teams that keep RMSE habits when predicting rare events are the teams that ship 99% accurate fraud models that catch nothing.
Ch2 panel 200
This stretch centers on DataPortals. Source cue: 2 End to End Machine Learning Project In this chapter you will work through an example project end. Restate the mechanism and the validation discipline that keeps it honest.
Since publicationcheck: sklearn 1.9.x APIs for pipelines and HGB; keep trees as tabular default. Bank/EA: baselines before nets.
This stretch centers on DataPortals, StatLib. Source cue: U S Government s Open Data DataPortals org Wikipedia s list of machine learning datasets In this chapter.
Bank/EA: baselines before nets.
This stretch centers on Ch2. Source cue: learn from this data and be able to predict the median housing price in any district given all.
Bank/EA: baselines before nets.
Ch2 panel 201
This stretch centers on Ch2. Source cue: that your model s output a prediction of a district s median housing price will be essential to.
Bank/EA: baselines before nets.
This stretch centers on Ch2. Source cue: more than 30 This is why the company thinks that it would be useful to train a model.
Bank/EA: baselines before nets.
This stretch centers on Ch2. Source cue: breaks down the downstream components can often continue to run normally at least for a while by just.
Bank/EA: baselines before nets.
Ch2 panel 202
This stretch centers on MapReduce, RMSE. Source cue: predict a value More specifically this is a multiple regression problem since the system will use multiple features.
Bank/EA: baselines before nets.
Closing
End-to-end discipline beats clever models. Frame, split, explore, prepare, shortlist, fine-tune, present, monitor.
3. Classification
After the housing regression, Géron switches to classification on MNIST: 70,000 handwritten digits. A binary 5-detector with SGDClassifier teaches why accuracy lies, how to read a confusion matrix, and how precision, recall, F1, PR curves and ROC AUC trade off. The chapter then scales to multiclass (OvR/OvO), error analysis, multilabel and multioutput cleaning, with exercises that include a spam classifier.
MNIST: the hello-world of classifiers
Study loop.
70,000 28×28 grayscale digit images, labelled 0-9, from high-school students and US Census Bureau employees. New algorithms still get an MNIST sanity check. Fetch via OpenML:
MNIST load and official split.
```python from sklearn.datasets import fetch_openml mnist = fetch_openml("mnist_784", as_frame=False) ```
`sklearn.datasets` families. `fetch_*` downloads real sets; `load_*` ships small toys; `make_*` synthesises arrays for tests. Most return a Bunch (dict-like) with `DESCR`, `data`, `target`, often `feature_names` and `categories`. `fetch_openml` defaults to DataFrame/Series unless `as_frame=False`, which you want here for NumPy-style indexing.
Shapes. `X` is `(70000, 784)` uint8-like intensities 0-255; `y` is string labels (`'0'`…`'9'`). Reshape a row to 28×28 to plot (Figure 3-1). The official split is 60,000 train / 10,000 test already shuffled in the conventional ordering:
```python X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:] ```
Figure 3-2 shows a grid of digits. Keep the test set locked, same ethics as Chapter 2.
Training a binary classifier
Task: detect 5s versus non-5s. Build labels `y_train_5 = (y_train == '5')`. Fit `SGDClassifier` (stochastic gradient descent on a linear model; defaults work as a teaching SVM-like classifier):
Binary 5-detector path.
```python from sklearn.linear_model import SGDClassifier sgd_clf = SGDClassifier(random_state=42) sgd_clf.fit(X_train, y_train_5) ```
Predict on a known 5 image to sanity-check. SGD is sensitive to scaling and to `random_state`; pin seeds for bake-offs.
Performance measures
Ops contrast.
Accuracy and why it fails
Cross-validated accuracy near 95% looks great until you notice that about 90% of images are not 5s. A DummyClassifier that always predicts the majority class (not-5) also scores ~90%. Accuracy is a poor headline metric under class imbalance. This is the chapter's first hard lesson and the one credit-risk and fraud teams relearn every year.
Why accuracy is not enough.
Manual StratifiedKFold. For control beyond `cross_val_score`, clone the estimator per fold, fit on train indices, score on val indices, print fold accuracies. Stratification keeps 5/non-5 ratios stable across folds.
Confusion matrices
Count every (true class, predicted class) pair. For binary 5-detection: TN, FP, FN, TP (Figure 3-3). Use `cross_val_predict` to get out-of-fold predictions on the training set, then `confusion_matrix(y_train_5, y_train_pred)`. Never compute the matrix on the locked test set while still choosing models.
Binary confusion cells.
Precision, recall and F1
Precision = TP / (TP + FP) - of the predicted 5s, how many are right? Recall = TP / (TP + FN) - of the true 5s, how many did you catch? (also sensitivity / TPR)
Precision versus recall.
Book ballpark for the SGD 5-detector: precision ≈0.84, recall ≈0.65. Suddenly 95% accuracy looks less heroic.
F1 is the harmonic mean (Equation 3-3): `F1 = 2 × precision × recall / (precision + recall)`. It punishes extreme trade-offs. Call `f1_score(y_train_5, y_train_pred)`. Use F1 when you need one number and care about both errors; do not worship it when the business weights FP and FN asymmetrically.
The precision/recall trade-off
Practice hub.
SGD exposes `decision_function` scores. Raising the threshold makes positive calls rarer: precision tends to rise, recall falls. Example: threshold 0 may catch a sample 5; threshold 3000 may miss it.
Threshold selection flow.
Build the curve. `y_scores = cross_val_predict(..., method="decision_function")` then `precision_recall_curve(y_train_5, y_scores)`. Plot precision and recall versus threshold (Figure 3-5). Precision is bumpier than recall: a single threshold step can drop precision (for example 4/5 → 3/4) while recall only moves one way as the threshold rises.
PR curve (Figure 3-6). Precision versus recall directly. Pick an operating point from business cost (for example, demand ≥90% precision and accept the recall you get). `threshold_for_precision = thresholds[np.argmax(precisions >= 0.90)]`.
The ROC curve
Fit discipline.
ROC plots TPR (recall) versus FPR where `FPR = FP / (FP + TN)`. `roc_curve` + `roc_auc_score` summarise the curve; AUC 1.0 is perfect, 0.5 is random. Book SGD AUC ≈ 0.96.
Which curve to lead with.
PR versus ROC tip. Prefer PR when positives are rare or when false positives hurt more than the ROC picture suggests. ROC can look excellent under heavy imbalance while PR still shows a long way to the top-right corner. Fraud detection almost always wants PR (and precision at fixed recall) in the steering deck.
Warning. You cannot compare models on different test mixes without care; curves inherit the class prior.
Random forest binary comparison
Error parts.
`RandomForestClassifier` has no `decision_function`; use `predict_proba` and take the positive column as scores for `precision_recall_curve` and ROC. Forests often dominate SGD on PR/F1 for MNIST 5-detection in the book. Compare curves on the same CV predictions, not on training scores.
Multiclass classification
Train vs serve.
Native multiclass: LogisticRegression, RandomForestClassifier, GaussianNB. Binary-only: SGDClassifier, SVC (without multiclass wrappers). Strategies:
Multiclass options in sklearn.
- OvR / OvA: one binary classifier per class (10 for digits); pick the highest score at serve time.
- OvO: one classifier per pair → `N(N−1)/2` = 45 for MNIST; each trains on a smaller subset. Prefer OvO when the base algorithm scales poorly with rows (classic SVMs). Prefer OvR for most other binary learners.
Scikit-Learn auto-wraps binary estimators with OvR or OvO when you `fit` on multiclass `y`. `decision_function` then returns one score per class. For SVC on full MNIST, expect slow fits; subsample while learning.
Error analysis. Fit a multiclass model; `cross_val_predict`; display confusion matrices with `ConfusionMatrixDisplay`. Normalise by true class (`normalize="true"`) to see recall-style errors; normalise by prediction (`normalize="pred"`) to see precision-style errors. Book figures show many confusions into 8. Interpreting percentages: a cell that says 36% of errors on 7s went to 9 is not "36% of 7s"; check the absolute confusion too (often ~3% of 7s).
Error analysis loop.
Improvement levers. More data on confusable digits; features such as closed-loop counts (8 has two, 6 one, 5 none); preprocessing with Scikit-Image/Pillow/OpenCV. Plotting 3-versus-5 mistakes (Figure 3-11) shows that some errors are human-hard; pixels alone may not finish the job.
Multilabel classification
Model promotion.
Each instance can have several binary labels. Example: `(y >= '7')` and `(y % 2 == 1)` → "large" and "odd". Wrap a binary classifier in `KNeighborsClassifier` (or another multilabel-capable model) and fit on a multilabel target matrix. Score with F1 per label averaged (`average="macro"` or `"weighted"`). Multilabel matters for tag prediction, document topics and sanction-list hit types.
Multilabel path.
Multioutput classification
Generalisation of multilabel where each label can have more than two values. Teaching example: denoise MNIST. Add noise to images; target is the clean image pixels (each pixel a multi-valued output). A k-NN multioutput model predicts a cleaned image (Figures 3-12, 3-13). This bridges toward image models in Part II without leaving sklearn.
Multioutput denoise sketch.
Exercise themes
Try a better MNIST classifier (for example `KNeighborsClassifier` with grid search on weights and n_neighbors); aim at 97%+ accuracy with care for compute. Data augmentation by shifting digits one pixel; retrain and measure. Spam classifier (harder): fetch spam/ham, prepare text (Chapter 2 pipeline habits + text vectorisers), build a classifier, report precision/recall not just accuracy. These exercises glue Chapters 2-3 into a portfolio piece.
SGDClassifier, RandomForestClassifier, precision_recall_curve, roc_auc_score, ConfusionMatrixDisplay and fetch_openml. Prefer ConfusionMatrixDisplay.from_predictions over hand-rolled heatmaps. For production text spam, also evaluate linear models on HashingVectorizer / TfidfVectorizer; deep transformers wait until Chapters 14-15.Deeper: fetching MNIST and Bunch fields
Why `as_frame=False`. DataFrames are great for housing; for 784 pixel columns they add overhead and awkward column names. NumPy lets you reshape and slice quickly. If you need a DataFrame later, wrap explicitly.
Label dtype. OpenML may return string labels. Comparisons like `y_train == '5'` are correct for strings; if you cast to int with `y.astype(np.uint8)`, compare to `5` without quotes. Mixing the two silently yields all-False masks and a classifier that never sees positives. Assert `y_train_5.mean()` is near 0.09 before fitting.
Train order. The conventional MNIST order is already shuffled. If you ever concatenate extra data, reshuffle with a fixed `RandomState` or you will train on sorted digits and validate on a different digit mix.
Deeper: SGDClassifier as a teaching default
What it optimises. Linear classifier trained with stochastic gradient descent; loss defaults have varied by version (hinge ≈ linear SVM historically; check `loss` in 1.9.1). It supports `partial_fit` for online learning, which links back to Chapter 1's online versus batch discussion.
Hyperparameters you will touch. `alpha` (regularisation strength), `loss`, `penalty`, `max_iter`, `tol`, `random_state`. Scale inputs for best behaviour; raw pixels 0-255 are a rough starting point for the book, but `StandardScaler` on pixels often helps SGD. Forests care less about scaling.
Warm-up prediction. Take one digit you can recognise by eye; call `predict` and `decision_function`. If the score is barely positive, you are near the boundary; that digit is a good threshold demo later.
Deeper: the dummy baseline ritual
```python from sklearn.dummy import DummyClassifier d = DummyClassifier(strategy="most_frequent") # cross_val_score(d, X_train, y_train_5, cv=3, scoring="accuracy") ```
Always log the dummy. If your model is within a point of the dummy, you have not learned. For fraud, the dummy that always says "legit" will post 99%+ accuracy; that is why accuracy is banned in those reviews.
Deeper: confusion matrix numeracy
Suppose CV predictions yield approximately:
| Pred non-5 | Pred 5 | -- | ||
|---|---|---|---|---|
| Actual non-5 | TN ≈ 53k | FP ≈ 700 | Actual 5 | FN ≈ 1.9k |
Precision ≈ 3530/(3530+700) ≈ 0.84. Recall ≈ 3530/(3530+1891) ≈ 0.65. Accuracy ≈ (53k+3.5k)/60k ≈ 0.95. Watch how accuracy hides 1.9k missed 5s. In fraud, FN is stolen money; in cancer screening, FN is missed disease; in spam, FN is inbox spam. Name the costly cell before you tune.
Deeper: F1 and asymmetric costs
Harmonic mean stays near the worse of precision and recall. If the business needs precision 0.99 and tolerates recall 0.2, F1 looks bad while the system may be correctly tuned. Use F1 for balanced teaching and leaderboards; use cost-weighted metrics or constrained optimisation for production. Document the choice in the model card.
Deeper: threshold selection workshop
1. Collect out-of-fold `decision_function` scores. 2. Compute precision/recall arrays. 3. Find the smallest threshold that achieves the precision floor (or recall floor) required by policy. 4. Freeze that threshold as a serve-time constant next to the model artefact. 5. Monitor weekly whether the realised precision drifts as the score distribution shifts.
Never pick the threshold on the locked test set. Treat threshold like a hyperparameter with its own validation fold or a policy constant set from CV curves.
Deeper: why precision bumps
Figure 3-4 in the book (score-sorted digits with a moving threshold) shows that raising the threshold by one instance can remove a true positive that was "carrying" precision, so precision drops even though you became stricter. Recall cannot increase when the threshold rises; it steps down only when a true positive crosses below the threshold. That asymmetry is why the precision curve looks noisy.
Deeper: forest probabilities as scores
`predict_proba` returns columns that sum to 1. For binary, use `[:, 1]`. Calibrated probabilities are not guaranteed for raw forests; for thresholding on probability (rather than ranking), consider `CalibratedClassifierCV` later. For PR/ROC comparisons, ranking quality matters more than perfect calibration.
Deeper: OvR versus OvO decision sheet
| Question | Prefer | |
|---|---|---|
| Many classes, large N, slow pairwise blow-up? | OvR | Base learner is kernel SVM on large rows? |
| OvO | Need simple explainability of one-vs-rest scores? | OvR |
Scikit-Learn's automatic choices are decent defaults; override with `OneVsRestClassifier` / `OneVsOneClassifier` when you measure a reason.
Deeper: error analysis playbook
1. Plot raw and normalised confusion matrices. 2. Identify the brightest off-diagonal cells. 3. Sample misclassified images from those cells. 4. Separate "human-ambiguous" from "obvious model mistakes". 5. For obvious mistakes, check preprocessing, class weights and data bugs. 6. For ambiguous cases, consider richer features or human review. 7. Log a ticket per systematic failure mode; do not only bump `max_iter`.
On MNIST, chasing 8-confusions with loop-count features is the book's concrete illustration of feature engineering after metrics.
Deeper: multilabel metrics pitfalls
Macro F1 treats labels equally; weighted F1 weights by support. If "odd digit" is balanced but "large digit" is not, the two averages disagree. Report both. Hamming loss is another multilabel summary; still pair it with per-label recall when labels are alerts.
Deeper: multioutput denoising as a conceptual bridge
The noisy-pixel task shows that "classification" APIs can predict structured outputs. Each pixel is a label with 256 classes in the naive view; k-NN predicts a plausible clean patch from similar noisy neighbours. Part II will attack images with CNNs that share computation across pixels; remember this baseline when someone claims deep learning is the only way to denoise.
Bank and EA take: classification metrics that survive review
Fraud / ATO. Positives rare. Lead with PR AUC, precision at recall = investigation capacity, and dollar-weighted FN. Dummy majority baseline mandatory. Thresholds owned by ops, not by data scientists alone.
Teaching set versus fraud reality.
Credit decisioning. Often binary "approve" with regulatory constraints. Disparate error rates across slices matter as much as global recall. Confusion matrices per slice belong in the packet.
Ops EA triage. Multilabel tags on tickets (billing, outage, abuse) match the multilabel section; macro F1 for rare tags, weighted F1 for volume.
Exercise coaching
97% MNIST knn. Start with `KNeighborsClassifier(n_neighbors=3)`; grid `weights` in {uniform, distance} and a few odd `n_neighbors`. Use a smaller search CV while iterating. Data augmentation: shift images ±1 pixel; concat to training set; watch memory.
Spam classifier. Use public spam corpora; `train_test_split` with stratify; `TfidfVectorizer` inside a pipeline; compare multinomial NB and linear SGD; report precision/recall at a spam-threshold chosen on CV. Do not celebrate accuracy.
Figure map (Chapter 3)
| Fig | Meaning | |
|---|---|---|
| 3-1 | Single MNIST digit | 3-2 |
| Digit grid | 3-3 | Illustrated confusion matrix |
| 3-5 | Precision/recall vs threshold | 3-6 |
| PR curve | 3-7 | ROC curve |
| 3-9/3-10 | Multiclass confusion (counts / normalised) | 3-11 |
| 3-vs-5 error collage | 3-12/3-13 | Denoise multioutput example |
Pitfalls checklist
1. Celebrating 95% accuracy on 5-detection. 2. Fitting threshold on the test set. 3. Comparing ROC AUC only under heavy imbalance. 4. Mixing string and int labels in masks. 5. Reading error-normalised confusion cells as absolute rates. 6. Forgetting `method="decision_function"` / `predict_proba` in `cross_val_predict` when building curves. 7. Shipping without a dummy baseline in the model card.
Closing expansion
Chapter 3 is the metrics chapter disguised as an MNIST chapter. Digits are disposable; the precision/recall muscle is not. When Chapter 4 derives logistic regression and softmax, you will finally see where `decision_function` scores come from. Keep your PR-curve code handy; you will reuse it on every imbalanced corporate dataset you touch.
Worked scorecard: SGD versus forest on 5s
Build one markdown table in your notebook notes with columns: model, CV accuracy, precision, recall, F1, PR AUC, ROC AUC, wall time. Fill SGD and RandomForest using the same `cross_val_predict` discipline. Expect forest to win on F1/PR in the book's setting; still keep SGD as a fast baseline and online-learning option. Add a DummyClassifier row so the table teaches imbalance at a glance. When you later swap in a neural net from Part II, append a row rather than deleting the classical baselines; regression tests need them.
Precision floors in policy language
Translate business sentences into thresholds. "We can manually review 500 alerts/day" plus expected traffic → a recall target or a score cutoff. "False approve rate must stay under 2%" → a precision floor on the approve class. Write the sentence in the model card, then derive the threshold from CV curves. If no threshold meets the floor, you need better features or more reviewers, not a braver cutoff on the test set.
StratifiedKFold code literacy
```python from sklearn.model_selection import StratifiedKFold from sklearn.base import clone skf = StratifiedKFold(n_splits=3, shuffle=True, random_state=42) for train_idx, val_idx in skf.split(X_train, y_train_5): clf = clone(sgd_clf) clf.fit(X_train[train_idx], y_train_5[train_idx]) # score on val_idx ```
clone is mandatory; refitting the same object leaks state across folds. `shuffle=True` matters if the dataset is ordered; MNIST's conventional order is already shuffled, but your corporate extract may be sorted by date or label.
Softmax preview
Multiclass logistic regression and Part II networks emit softmax probabilities across digits. OvR scores are not probabilities unless calibrated. When a stakeholder asks "probability it is an 8", know whether you are quoting a calibrated probability, a normalised score, or a forest vote fraction. Chapter 4's softmax section makes the maths explicit; Chapter 3 only needs you to avoid overclaiming.
Data augmentation detail for the exercise
Shifting digits by one pixel in each direction (and optionally combinations) multiplies training size and teaches invariance. Implementation sketch: pad, crop, concat. Watch memory: 60k × 5 shifts is large. Evaluate whether CV accuracy gains survive on the true test set once; if gains vanish, you overfit the augmentation pipeline.
Spam exercise architecture
1. Acquire a public ham/spam corpus with licenses you can respect. 2. Stratified split. 3. Pipeline: `TfidfVectorizer` → `SGDClassifier(loss="log_loss")` or multinomial NB. 4. Metrics: precision/recall/F1 + PR curve; ignore accuracy. 5. Error analysis: list false positives that are newsletters users want; adjust features or thresholds. 6. Persist pipeline with joblib; serve a `predict_proba` threshold.
This exercise is the bridge from MNIST toys to text operations teams actually run.
Metric formulas quick sheet
- Accuracy = (TP+TN) / (TP+TN+FP+FN)
- Precision = TP / (TP+FP)
- Recall = TP / (TP+FN)
- FPR = FP / (FP+TN)
- F1 = 2PR/(P+R)
- TPR = Recall
Memorise these; interviews and design reviews use them without warning.
A critique of Chapter 3's scope
The chapter is excellent on binary metrics and thin on calibration, cost-sensitive learning and prevalence shift. When the positive rate in production differs from MNIST's 9%, the same score threshold yields different precision. Add prevalence-aware evaluation (or re-calibration) before you cite classroom PR numbers in a live fraud deck.
Test yourself questions (Chapter 3)
1. Why can a 95% accurate 5-detector be nearly useless? 2. Write precision and recall from TP, FP, FN. 3. When do you prefer PR curves to ROC curves? 4. How do you choose a decision threshold without burning the test set? 5. OvR versus OvO: which fits a kernel SVM on large N, and why? 6. What does a DummyClassifier row prove in a model review? 7. How do you read a confusion cell normalised by true class versus by prediction? 8. What changes when labels are multilabel instead of multiclass?
If you can answer without notes, you are ready for Chapter 4's maths. If not, rerun the notebook and narrate each plot aloud.
Extended numeric example
A batch of 1,000 transactions includes 20 frauds. A model flags 40 alerts: 16 true frauds and 24 false alarms. TP=16, FP=24, FN=4, TN=956. Precision=16/40=0.40. Recall=16/20=0.80. Accuracy=972/1000=0.972. F1≈0.53. The accuracy number would get a green dashboard; the precision number would get a budget conversation about analyst headcount. This is Chapter 3 in the wild.
Linking `cross_val_predict` to honest curves
`cross_val_predict` concatenates out-of-fold predictions so each training instance receives a score from a model that did not train on it. That is why PR/ROC built from those scores are not as wildly optimistic as training scores. It is still not a locked test evaluation. Final numbers on the 10,000 MNIST test images wait until the model and threshold are frozen.
What "hello world" hides
MNIST digits are centred and size-normalised. Real camera digits are not. The chapter's metrics transfer; the accuracy numbers do not. When product managers quote MNIST to demand 99% on receipts, show them Figure 3-11's ambiguous 3/5 pairs and recalibrate expectations.
System types recap table
| System | Output | Example in chapter | Primary metrics | |
|---|---|---|---|---|
| Binary | one bit | 5 vs not-5 | PR, F1, ROC | Multiclass |
| one of K | digit 0-9 | confusion, per-class F1 | Multilabel | bit vector |
| large & odd | macro/weighted F1 | Multioutput | structured | denoise pixels |
Keep this table in your notes; Part II adds sequences and generations but still collapses to these evaluation shapes when you force a decision.
Closing scoreboard discipline
Ship a classifier only with: dummy baseline, CV PR curve, chosen threshold rationale, slice metrics, and a locked test number you look at once. Everything else is commentary.
One more bank vignette
A payments team ships a "fraud accuracy 99.2%" slide. You ask for the confusion matrix and the PR curve at the current threshold. Precision is 0.11; analysts drown; customers get declines. You replace the slide with Chapter 3's scoreboard and a DummyClassifier row. Leadership changes the KPI to precision at 70% recall. That is the entire chapter's ROI outside MNIST.
Ch3 deep panel 100
This stretch centers on Ch3 ideas. Source cue: common supervised learning tasks are regression predicting values and classification predicting classes Chapter explored regression task, predicting housing values, using.
If the book shows code here, retype it from memory and diff.
This stretch centers on NumPy, DataFrame, DataFrames. Source cue: dataset from OpenML.org: from sklearn.datasets import fetch openml mnist fetch openml 'mnist 784', frame False The sklearn.datasets package contains mostly.
If the book shows code here, retype it from memory and diff.
This stretch centers on DataFrames, NumPy. Source cue: ideal for that, preferable set frame False get the data NumPy arrays instead. Let look these arrays:.
If the book shows code here, retype it from memory and diff.
Ch3 deep panel 101
This stretch centers on Ch3 ideas. Source cue: white and 255 black: import matplotlib.pyplot plt def plot digit image data image image data.reshape 28, plt.imshow image, cmap binary.
If the book shows code here, retype it from memory and diff.
Closing
Classification metrics and thresholds are policy. Accuracy alone is an interview trap.
4. Training Models
Chapter 4 opens the black box. Linear regression gets a closed-form normal equation and an SVD / pseudoinverse path; batch, stochastic and mini-batch gradient descent optimise the same MSE when closed form will not scale. Polynomial features, learning curves and the bias/variance split diagnose under- and overfit. Ridge, lasso, elastic net and early stopping regularise. Logistic and softmax regression turn the same linear machinery into classifiers with cross-entropy.
Linear regression
Study loop.
Model. ŷ = θ₀ + θ₁x₁ + … + θₙxₙ, or vector form ŷ = h_θ(x) = θ·x with x₀=1. Life-satisfaction teaching line from Chapter 1 is the one-feature case.
Linear regression predict and train.
Train by MSE. For m instances, MSE(θ) = (1/m) Σ (θ·x with x0 = 1. For m instances, MSE(θ) = (1/m) Σ (θ·x⁽ⁱ⁾ − y⁽ⁱ⁾)². Minimising MSE (or RMSE) yields the same θ̂ for linear models; MSE is smoother to optimise.
Normal equation
θ̂ = (X⊺X)⁻¹ X⊺ y when X⊺X is invertible. Implement with NumPy on a synthetic line (Figure 4-1, 4-2). Complexity: inverting n×n is roughly O(n³); fine for small n, painful when features explode (for example, big polynomial expansions).
Closed-form options.
SVD / pseudoinverse path
θ̂ = X⁺ y with Moore-Penrose pseudoinverse via SVD. Scikit-Learn's `LinearRegression` uses this approach: handles singular X⊺X, complexity ~O(n²) for large m relative to n in typical regimes. Prefer it over hand-coded normal equations.
Gradient descent
Idea. Start with θ, subtract η ∇_θ MSE repeatedly. Learning rate η too small → crawl (Figure 4-4); too large → diverge (Figure 4-5). Non-convex costs add local minima and plateaus (Figure 4-6); MSE for linear regression is convex and smooth, so GD reaches the global minimum with a sane η.
Gradient descent loop.
Feature scaling matters. Elongated bowls make GD zig-zag; StandardScaler makes contours rounder and convergence faster. Always scale before GD-based linear models.
Batch gradient descent
Gradient uses the whole training set. Partial derivative for θⱼ is (2/m) Σ (θ·x⁽ⁱ⁾ − y⁽ⁱ⁾) xⱼ⁽ⁱ⁾. Vector form: ∇ MSE = (2/m) X⊺ (Xθ − y). Update: θ ← θ − η ∇MSE. Stable but slow per step when m is huge.
Batch versus stochastic GD.
Convergence. For convex MSE, error falls roughly as O(1/iterations) under standard schedules. Practically: plot learning curves of train/val error versus iterations; stop when gains flatten.
Stochastic gradient descent
Each step uses one random instance. Path is noisy (Figure 4-10) but very fast per update and can escape shallow traps on non-convex costs. Use a learning schedule that decays η over time so the noise settles. `SGDRegressor` implements this. Randomness means results wander; pin seeds and average or anneal carefully.
Mini-batch gradient descent
Each step uses a small batch. Compromise between batch's stability and SGD's speed; GPU-friendly. Figure 4-11 compares paths in parameter space. Table 4-1 in the book summarises: normal equation / SVD vs batch / SGD / mini-batch on m, n, support for out-of-core, and sklearn class names.
| Algorithm | m large | n large | Out-of-core | sklearn | |
|---|---|---|---|---|---|
| Normal eq | slow | bad (n³) | no | - | SVD / LinearRegression |
| OK | OK-ish | no | LinearRegression | Batch GD | slow/step |
| OK | no | - | SGD | good | OK |
| yes | SGDRegressor | Mini-batch | good | OK | yes |
Polynomial regression
Ops contrast.
Add powers of features (`PolynomialFeatures`) then run linear regression: still linear in θ, nonlinear in x. A quadratic can fit a curve (Figures 4-12, 4-13). High degree → wild oscillation (Figure 4-14) and overfit.
Polynomial regression pipe.
Warning: `PolynomialFeatures` with degree d on n features creates O(nᵈ) columns; degree 3 on 100 features is already brutal. Combine with ridge/lasso or cut degree.
Learning curves and bias/variance
Practice hub.
Learning curves plot train and val error versus training set size (Figures 4-15, 4-16). Underfit: both errors high and close. Overfit: train error low, val error high, gap large. More data helps overfit more than underfit; simpler models or regularisation help overfit; richer features or models help underfit.
Reading learning curves.
Bias / variance / irreducible error. Bias: wrong assumptions (straight line for a curve). Variance: sensitivity to training sample. Irreducible: noise in labels. Total error ≈ bias² + variance + irreducible (Figure 4-17). Regularisation trades a little bias for less variance.
Regularised linear models
Fit discipline.
Ridge (ℓ2)
J(θ) = MSE(θ) + α Σᵢ₌₁ⁿ θᵢ² (usually leave θ₀ unregularised). α=0 → linear regression; large α → coefficients shrink toward zero. Closed form: θ̂ = (X⊺X + αA)⁻¹ X⊺ y with A = diag(0,1,…,1). `Ridge` in sklearn. Scale features before ridge; α is scale-sensitive.
Regulariser choice.
Lasso (ℓ1)
J(θ) = MSE(θ) + α Σ |θᵢ|. Tends to zero out weights → sparse models / embedded feature selection (Figure 4-20). Non-differentiable at 0; solvers use coordinate descent / subgradients. `Lasso` class. Prefer lasso when you believe few features matter; prefer ridge when many small contributors are real.
Elastic net
Mix ℓ1 and ℓ2 with a ratio parameter (l1_ratio). Default choice when you want sparsity with more stability than pure lasso among correlated features. `ElasticNet`.
Early stopping
Stop iterative training when validation error bottoms (Figure 4-21). Simple, effective regulariser for GD-based models and neural nets. Implement by tracking best_val and best_model; restore on patience breach. Tip: warm up enough steps so early noise does not stop you too soon.
Early stopping loop.
- Ridge when all features likely help a little.
- Lasso when you want automatic selection.
- Elastic net when features are correlated and you still want sparsity.
- Early stopping whenever you iterate (GD, SGD, boosting, nets).
Logistic regression
Error parts.
Estimate probabilities with the logistic (sigmoid) σ(s) = 1/(1+e^{-s}) (sigmoid). Estimate probabilities with the logistic σ(s) = 1/(1+e^{-s}) (Equation 4-15, Figure 4-22). Model: p̂ = σ(θ·x). Predict 1 if p̂ ≥ 0.5 (score ≥ 0).
Logistic regression path.
Cost. Log loss: −log(p̂) if y=1, −log(1−p̂) if y=0. Convex; GD works. Partial derivatives resemble linear regression's but with (p̂ − y) as the error term (Equation 4-19).
Iris decision boundary. Classic demo (Figures 4-23 to 4-25): one feature (petal width) separates virginica; two features draw a linear boundary in feature space. `LogisticRegression` in sklearn; regularisation strength is `C` (inverse of α - larger C means less regularisation).
Softmax regression
Train vs serve.
Multiclass. Score sₖ(x) = θₖ·x per class; probabilities p̂ₖ = exp(sₖ)/Σⱼ exp(sⱼ) (Equations 4-20, 4-21). Predict argmaxₖ p̂ₖ. Cross-entropy cost (Equation 4-23) generalises log loss; gradient for class k is (1/m) Σ (p̂ₖ⁽ⁱ⁾ − yₖ⁽ⁱ⁾) x⁽ⁱ⁾.
Softmax regression hub.
Iris 3-class boundaries (Figure 4-26). Softmax is the classification head you will see again in neural nets (Part II). Tip: when classes are exclusive, softmax; when labels are multilabel, use independent sigmoids instead.
LinearRegression, Ridge, Lasso, ElasticNet, SGDRegressor, SGDClassifier, LogisticRegression (with multi_class / multinomial softmax paths) and PolynomialFeatures. Watch the meaning of C versus alpha. For very wide data, prefer SGD* or HistGradientBoosting* (Chapter 6) over dense closed forms.Linear chapter to bank models.
Exercise themes
Model promotion.
Explain when to prefer ridge vs lasso vs elastic net. Implement GD variants and compare paths. Plot learning curves for poly degrees. Train softmax on Iris; visualise boundaries. Derive gradients once by hand - the exercise is the point.
Softmax temperature note
Study loop.
Dividing logits by temperature T>1 flattens probabilities; T<1 sharpens. Training usually uses T=1; distillation later uses elevated T. Chapter 4 multiclass section assumes T=1.
Gradient descent cheatsheet
Batch: accurate gradient, expensive. SGD: noisy, cheap, online. Mini-batch: middle path, GPU native. Learning rate schedules beat hero constant LRs on long runs.
Normal equation and SVD path
Closed form θ̂ = (X⊺X)⁻¹ X⊺y when X⊺X is invertible. Prefer the pseudoinverse path θ̂ = X⁺ y via SVD: more stable when features are correlated. `LinearRegression` in sklearn uses the SVD route. Complexity grows steeply with feature count n; gradient methods win when n is large or data is streamed.
Gradient descent variants in production
Batch GD uses the full sample every step: stable, slow. SGD picks one row: noisy, online-capable. Mini-batch (32-256) is the GPU default. Learning rate too high diverges; too low crawls. Feature scaling is mandatory for GD on mixed units; the normal equation cares less about scale for the fit but interpretation still does.
Polynomial features and the bias-variance cartoon
`PolynomialFeatures(degree=d)` expands interactions. Degree 1 underfits a quadratic; degree 300 memorises noise. Learning curves: plot train and valid error versus training size. Gap large → variance; both high → bias. Regularise or simplify when variance dominates; add features or capacity when bias dominates.
Ridge, lasso, elastic net
Ridge (L2) shrinks coefficients, keeps all features. Lasso (L1) zeros some coefficients for sparse models. Elastic net mixes both; use when features are correlated and you still want selection. Scale first. `RidgeCV` / `LassoCV` pick α by CV. Early stopping on iterative solvers is another regulariser.
Logistic and softmax
Sigmoid σ(z) = 1/(1+e⁻z); train with log loss. Softmax generalises to K classes; cross-entropy pairs with it. Decision boundaries are linear in feature space unless you expand features. Thresholds are business choices, not defaults of 0.5.
Equation checklist
Life satisfaction line; vector linear predict; MSE; normal equation; gradients; GD step; ridge/lasso costs; sigmoid; logistic derivatives; softmax; cross-entropy and gradient. Re-derive each once on paper.
Bank and EA takes
Credit scorecards still ship as linear/logistic with monotone constraints; GD and regularisation are how you fit them at scale. EA: logistic on incident features for severity; ridge when collinear metrics fight.
Pitfalls
1. GD without scaling. 2. Huge poly degree without regularisation. 3. Softmax + MSE. 4. Reading L1 zeros as causal. 5. Tuning threshold on the test set. 6. Ignoring class imbalance in log loss.
Test yourself
Normal equation vs SVD? When SGD over batch? Bias vs variance on a learning curve? Ridge vs lasso? Softmax loss?
Linear regression deep dive
Hypothesis: ŷ = θ₀ + θ₁x₁ + … + θₙxₙ = θ⊺x when x₀=1. MSE cost J(θ) = (1/m) Σ (θ⊺x⁽ⁱ⁾ - y⁽ⁱ⁾)². Normal equation solves ∇J=0 in one shot when feasible. SVD/pseudoinverse handles singular X⊺X. `np.linalg.lstsq` and sklearn `LinearRegression` are the daily drivers.
Computational note: normal equation is O(n³) in features for the inversion-dominated view; SVD path similar order. Gradient descent is O(m n) per epoch. Cross over when n is thousands to tens of thousands depending on m and hardware.
Gradient descent geometry
Gradient ∇J points uphill; step opposite with learning rate η. Contours of MSE for linear regression are convex bowls, so GD finds the global minimum if η is sane. Feature scaling rounds the bowl so one η works across coordinates. Without scaling, you zigzag.
Batch GD: one update per full pass. Plot J versus epochs; it should descend smoothly. SGD: noisy path that can escape shallow traps in nonconvex problems (less relevant for plain linear MSE). Mini-batch: vectorised sweet spot. Learning schedules (time-based, plateau-based) beat fixed η on long runs.
Polynomial regression practice
`Pipeline([("poly", PolynomialFeatures(degree)), ("lin", LinearRegression())])`. For degree d and n features, the expanded width is C(n+d, d). Explodes fast - regularise or switch to splines/GAMs when interactions matter but width hurts. Always evaluate on a valid set; training MSE alone lies.
Learning curves protocol
1. Shuffle and take prefixes of training data of growing size. 2. Fit. 3. Score train prefix and full valid set. 4. Plot. High bias: both curves meet high. High variance: train low, valid high, gap persists as m grows. Fix bias with capacity/features; fix variance with data, regularisation, or simpler models.
Regularisation details
Ridge: J = MSE + α Σ θⱼ² (usually skip θ₀). Closed form (X⊺X + αI)⁻¹X⊺y. Lasso: α Σ |θⱼ|; soft-thresholding geometry; sparse solutions. Elastic net: r L1 + (1-r) L2 mix. Plot coefficient paths versus α. Early stopping: stop iterative GD when valid error rises - equivalent to a form of regularisation.
Logistic regression details
p̂ = σ(θ⊺x). Loss = -y log p̂ - (1-y) log(1-p̂). Gradient = (1/m) X⊺(p̂ - y). No closed form; use iterative solvers. Regularise with L2 by default in sklearn. Decision threshold is not 0.5 when class costs differ - tune on PR or cost curves (Chapter 3).
Softmax regression details
Logits z_k = θₖ⊺x; p̂_k = exp(z_k)/Σ exp(z_j). Cross-entropy loss. Gradient w.r.t. logits is p̂ - y_onehot. Equivalent to multinomial logistic regression. Linear boundaries in feature space; add interactions or use nonlinear models when boundaries curve.
Hyperparameter field card
| Model | Key knobs | |
|---|---|---|
| Linear/SGD | eta0, alpha, penalty, max_iter | Ridge/Lasso |
| alpha, l1_ratio (EN) | Logistic | C (=1/α), penalty, solver, class_weight |
A closer look
Linear models underfit complex tabular interactions that trees capture. They win on calibrate-able scores, sparse interpretability, and small-data regimes. Do not force a 200-feature poly expansion when HGB would finish in seconds with better accuracy.
Bank vignette
A scorecard team fits logistic with L1 to keep ten strong variables for adverse action. They compare to a ridge model on all features as a challenger. Thresholds set by capital and approval-rate targets, not by accuracy.
EA vignette
Ridge on standardised SLIs predicts ticket volume. Coefficients are read in standup. When nonlinear seasonality appears, they add calendar features before reaching for a net.
Exercise themes expanded
Implement GD three ways; compare paths on a scaled and unscaled design. Derive softmax gradients. Plot ridge paths. Build learning curves for poly degrees 1, 2, 20. Train Iris softmax; draw boundaries.
Linear regression revisited
Life-satisfaction style example: one feature, fit a line, read θ₀ and θ₁. Vector form ŷ = θ⊺x with x₀=1. Cost MSE(θ) = (1/m) Σ (θ⊺x⁽ⁱ⁾ − y⁽ⁱ⁾)². Convex in θ for linear models - one global basin.
Normal equation: θ̂ = (X⊺X)⁻¹ X⊺y when invertible. Pseudoinverse / SVD: θ̂ = X⁺ y always defined; sklearn `LinearRegression` uses this path. Computational complexity: inversion-dominated view ~O(n³) in features; prefer GD when n is large or data is streamed.
Gradient descent family
Batch GD: ∇MSE = (2/m) X⊺(Xθ − y); step θ ← θ − η ∇. Contours are ellipsoids; unscaled features make skinny valleys. Convergence rate for strong convexity improves with better conditioning - hence scaling.
SGD: one random row per step; noisy path; supports out-of-core. Mini-batch: vectorised compromise (32-256 common). Table in the book contrasts out-of-core support, scaling needs, and speed versus the normal equation.
Learning rate too high diverges (cost explodes); too low crawls. Plot training curves. Schedules beat fixed η on long runs (more in Chapter 11).
Polynomial regression and learning curves
`PolynomialFeatures(degree=d)` expands interactions; width is C(n+d,d). Degree 1 underfits a quadratic; huge degree memorises Gaussian noise. Always pipeline poly + model so CV does not leak.
Learning curves: train and valid error versus training size. Bias/variance trade-off: high bias → both errors high and close; high variance → train low, valid high. Irreducible error is noise you cannot beat. Fix bias with capacity/features; fix variance with data, regularisation, or simpler models.
Regularised linear models in full
Ridge: MSE + α Σ θⱼ²; closed form (X⊺X + αI)⁻¹X⊺y; shrinks but keeps features. Lasso: α Σ |θⱼ|; sparse solutions; soft-thresholding geometry. Elastic net: r L1 + (1−r) L2. Scale features first. `RidgeCV` / `LassoCV` pick α.
Early stopping on iterative solvers is another regulariser: stop when valid error rises. Equivalent in spirit to limiting effective capacity along the optimisation path.
Logistic and softmax in full
Logistic: p̂ = σ(θ⊺x) with σ(z)=1/(1+e⁻z). Log loss −y log p̂ − (1−y) log(1−p̂). Gradient (1/m) X⊺(p̂ − y). Decision boundary is linear in feature space; threshold is a business knob (Chapter 3).
Softmax: logits z_k = θₖ⊺x; p̂_k = exp(z_k)/Σⱼ exp(zⱼ). Cross-entropy loss; gradient w.r.t. logits is p̂ − y_onehot. Iris-style 2D plots show linear regions per class.
Equation and figure checklist
Figures 4-1… cover the life-satisfaction line, GD paths, learning-rate effects, batch vs SGD vs mini-batch trajectories, poly under/overfit, learning curves, ridge/lasso paths, logistic curves, and softmax boundaries. Re-sketch each once.
Equations to re-derive: linear predict; MSE; normal equation; gradients; GD step; ridge/lasso costs; sigmoid; logistic gradient; softmax; cross-entropy.
Bank and EA applications
Bank: scorecards as logistic with monotone constraints; ridge challengers; thresholds from capital and approval-rate targets. EA: ridge on standardised SLIs for ticket volume; add calendar features before reaching for a net when seasonality appears.
Exercise themes
Implement batch, SGD, and mini-batch GD; compare paths on scaled vs unscaled data. Derive softmax gradients on paper. Plot ridge coefficient paths. Learning curves for poly degrees 1, 2, 20. Softmax on Iris with boundary plots.
Computational complexity table in prose
The book contrasts normal equation, SVD, batch GD, stochastic GD, and mini-batch GD on axes that interviewers love: large m, large n, out-of-core support, need for feature scaling, and online learning. Normal equation and SVD dislike huge feature counts. Batch GD wants the whole set in memory. SGD and mini-batch stream. Scaling is mandatory for the GD family and optional for closed forms only in a narrow numerical sense.
Memorise the qualitative table, then prove it to yourself with a timing script on synthetic data as n grows. That script is worth more than rereading the paragraph.
GD path geometry
Figures in Chapter 4 show batch paths as smooth lines down a bowl, SGD as noisy zigzags, and mini-batch as a compromise. Learning rate plots show divergence, slow crawl, and a healthy descent. When features are unscaled, the bowl becomes a ravine and a single learning rate fights the axes. StandardScaler rounds the bowl.
Momentum foreshadowed here and formalised in Chapter 11 averages gradients over time to damp oscillation along steep walls. For linear MSE you do not need it to find the minimum, but the pictures transfer to nonconvex deep nets.
Regularisation paths
Ridge paths shrink coefficients smoothly toward zero without hitting exact zero. Lasso paths hit zero and stay there, producing sparse models. Elastic net paths sit between. Plotting paths versus alpha teaches more than any paragraph about what the penalty buys. Cross-validation picks alpha; paths explain the pick to stakeholders.
Early stopping curves show validation error falling then rising while training error keeps falling. The vertical line at the best validation epoch is your regulariser. Save that checkpoint, not the final epoch, unless you intentionally retrain on train plus valid afterward.
Logistic geometry and softmax regions
Sigmoid saturates at both ends, which historically caused vanishing gradients in deep nets and motivates better activations later. For logistic regression itself, saturation means confident probabilities and small gradients for easy points, which is fine. Decision boundaries are linear; probability contours bend through the sigmoid.
Softmax regions on Iris petals show three linear wedges. Misclassifications hug the boundaries. Adding petal area as a feature can tilt boundaries without leaving the linear family. When wedges will not cut the data, leave the linear family for trees or nets.
Bank scorecard workflow
Start with logistic regression on hand-binned characteristics. Apply monotone constraints where regulation or policy demands. Use L1 to keep the reason code list short. Challenge with ridge on a richer feature set. Calibrate. Set thresholds from capital and approval-rate targets. Monitor population stability and coefficient drift monthly.
Do not replace this stack with an uncalibrated deep net because a Kaggle notebook said so. Deep models can challenge, but the champion must meet explainability and stability bars.
Equation drill sheet
Re-derive on paper: linear prediction, MSE, normal equation, gradient of MSE, GD step, ridge closed form, lasso soft thresholding intuition, sigmoid, logistic loss and gradient, softmax, cross-entropy gradient. If you cannot re-derive, you do not own the chapter yet.
Ch4 deep panel 100
This stretch centers on Ch4 ideas. Source cue: machine learning models and their training algorithms mostly like black boxes. you went through some the exercises the previous chapters,.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch4 ideas. Source cue: linear regression model, one the simplest models there is. will discuss two very different ways train it: Using closed-form equation.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch4 ideas. Source cue: curves, and then will look several regularization techniques that can reduce the risk overfitting the training set. Finally, will examine.
If the book shows code here, retype it from memory and diff.
Ch4 deep panel 101
This stretch centers on Ch4 ideas. Source cue: you make sense the equations. Linear Regression Chapter looked simple linear model life satisfaction Equation 4-1 Equation 4-1. simple linear.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch4 ideas. Source cue: using vectorized form, shown Equation 4-3. Equation 4-3. Linear regression model prediction vectorized form this equation: the hypothesis function, using.
If the book shows code here, retype it from memory and diff.
This stretch centers on RMSE, MSE. Source cue: this book will use this notation avoid switching between dot products and matrix multiplications. OK, that the linear regression model.
If the book shows code here, retype it from memory and diff.
Ch4 deep panel 102
This stretch centers on MSE. Source cue: the function easier optimize and/or because has extra terms needed during training only e.g., for regularization good performance metric close.
If the book shows code here, retype it from memory and diff.
This stretch centers on MSE, NumPy. Source cue: vector simplify notations, will just write MSE instead MSE The Normal Equation find the value that minimizes the MSE, there.
If the book shows code here, retype it from memory and diff.
This stretch centers on NumPy, TensorFlow, PyTorch. Source cue: np.linalg compute the inverse matrix, and the operator for matrix multiplication: from sklearn.preprocessing import add dummy feature add dummy feature.
If the book shows code here, retype it from memory and diff.
Ch4 deep panel 103
This stretch centers on LinearRegression. Source cue: .
If the book shows code here, retype it from memory and diff.
This stretch centers on LinearRegression. Source cue: .
If the book shows code here, retype it from memory and diff.
This stretch centers on LinearRegression. Source cue: about n2.4 depending the implementation. other words, you double the number features, you multiply the computation time roughly 22.4 5.3.
If the book shows code here, retype it from memory and diff.
Ch4 deep panel 104
This stretch centers on Ch4 ideas. Source cue: many features will take roughly twice much time. Now will look very different way train linear regression model, which better.
If the book shows code here, retype it from memory and diff.
This stretch centers on MSE. Source cue: you have reached minimum practice, you start filling with random values this called random initialization Then you improve gradually, taking.
If the book shows code here, retype it from memory and diff.
This stretch centers on MSE. Source cue: might jump across the valley and end the other side, possibly even higher than you were before. This might make.
If the book shows code here, retype it from memory and diff.
Ch4 deep panel 105
This stretch centers on Ch4 ideas. Source cue: pick any two points the curve, the line segment joining them never below the curve. This implies that there are.
If the book shows code here, retype it from memory and diff.
This stretch centers on StandardScaler. Source cue: direction the global minimum, and ends with long march down almost flat valley. will eventually reach the minimum, but will.
If the book shows code here, retype it from memory and diff.
This stretch centers on MSE. Source cue: Most models have more than one model parameter. Therefore, implement gradient descent, you need compute the gradient the cost function.
If the book shows code here, retype it from memory and diff.
Ch4 deep panel 106
This stretch centers on MSE. Source cue: vector, denoted MSE contains all the partial derivatives the cost function one for each model parameter Equation 4-7. Gradient vector.
If the book shows code here, retype it from memory and diff.
This stretch centers on MSE. Source cue: the normal equation SVD decomposition. Once you have the gradient vector, which points uphill, just the opposite direction downhill. This.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch4 ideas. Source cue: eta Figure 4-8 shows the first steps gradient descent using three different learning rates. The line the bottom each plot.
If the book shows code here, retype it from memory and diff.
Ch4 deep panel 107
This stretch centers on MSE. Source cue: epochs. too low, you will still far away from the optimal solution when the algorithm stops; but too high, you.
If the book shows code here, retype it from memory and diff.
Closing
Training algorithms are the engine room under every later chapter. Trees come next.
5. Decision Trees
Decision trees split feature space into rectangles for classification and regression.
Training and visualising
Study loop.
Fit `DecisionTreeClassifier(max_depth=2)` on Iris petal length/width. Export with `export_graphviz` to a `.dot` file; render with Graphviz (`Source.from_file`). Figure 5-1 shows nodes with samples, value (class counts), gini, and predicted class. `tree_clf.tree_` exposes the structure programmatically.
Train and visualise a shallow tree.
Making predictions
Start at the root: petal length ≤ 2.45? Yes → setosa leaf. No → ask petal width ≤ 1.75? Yes → versicolor; no → virginica. No scaling required - a major practical advantage over Chapter 4's GD models.
Prediction walk on Figure 5-1.
Gini impurity of node i: G_i = 1 − Σ_k p_{i,k}². Pure node → G=0. Scikit-Learn's CART builds binary trees; ID3-style algorithms can have multiway splits.
Gini impurity computation.
Figure 5-2 draws the orthogonal boundaries: vertical line at length 2.45, horizontal at width 1.75. Deeper `max_depth` adds more cuts.
White box versus black box
Ops contrast.
Trees are white box: humans can read the rules. Forests and nets are often black box even when you can dump the arithmetic. Interpretable ML matters in healthcare, finance, courts and HR. Use trees when a rule list is the deliverable; use forests when predictive power wins and you explain with importances/SHAP later.
Class probabilities
Practice hub.
Traverse to a leaf; return class proportions among training samples in that leaf. Example [5, 1.5] → ≈ [0, 0.907, 0.093]. Note: probabilities are piecewise constant inside each rectangle - a flower deep in virginica territory but inside a versicolor-majority leaf still gets versicolor probabilities.
CART training algorithm
Fit discipline.
Search feature k and threshold t_k minimising J = (m_left/m) G_left + (m_right/m) G_right (Equation 5-2). Recurse until `max_depth`, or no impurity-reducing split, or other stops (`min_samples_split`, `min_samples_leaf`, `max_leaf_nodes`, …).
CART split search.
Greedy, not globally optimal. Finding the true global tree is NP-hard; CART settles for good local splits. Complexity roughly O(n × m log m) with sklearn's implementation under typical settings.
Entropy option. `criterion="entropy"` uses information gain; Gini is default and usually similar. Raschka-style analyses find small differences; pick one and CV if obsessed.
Regularisation hyperparameters
Error parts.
| Hyperparameter | Effect when tightened | |
|---|---|---|
| max_depth | Shallower trees | min_samples_split |
| Need more samples to split | min_samples_leaf | Larger leaves |
| min_weight_fraction_leaf | Leaf mass floor | max_leaf_nodes |
| Cap leaves | max_features | Random feature subset per split |
Main tree regularisers.
On `make_moons`, an unregularised tree overfits; `min_samples_leaf=5` (book demo) generalises better (≈0.90 → ≈0.92 on a noisy test in the text's seeds). Decreasing max_depth is the first answer when overfitting; scaling does nothing useful for trees when underfitting.
Regression trees
Train vs serve.
`DecisionTreeRegressor` predicts the mean target in each leaf. CART minimises weighted MSE of children (Equation 5-4). Unregularised regression trees staircase through noise (Figure 5-6 left); `min_samples_leaf=10` smooths (right).
Classification vs regression trees.
Limitations
Model promotion.
Axis orientation. Splits are orthogonal to axes. A 45° rotation can force a staircase (Figure 5-7). Mitigation preview: scale + PCA rotate (Chapter 7), then tree on principal components (Figure 5-8) - sometimes one rotated feature suffices.
Rotation limitation path.
Missing values. Modern sklearn trees handle missing values natively; no imputer required for that reason alone (still think about missingness as signal).
High variance. Small data or seed changes → different trees (Figure 5-9). Fix: average many trees → random forests (Chapter 6).
Why Chapter 6 exists.
DecisionTreeClassifier/Regressor in scikit-learn 1.9.1 still expose Gini/entropy/MSE/Friedman MSE criteria, the regularisation knobs above, and native missing-value support. export_graphviz remains; also try sklearn.tree.plot_tree for quick Matplotlib renders without Graphviz installs.Exercise themes
Gini of a child is generally ≤ parent (weighted), not always each child. Overfit → reduce max_depth. Underfit → scaling will not save you. CART cost ~ O(n m log m): 10× rows ≫ 10× time; 2× features ≈ 2× time. Moons grid search on max_leaf_nodes → ~85-87% test. Manual forest via 1000 bootstrapped trees + mode vote beats one tree - the Chapter 6 preview.
White-box reporting checklist
1. Print `plot_tree` or Graphviz for the shipped shallow model. 2. List the exact predicates on the path for a denied application. 3. Report leaf sample counts so nobody trusts a leaf of size 2. 4. Compare to a forest's accuracy to price the interpretability tax.
Gini versus entropy worked values
For a binary leaf with proportions p and 1−p: Gini = 2p(1−p); entropy = −p log2 p − (1−p)log2(1−p). Both peak at p=0.5 and are zero at p∈{0,1}. Gini is slightly cheaper to compute; entropy is slightly sharper near 0/1 in some charts. In practice CV rarely cares.
CART greediness example
A first split that looks mediocre can unlock two great child splits; CART will not see that. Extremely random trees and some look-ahead variants exist in the literature; sklearn stays greedy. Ensembles paper over many greedy mistakes.
Regularisation recipe for moons-like data
Start with `max_depth=None`, `min_samples_leaf=1` to see overfit. Raise `min_samples_leaf` through {1,5,10,20} with CV. Add `max_leaf_nodes` grid.
Regression staircases
Because predictions are piecewise constant, regression trees produce staircases. That is fine for many targets; bad if you need smooth derivatives. For smooth curves, prefer Chapter 4 polys/splines or boosting with small trees.
PCA + tree pattern
```python pipe = make_pipeline(StandardScaler(), PCA(), DecisionTreeClassifier(max_depth=2)) ``` Use when orthogonal splits fight the geometry. PCA components may hurt interpretability; document the trade.
Variance demo
Retrain the same Iris tree with different `random_state` values; overlay boundaries. The spread is why one tree is a shaky champion on noisy tabular data.
Bank vignette
You CV within that constraint, publish the Graphviz, and keep a forest champion in the background for challenger debates. When the forest beats the tree by 2 Gini points, the committee still ships the tree for the regulated decision and uses the forest for account management scoring.
EA vignette
On-call runbooks generated from a tree on alert features: if severity high and service in {payments, login} and error_rate > τ → page. Shallow trees beat neural nets when the consumer is a tired human.
Iris walkthrough in numbers
Root: 150 samples, classes [50,50,50], Gini = 1 − 3×(1/3)² = 2/3. Left after length ≤ 2.45: 50 setosa, Gini 0. Right: 100 samples mixed. Second split width ≤ 1.75 yields ≈54 mostly versicolor (Gini ≈ 0.168) and ≈46 mostly virginica. Recreate these counts from `tree_.value` to trust the export.
Graphviz versus plot_tree
`export_graphviz` needs the Graphviz system package; CI images often lack it. Prefer `sklearn.tree.plot_tree(tree_clf, feature_names=..., class_names=..., filled=True)` for notebooks and PDFs. Keep `.dot` when legal teams want a vector diagram in the model file.
Why no scaling
Monotone transforms of a single feature do not change orderings of thresholds along that axis. StandardScaler still changes relative scales across features, which can change which feature CART picks when impurities are close - but trees do not require scaling the way GD does. Document when you scale anyway for PCA pipelines.
Information gain sketch
Entropy H = −Σ p log2 p. Gain = H(parent) − weighted H(children). CART can optimise Gini or entropy; both measure impurity. For multi-class with rare classes, try both under CV once.
Stopping rules interactions
`max_depth=3` and `max_leaf_nodes=8` can conflict; the tighter constraint wins in practice. Prefer one primary knob per search to keep grids honest. `min_impurity_decrease` stops splits that buy too little purity - useful on large noisy sets.
Complexity intuition
Sorting feature columns dominates. Roughly O(n m log m) for the usual sklearn path. Ten million rows is not ten times an hour if memory thrashes; profile before promising SLA. Feature count scales nearer to linear in n when m is large.
Moons experiment protocol
1. `make_moons(10000, noise=0.4, random_state=42)`. 2. Stratified 80/20 split. 3. Grid `max_leaf_nodes` in {2,4,8,16,32,64} with 3-fold CV. 4. Refit best on full train; report test accuracy (~85-87% in the book). 5. Save the Graphviz of the chosen tree for the write-up.
Manual forest preview (exercise 8)
Majority vote preview of forests.
Draw 1000 subsets of 100 rows (`ShuffleSplit`). Train one regularised tree each. Majority vote on the test set. Expect ~0.5-1.5 pp lift over the single tree - the core random-forest intuition before Chapter 6's API.
Regression MSE leaf maths
ŷ_node = mean of y in the node. MSE_node = mean (ŷ − y_i)². CART picks the split minimising size-weighted child MSE. Deep trees can drive training MSE near zero and memorise noise; leaf floors fight that.
Axis-aligned failure modes in banking
Debt-to-income and utilisation often interact diagonally. A tree needs many stairs; a linear model with an interaction term needs one. Prefer forests or GAMs when auditors accept less literal rule lists.
Seed sensitivity protocol
Fit twenty trees with seeds 0..19 on the same data; plot test accuracy histogram. Wide spread → do not ship a single tree; bag or boost.
Interpretable ML adjacent tools
`treeinterpreter`, SHAP TreeExplainer (especially for forests), and simple rule lists from `skope-rules`-style packages. Chapter 5 stays with native trees; know the ecosystem when white-box is a hard requirement.
Multioutput trees
`DecisionTreeRegressor` accepts multi-column y; leaves predict a vector mean. Useful for multi-horizon forecasts when you accept staircase geometry.
Sparse and high-cardinality categoricals
Trees handle integer-encoded categories without one-hot if you accept ordinal artefacts; better: native categorical support in HistGradientBoosting (Chapter 6 era) or careful target encoding with CV. Pure Chapter 5 trees plus one-hot explode on high cardinality.
Cost complexity pruning
Cost-complexity pruning path.
`ccp_alpha` prunes after a full grow. Path of subtrees vs alpha; pick by CV. Often cleaner than juggling five min_* knobs. Prefer ccp_alpha grids on large noisy problems.
Pitfalls
1. Trusting leaf probabilities far from leaf centres. 3. Scaling features because Chapter 4 said so and expecting miracles. 4. Ignoring rotation sensitivity on engineered ratios. 5. Shipping one tree without a variance check across seeds. 6. Forgetting `random_state` in published notebooks. 7. Using trees on raw text counts without sparse-aware pipelines.
Test yourself
What is Gini? Why is CART greedy? Name four regularisers. How do regression leaves predict? Why do forests exist? Does scaling fix underfit trees? What does ccp_alpha do?
Leaf probability flatness demo
Pick a leaf rectangle on Iris. Sample ten points uniformly inside it; `predict_proba` is identical for all ten. That is the geometry: trees do not interpolate smoothly inside a leaf. Downstream calibration (Platt/isotonic on OOF) can reshape scores without changing the partition.
Adverse-action path dump
```python node = 0 x = applicant_features while True: left = tree.children_left[node] if left == -1: break feat, thr = tree.feature[node], tree.threshold[node] node = left if x[feat] <= thr else tree.children_right[node] ``` Emit the predicates as English for the letter.
Worked Gini table
| Node | Counts | Gini | |
|---|---|---|---|
| Root | 50/50/50 | 0.667 | Left |
| 50/0/0 | 0.000 | Mid | 0/49/5 |
| 0.168 | Right | 0/1/45 | 0.043 |
Rebuild from `tree_.value` after every retrain. Numbers drifting after a sklearn upgrade are a regression test failure.
Comparison to linear models
Linear/logistic models need scaled features and struggle with interactions unless you engineer them. Trees swallow interactions as successive splits. Linear models extrapolate as lines; trees extrapolate as constant leaves - dangerous outside the training hull. Pick trees for messy tabular interactions; pick linear when the form must stay a scorecard.
Exercise answer sketches
1. Weighted child Gini ≤ parent; individual child can rise. 3. Yes - decrease max_depth when overfitting. 4. No - scaling rarely fixes underfit trees. 5. CART ~ O(n m log m): 10× rows ≫ 10× time (sorts dominate). 6. ~2× features ≈ ~2× time if m fixed. 7-8. Moons grid + manual forest as in the chapter lab.
Closing
Trees are the readable nonlinear baseline. Chapter 6 bags and boosts them into the strongest classical tabular weapons in the book.
6. Ensemble Learning and Random Forests
Wisdom of the crowd for predictors: vote, bag, boost, stack. Géron walks hard/soft voting, bagging vs pasting, random forests and extra-trees, impurity feature importance, AdaBoost, residual gradient boosting, fast HistGradientBoosting, and stacking blenders - with a when-to-use table that maps cleanly onto bank and ops work.
Why ensembles
Study loop.
Aggregate many predictors; often beat the best single model. Cost: compute, deploy complexity, interpretability. Competition winners (Netflix Prize lore) lean on ensembles near the end of a project after solid singles exist.
Voting classifiers
Train diverse models (logistic, SVM, forest, kNN). Hard voting = majority class. Soft voting = average predicted probabilities (usually better when models are calibrated-ish). Weak learners that err differently can form a strong learner via the law of large numbers (biased coin analogy, Figure 6-3).
Voting classifier flow.
```python VotingClassifier( estimators=[("lr", lr), ("rf", rf), ("svc", svc)], voting="soft", ) ``` Book moons demo: soft voting ≈ 92% test. Diversity matters more than cloning one strong model five times.
Bagging and pasting
Ops contrast.
Same algorithm, different random subsets. Bagging = with replacement; pasting = without. Aggregate by mode (class) or mean (reg). Individuals have higher bias; aggregation cuts variance (and can cut bias when errors cancel). Parallel-friendly.
Bagging versus pasting.
`BaggingClassifier(DecisionTreeClassifier(), n_estimators=500, max_samples=100, n_jobs=-1)`. `bootstrap=False` → pasting. Figure 6-5 style: bagged trees generalise better than one deep tree.
Out-of-bag (OOB). With bootstrap, ~37% of rows are left out per tree. Average OOB predictions ≈ validation score without a holdout. Set `oob_score=True`; read `oob_score_`.
Random patches / subspaces. Sample features too (`max_features<1.0`) and/or samples. Patches = both; subspaces = features only. Helps high-dimensional noisy data.
Random forests
Practice hub.
Bagging of decision trees with random feature subsets at each split (`max_features` defaults √n for classifiers). `RandomForestClassifier` / `Regressor` wrap the pattern with good defaults. Prefer the dedicated class over hand-rolled `BaggingClassifier(DecisionTree…)`.
Random forest construction.
Extra-trees. Random thresholds (`splitter="random"`) plus random features → more bias, less variance, faster fit. `ExtraTreesClassifier` (`bootstrap=False` by default). Try when forests overfit noisy wide data or you need speed.
RF versus extra-trees.
Feature importance. Mean impurity decrease across trees, weighted by sample mass, normalised to sum 1 → `feature_importances_`. Iris: petals dominate sepals. Use for screening; beware correlated features sharing credit. Prefer permutation importance when stakes are high.
Boosting overview
Fit discipline.
Train predictors sequentially, each fixing the previous. Less parallel than bagging; often higher accuracy with tuning.
AdaBoost
Error parts.
Misclassified instances get higher weights; next tree focuses on them. Learning rate shrinks weight updates. Final vote is weighted by predictor accuracy. `AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=…)`. Stumps are the classic weak learner. Cannot parallelise the sequence - scales worse than forests. Sensitive to outliers (they keep getting weight).
AdaBoost loop.
SAMME / SAMME.R: sklearn's multiclass AdaBoost variants; prefer the real/probabilistic path when available.
Gradient boosting (GBRT)
Train vs serve.
Fit the next tree to residual errors of the ensemble so far (regression view). Sum tree outputs for the prediction. Classification uses gradients of the loss (logistic/deviance).
Gradient boosting loop.
Manual three-tree quadratic demo in the book: train tree1 on y, tree2 on y−ŷ1, tree3 on residual2; predict by sum. `GradientBoostingRegressor(max_depth=2, n_estimators=3, learning_rate=1.0)`.
Learning rate / shrinkage. Small η + many trees usually beats η=1 with few trees. n_estimators too high → overfit; use early stopping (`n_iter_no_change`, validation fraction) or CV.
Subsample < 1.0 → stochastic gradient boosting: faster, more regularisation.
Figure 6-9/6-10 themes: underfit vs overfit ensembles as trees accumulate.
Histogram-based gradient boosting
Model promotion.
Bin features (default `max_bins=255`) → O(b×m) vs O(n×m×log m). Hundreds of times faster on large data; binning is itself a regulariser. Classes: `HistGradientBoostingClassifier/Regressor`.
HGB advantages.
Differences vs classic GBRT: early stopping auto if m>10k; no subsample; `max_iter` not `n_estimators`; limited tree knobs; native missing values and categorical integers (0 … < max_bins via `OrdinalEncoder`). First model to try on large heterogeneous tables in 2026 sklearn.
```python hgb = make_pipeline( OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1), HistGradientBoostingRegressor(), ) ```
Stacking
Study loop.
Train a blender (meta-learner) on out-of-fold predictions of base models (`cross_val_predict`), then refit bases on full data. `StackingClassifier` / `StackingRegressor`. Multi-layer stacks exist; complexity tax is real. Use when diverse strong bases remain complementary after simple voting.
Stacking training path.
Table 6-1 - when to use what
| Method | Use when | Example | |
|---|---|---|---|
| Hard vote | Diverse strong classifiers | Spam, disease | Soft vote |
| Probabilistic models | Credit risk, diagnosis | Bagging | High-variance bases |
| Financial risk | Pasting | Need independence, no replace | Segmentation |
| Random forest | Noisy high-dim tables | Churn, fraud | Extra-trees |
| Speed + variance cut | Real-time fraud | AdaBoost | Small/medium, weak learners |
| Credit scoring | GBRT | Max tabular accuracy | Housing, demand |
| HGB | Large tabular, speed | CTR, ranking | Stacking |
Chapter method map.
Tip from the book: RF, AdaBoost, GBRT, HGB are first tries on heterogeneous tabular data; little preprocessing.
Hyperparameter field notes
Forest. `n_estimators` until OOB flattens; `max_features` √n or log2 n; `min_samples_leaf` 1-10; `max_depth` None unless memory hurts. GBRT/HGB. `learning_rate` 0.01-0.1; trees until early stop; `max_depth` 2-8 (HGB often leaf-limited); `l2_regularization` on HGB. Vote/stack. Prefer uncorrelated bases; calibrate before soft vote if you can.
Bank takes
Fraud: extra-trees or HGB for speed; forests for importance screening. Credit risk: soft voting of logistic + HGB when you need both scores and a linear challenger. Adverse action still wants a shallow tree or linear model beside the champion ensemble.
EA / ops takes
Alert routing: bagged trees on ticket features. Capacity forecasts: HGB regression with early stopping. Stacking rarely worth the pager complexity unless the metric is a board OKR.
A closer look
Book soft-voting gains assume decent probability quality - dump raw SVM margins into soft vote without calibration and you can lose. Feature importance is not causal. AdaBoost on noisy labels can chase outliers. HGB categoricals must be integer-coded correctly or you leak / crash.
Exercise themes
Hard vs soft vote. OOB benefit. Extra-trees randomness. Importance meaning. AdaBoost vs GB residual view. Manual bagging of moons trees. Tune a forest; compare to HGB.
Pipeline patterns
```python # Fast tabular baseline pipe = make_pipeline( ColumnTransformer([...]), # nums pass-through; cats ordinal HistGradientBoostingClassifier(max_iter=300, learning_rate=0.05), ) # Forest + OOB rf = RandomForestClassifier(n_estimators=500, oob_score=True, n_jobs=-1) # Soft vote vc = VotingClassifier([("hgb", hgb), ("lr", lr)], voting="soft") ```
Bias-variance cartoon for bagging
Single deep tree: low bias, high variance. Bagged trees: similar bias, much lower variance. Boosted shallow trees: bias falls as trees add; variance rises if you go too far - early stopping is the brake.
OOB maths sketch
With bootstrap of size m, P(omit instance) ≈ (1−1/m)^m → 1/e ≈ 0.368. Expected OOB fraction ≈ 37%. Not magic validation for leaking pipelines - preprocessing fit inside each bootstrap if you are strict.
Permutation importance versus MDI
MDI (`feature_importances_`) is fast and biased toward high-cardinality features. Permutation importance on a validation set shuffles one column at a time and measures score drop - slower, fairer. Use both; trust permutation when ranking features for stakeholders.
Gradient boosting as GD in function space
Each tree approximates the negative gradient of the loss w.r.t. current predictions. Squared error → residuals. Logistic → y − p. That is why the chapter's residual demo is not a special case; it is the squared-error instance of a general pattern.
Early stopping discipline
Hold a validation slice (or use HGB's built-in). Stop when validation loss fails to improve for k rounds. Record `n_estimators_` actually used. Never report the training loss that kept falling.
Stacking leakage trap
If you train the blender on in-sample base predictions, it learns to trust overfit bases. Always OOF (`cross_val_predict` or Stacking's internal CV). After blender fit, refit bases on full data for deployment.
Diversity checklist
Different algorithms, different feature subsets, different hyperparameters, different training windows. Correlation of errors on a validation set: if ρ≈1, voting will not save you.
Compute budget
Forests and bagging parallelise across trees. Boosting is sequential (trees) but HGB vectorises bin work. Soft vote at inference runs every base - budget latency. Distil a soft-vote ensemble into one HGB if prod latency is tight.
Pitfalls
1. Soft-voting uncalibrated SVMs. 2. Huge `n_estimators` with η=1 on GBRT. 3. Reading MDI as causal. 4. Stacking without OOF. 5. Ignoring OOB when you have no holdout discipline. 6. Extra-trees without comparing to RF on your metric. 7. HGB with string categoricals not ordinal-encoded.
Test yourself
Hard vs soft? Bagging vs pasting? What makes a forest random? OOB rate? AdaBoost vs gradient boosting? Why HGB bins? How does stacking avoid leakage?
Voting deep dive
Diversity beats cloning. Five copies of the same logistic regressor vote as one. Pair linear models with trees and kernel methods so errors anti-correlate. Soft voting needs `predict_proba`; wrap decision-function models with `CalibratedClassifierCV` when you insist on soft vote.
Law of large numbers cartoon (Figure 6-3): a 51% coin looks fair in 10 tosses and decisive in 10,000. Ensembles need enough voters and enough independence. In credit, a bureau scorecard, an internal HGB, and a rules engine make a healthier vote than three HGBs with different seeds.
Hard voting ignores confidence; soft voting lets a 0.95/0.05 model outweigh a 0.51/0.49 shrug. That helps medical and fraud scores - and hurts if one model is wildly miscalibrated.
Bagging maths and practice
Bootstrap sample size usually equals m. Pasting without replacement needs `max_samples<1.0` or you clone the dataset. Aggregation: classification mode; regression mean. The house-price cartoon in the book: biases −40k and +50k average to +5k; variances add under independence and divide by n² for the mean - variance halves with two independent equal-variance regressors.
`n_jobs=-1` parallelises tree fits. Set `random_state` for reproducible OOB. `bootstrap_features` and `max_features` enable random subspaces/patches.
Warm-start patterns: grow `n_estimators` until OOB flattens; stop adding trees when the curve bends.
Random forest knobs that matter
| Knob | Typical start | Notes | |
|---|---|---|---|
| n_estimators | 300-800 | Until OOB plateaus | max_features |
| sqrt / log2 | Classification defaults | min_samples_leaf | 1-10 |
| Raise on noisy labels | max_depth | None | Cap if memory or latency |
| class_weight | balanced | Fraud/churn skew | oob_score |
`max_samples` < 1.0 speeds training and adds randomness. Missing values: recent sklearn forests handle them natively - still analyse missingness as a feature when it predicts risk.
Extra-trees specifics
Random thresholds skip the expensive sort/search for optimal cuts. Higher bias, lower variance, faster fit. `bootstrap=False` by default - every tree sees all rows, randomness comes from features/thresholds. Compare RF vs ET with the same budget; ET often wins on wide sensor data and loses on small clean tables.
Feature importance protocols
1. Fit forest; plot MDI bars. 2. Compute permutation importance on a locked validation set. 3. Cluster correlated features; report group importance. 4. For regulated models, prefer linear coefficients or SHAP with documented baselines over raw MDI.
MNIST example in the book highlights pixel importances as a heatmap - good pedagogy, weak production story. On tabular fraud, importances guide feature store pruning.
AdaBoost algorithm sketch
1. Init weights w_i = 1/m. 2. Fit weak learner; compute weighted error r. 3. Predictor weight α ∝ log((1−r)/r). 4. Multiply misclassified weights by e^α (times learning rate). 5. Renormalise; repeat. 6. Final prediction = weighted vote of α-weighted learners.
Use stumps (`max_depth=1`) unless you have reason. Cap `n_estimators` and shrink `learning_rate`. Outliers: consider `algorithm` variants and cleaning labels before AdaBoost.
Gradient boosting residual loop
```text F0 = mean(y) # or 0 for t in 1..T: resid = y - F_{t-1}(X) # or −dL/dF h_t = Tree.fit(X, resid) F_t = F_{t-1} + η * h_t ``` Classification replaces resid with the loss gradient. Shrinkage η is the learning rate. Stochastic GB samples rows each round (`subsample=0.8`).
Regularise with `max_depth` 2-3 for shallow boosters, or `max_leaf_nodes`, plus L2 on leaves in HGB/XGBoost. Early stopping beats guessing T.
HGB production checklist
- Encode categoricals to ints in [0, max_bins).
- Keep nulls; HGB handles them.
- `early_stopping=True` with a validation fraction.
- Tune `learning_rate`, `max_leaf_nodes`, `l2_regularization`, `max_iter`.
- Monitor train vs valid loss curves.
- Export `n_iter_` actually used.
- For ranking/CTR at huge scale, benchmark LightGBM - but keep HGB if sklearn-only is a constraint.
Stacking procedures
1. Choose diverse bases (HGB, linear, maybe kNN). 2. `StackingClassifier(estimators=..., final_estimator=LogisticRegression(), cv=5)`. 3. Inspect blender coefficients - negative weights can mean corrective roles. 4. Avoid deep multilayer stacks in prod unless the metric pays for the ops cost. 5. Serialize the whole stack; test latency.
Table 6-1 applied to a bank desk
- Card fraud real-time: ExtraTrees or light HGB.
- Application credit: soft vote logistic + HGB; shallow tree for letters.
- Collections propensity: HGB regression/classification with early stop.
- AML typology: RF importances → investigator features; champion HGB.
- Marketing uplift: careful - need uplift models, not plain GB of purchase.
Netflix Prize and modern comps
The book nods at Netflix Prize ensembles. Modern tabular comps still crown boosted trees and stacks; deep nets win when data is unstructured. Do not stack fifteen models for an internal churn model with 8k rows - you will overfit the leaderboard of one validation set.
Critical API notes
`HistGradientBoosting*` remains the sklearn-native speed path. `GradientBoosting*` is finer-grained but slower. XGBoost 2.x / LightGBM / CatBoost remain external champions; wrap them in sklearn-compatible APIs if you must mix. `VotingClassifier` still requires uniformly labelled classes. `StackingClassifier` `passthrough=True` appends original features to blender inputs - powerful and leak-prone if you misunderstand shapes.
End-to-end moons lab
1. Soft-vote logistic + SVC(probability=True) + RF. 2. Bag 500 trees with max_samples=100; compare OOB to test. 3. RF with oob_score; plot importances (synthetic moons → both features). 4. HGB with early stopping. 6. Stack HGB+logistic; compare to soft vote. Record accuracies in one table; ship the simplest model within 0.5 pp of the best.
Latency and footprint
| Ensemble | Train parallel | Infer cost | |
|---|---|---|---|
| Vote/Stack | Per base | Sum of bases | Bag/RF/ET |
| Trees | All trees | AdaBoost | No |
| All learners | GBRT/HGB | Limited | All trees |
Distil when p99 latency budgets fail: train a student HGB on teacher probabilities.
Pitfalls (expanded)
Ensemble pitfalls.
1. Soft-voting uncalibrated SVMs. 2. Huge `n_estimators` with η=1 on GBRT. 3. Reading MDI as causal. 4. Stacking without OOF. 5. Ignoring OOB when you have no holdout discipline. 6. Extra-trees without comparing to RF. 7. HGB with string categoricals. 8. Class imbalance without weights or resampling. 9. Tuning on the test set while stacking. 10. Deploying a 2 GB stack for a 1 ms budget.
Test yourself
Hard vs soft? Bagging vs pasting? Forest randomness sources? OOB≈? AdaBoost vs GB? HGB why bins? Stacking leakage fix? When ET over RF?
Wisdom-of-crowd limits
Crowds win when errors are uncorrelated. Shared label noise, shared feature leaks, and shared train windows correlate errors and kill the gain. In banking, three models trained on the same bureau pull with the same time split are not three opinions.
Soft voting calibration recipe
```python from sklearn.calibration import CalibratedClassifierCV bases = [ ("lr", lr), ("svm", CalibratedClassifierCV(svc)), ("hgb", hgb), ] VotingClassifier(bases, voting="soft") ``` Plot reliability diagrams before celebrating soft-vote lifts.
Bagging variance cartoon numbers
Var(mean of n i.i.d.) = σ²/n. Real trees are correlated, so effective n is smaller. RF's feature randomness lowers correlation; that is why it beats naive bagging of full trees on many datasets.
Random forest on imbalanced fraud
`class_weight="balanced_subsample"` reweights per bootstrap. Pair with PR-AUC and cost-weighted thresholds, not accuracy. OOB predictions can feed threshold selection if you treat them as validation scores carefully.
Extra-trees versus RF experiment
Same `n_estimators`, compare fit time and valid PR-AUC. ET often trains 2-5× faster. If ET ≤ RF within noise, prefer ET for real-time feature stores.
AdaBoost multiclass note
SAMME uses class errors; SAMME.R uses probabilities. Prefer probabilistic path when weak learners expose `predict_proba`. For noisy multiclass, HGB usually dominates AdaBoost in 2026 practice.
GBRT shrinkage grid
Try η ∈ {0.01,0.03,0.1} with early stopping. Smaller η needs more trees but often generalises better. Record the pair (η, n_iter_) in the model card.
HGB categorical gotcha
`OrdinalEncoder` order is arbitrary; HGB treats categories as unordered bins when marked categorical - use `categorical_features` indices. Unknown categories at serve time need a dedicated code (e.g. −1) handled in training.
Stacking blender choices
Linear blender: interpretable weights. RF blender: nonlinear meta. Start linear; upgrade only if CV proves it. `passthrough=True` lets the blender see original features - powerful on small stacks, easy to overfit.
Competition versus production
Kaggle: stack everything. Production: one HGB or RF with monitoring. Promote a stack only when lift clears ops cost and latency budget.
Monitoring ensembles
Track per-base score drift, not only ensemble score. A silent base failure can be masked by siblings. Log each base probability in shadow mode for a week after deploy.
Bank ensemble architecture
Champion HGB; challenger RF; linear scorecard for adverse action. Weekly PSI on features; monthly retrain with locked validation month. Soft vote only if calibration team signs off.
EA ensemble architecture
HGB on merge metrics predicts backlog risk; RF importances feed coaching dashboards; rules tree pages humans. No stacking in the pager path.
Exercise themes expanded
Explain hard vs soft with a 3-model toy. Compute OOB fraction ≈ 1/e. Show ET random thresholds. Derive AdaBoost α update. Implement residual boosting by hand on a quadratic. Build StackingClassifier with cv=5 and compare to Voting.
Full hyperparameter tables
RF/ET: n_estimators, criterion, max_depth, min_samples_split/leaf, max_features, bootstrap, oob_score, class_weight, n_jobs, random_state, max_samples. AdaBoost: estimator, n_estimators, learning_rate, algorithm. GBRT: loss, learning_rate, n_estimators, subsample, max_depth, min_samples_*, max_features, validation_fraction, n_iter_no_change. HGB: loss, learning_rate, max_iter, max_leaf_nodes, max_depth, min_samples_leaf, l2_regularization, max_bins, categorical_features, early_stopping, validation_fraction. Stack/Vote: estimators, final_estimator / voting, cv, passthrough.
Figure inventory
6-1 diverse classifiers; 6-2 hard vote; 6-3 coin law of large numbers; 6-4 bagging/pasting sample diagram; 6-5 decision boundaries bagging; 6-6 RF; 6-7 feature importance; 6-8 AdaBoost boundaries; 6-9 GBRT residual panels; 6-10 overfit/underfit boosting; 6-11 stacking blender; 6-12 blender training; 6-13 multilayer stack.
One-page chooser
Need speed on large data → HGB. Need importances yesterday → RF. Need calibrated blend of different families → soft vote or stack with OOF. Need real-time and noisy width → ET. Need tiny clean data and story → AdaBoost stumps or shallow tree. Default: HGB, then RF.
Diversity correlation lab
On a validation set, compute pairwise correlation of base-model errors (0/1). If mean ρ > 0.6, voting gains will be small - change bases or features. Log ρ next to the ensemble score in the experiment tracker.
Production serialisation
Prefer `joblib` dump of the full sklearn pipeline. Pin versions. On load, run a golden-file predict check. For stacks, ensure all bases and the blender load atomically.
OOB versus holdout
OOB is convenient, not sacred. Time-based tasks need time holdouts; OOB can leak through preprocessing fit on full data. Nest preprocessing inside each bootstrap or accept the optimistic bias consciously.
Final ensemble lab scorecard
Record for moons or your tabular task: single tree, bagging, RF, ET, AdaBoost, GBRT, HGB, soft vote, stack. Sort by valid metric; star the simplest model within 0.5 pp of the best. That star is what you deploy.
Deploy note
Serialize with joblib; pin sklearn; golden-file predict. Monitor per-base drift for votes and stacks.
AdaBoost weight update worked
Given weighted error r in (0,0.5), predictor weight alpha = learning_rate * log((1-r)/r). Misclassified weights multiply by exp(alpha) then renormalise. If r>=0.5 the weak learner is worse than chance; stop or discard. Stumps keep the hypothesis class weak on purpose so the sequence can keep finding new errors to fix.
Ensemble field notes
Bagging averages low-bias high-variance trees on bootstrap samples; Random Forests add feature randomness at splits. Boosting (AdaBoost, Gradient Boosting, HistGradientBoosting) fits additive stages to residuals or reweighted examples. Stacking blends heterogeneous learners via a meta-model on out-of-fold predictions.
Since publicationtabular default remains HistGradientBoosting / XGBoost / LightGBM-class tools before deep nets. Bank: forests for challengers and fraud; boosting for ranking. EA: forests on alert features when interpretability via importances is enough.
Ensemble field notes
Hard voting takes the mode of class labels; soft voting averages predicted probabilities and usually wins when members are well calibrated. On the moons toy set the book shows a soft-voting lift into the low-90s accuracy: treat that as a teaching demo, not a production claim. Always hold out a test set the ensemble never saw during member selection.
Bagging samples training instances with replacement; pasting samples without. Bagging often reduces variance more because bootstrap diversity is larger. In scikit-learn, `BaggingClassifier` / `BaggingRegressor` take `bootstrap=True` for bagging and `False` for pasting. Set `n_jobs=-1` for CPU parallelism; pin `random_state` for reproducibility.
Out-of-bag (OOB) evaluation uses instances left out of each bootstrap sample as a free validation set when `oob_score=True`. It is not a substitute for a true held-out test when you also tune hyperparameters on OOB. Use OOB for a quick sanity check while you still keep a locked test set for the final score.
Random patches sample both instances and features; random subspaces sample features on the full instance set. Feature sampling is why random forests decorrelate trees beyond plain bagging of decision trees. Extra-Trees (`ExtraTreesClassifier`) randomise split thresholds further and can train faster at a small accuracy cost.
Feature importance from a random forest (MDI / Gini importance) is biased toward high-cardinality features. Prefer permutation importance on a validation set when the bank scorecard team will act on the ranking. Figure 6-6 style MNIST heatmaps are pedagogy: tabular credit features need the same discipline plus leakage checks.
Boosting mechanics worth memorising
AdaBoost reweights misclassified instances so the next weak learner focuses on them. The ensemble prediction is a weighted vote (Equation 6-4 in the book). Learning rate shrinks each learner's contribution; too high and you overfit noise, too low and you need many estimators. `SAMME` (discrete) is the modern default path in recent sklearn; treat old `SAMME.R` notes as historical.
Gradient boosting fits each tree to the negative gradient of the loss (residuals for squared error). Shrinkage (`learning_rate`) times `n_estimators` is the main capacity dial. Early stopping via `n_iter_no_change` and a validation fraction stops you from the right-hand panel of Figure 6-10 (too many trees). Subsample < 1.0 gives stochastic gradient boosting and often better generalisation.
Histogram-based gradient boosting (`HistGradientBoosting*`) bins features and is the sklearn workhorse for medium/large tabular data. Native missing-value support removes a whole imputation branch from many bank pipelines. Prefer it over hand-rolled GBRT unless you need an external library's extras.
XGBoost / LightGBM / CatBoost (Oct 2026 still the industrial trio) sit outside the book's sklearn-first path. Use them when you need GPU training, categorical handling (CatBoost), or competition-grade defaults. Keep a HistGB baseline so you know the library lift is real.
Stacking and when to use what
Stacking trains a blender (meta-learner) on out-of-fold predictions of base models so the blender does not see in-sample predictions. Multilayer stacking (Figure 6-13) is powerful and easy to leak if you cheat on the fold discipline. Start with one blender layer and logistic regression or a small ridge model before a deep meta-network.
Table 6-1 style guidance still holds: voting for quick diverse classifiers; bagging/RF for variance-heavy trees; AdaBoost for simple weak learners; gradient / hist gradient for tabular accuracy; stacking when you can afford nested CV cost. For fraud and credit, ensembles win on ranking metrics (AUC/PR) more often than on raw accuracy.
Oral and exercise themes
Drill: hard vs soft voting; OOB benefit and its limit; bagging vs pasting; why random forests decorrelate trees; AdaBoost reweighting vs gradient residuals; early stopping for GBRT; when stacking leaks. Exercises in the source push you to train RF/ExtraTrees on MNIST-scale data, compare voting, and reason about feature sampling. Reproduce the decision-boundary figures mentally: smoother ensemble boundaries are the visual of variance reduction.
Closing
Ensembles turn mediocre diverse learners into a sharper committee. Bagging and forests cut variance; boosting cuts bias along the residual path; stacking learns how to weight specialists. Ship the simplest ensemble that clears the validation bar, log member versions, and never tune on the final test set.
7. Dimensionality Reduction
Too many features slow training and thin out the data - the curse of dimensionality. Géron covers projection versus manifold learning, then the workhorses: PCA (SVD, variance ratios, compression, randomized and incremental variants), random projection (Johnson-Lindenstrauss), LLE, and a tour of MDS, Isomap, t-SNE, LDA, and UMAP.
Why reduce
Study loop.
Speed, sometimes accuracy (less noise/redundancy), and 2D/3D visualisation for humans. Cost: dropped signal (JPEG analogy). Nets often learn their own compression - an extra PCA step is not always a win. Prefer reduction when m ≪ effective density needs, features are correlated, or you need plots for decision makers.
Curse of dimensionality
Intuition fails past 3D (Figure 7-1 tesseract). In a high-D unit hypercube, almost all points sit near a border. Average distance between two random points grows (unit square ~0.52; million-D cube ~408). Datasets become sparse; distance methods (kNN) weaken; SVMs and dense nets suffer; predictions extrapolate farther; models fit noise; interpretability dies. Fixing density by adding rows needs exponentially more data in n.
Projection versus manifold learning
Ops contrast.
Projection. Real data often near a low-D linear subspace (Figures 7-2, 7-3): drop to z1,z2 on that plane.
Two DR approaches.
Manifold learning. Subspaces can twist (Swiss roll, Figure 7-4). Flat projection squashes layers; unrolling preserves neighbourhoods (Figure 7-5). A d-manifold in n-space locally looks like R^d. Algorithms: LLE, Isomap, t-SNE, UMAP. Manifold hypothesis: real high-D data (MNIST digits) lie near low-D manifolds because generation has few degrees of freedom.
Caveat (Figure 7-6): the supervised task is not always simpler on the manifold. Reduction usually speeds training; better metrics are dataset-dependent.
PCA - preserving variance
Practice hub.
Choose the axis that maximises projected variance (= minimises reconstruction MSE). First PC = max variance direction; each next PC orthogonal and max remaining variance.
PCA via SVD.
SVD. Center X; X = U Σ V⊺; columns of V are PC directions (Equation 7-1). Sign/flip instability: retrain downstream models whenever PCA is refit.
```python X_c = X - X.mean(0) U, s, Vt = np.linalg.svd(X_c) W_d = Vt[:d].T X_d = X_c @ W_d # Equation 7-2 ```
sklearn: `PCA(n_components=2).fit_transform(X)` centers for you. `components_` holds PC rows.
Explained variance and choosing d
Fit discipline.
`explained_variance_ratio_` - e.g. [0.82, 0.11] ⇒ ~7% left for PC3. Prefer cumulative variance target (often 95%) over a magic integer d - except viz (d=2/3).
```python pca = PCA(n_components=0.95) X_r = pca.fit_transform(X_train) # MNIST → n_components_ == 154 ``` Plot cumsum (Figure 7-8); look for an elbow (~100 dims still strong on MNIST). Or CV `n_components` inside a pipeline with the supervised model - forests may want fewer PCs than linear models.
Compression and inverse
Error parts.
MNIST 784→154 at 95% variance: <20% size. `inverse_transform` ≈ W_dᵀ map back; error = reconstruction MSE (Equation 7-3). Figure 7-9: digits survive with mild blur.
Randomized PCA
Train vs serve.
`svd_solver="randomized"`: approx first d PCs in O(m d²)+O(d³) vs O(m n²)+O(n³). Great when d ≪ n. Default "auto" picks randomized when needed.
Incremental PCA
Model promotion.
`IncrementalPCA` + `partial_fit` on mini-batches for out-of-core / online. Or `np.memmap` file + `fit` when the solver supports it. Same target d=154 demo on MNIST batches.
Random projection
Study loop.
Multiply by a random Gaussian (or sparse) matrix. Johnson-Lindenstrauss: distances preserved within (1±ε) with high probability if d ≥ O(log m / ε²) - depends on m and ε, not n.
```python from sklearn.random_projection import ( johnson_lindenstrauss_min_dim, GaussianRandomProjection, SparseRandomProjection ) d = johnson_lindenstrauss_min_dim(m=5000, eps=0.1) # 7300 ``` Training almost free (shape only). Prefer SparseRandomProjection for memory/speed on wide/sparse text or genomics. Inverse via pseudoinverse is costly (O(d n²)). Fruit-fly olfaction analogy in the book: biology-ish locality-sensitive projection.
LLE
Nonlinear manifold method: (1) for each point, fit weights w to reconstruct it from k neighbours (Equation 7-4); (2) find low-D z preserving those weights (Equation 7-5). Unrolls Swiss roll (Figure 7-10) with local distances kept; global shape can stretch. Complexity heavy in the eigen step - poor on huge m. `sklearn.manifold.LocallyLinearEmbedding`.
Other techniques
| Method | Idea | Use | |
|---|---|---|---|
| MDS | Preserve pairwise distances | Low-D data | Isomap |
| Preserve geodesic NN-graph distances | Smooth manifolds | t-SNE | Keep neighbours close for viz |
| Cluster maps; not preprocessing | LDA | Supervised axes separating classes | Before another classifier |
Method chooser hub.
Figure 7-11: MDS/Isomap/t-SNE on Swiss roll - different distortions. Tip: t-SNE for slides; PCA/HGB pipeline for models.
umap-learn). For deep tabular/image compression, autoencoders (Chapter 18) and foundation-model embeddings often replace classical DR in production - keep PCA as the linear baseline and audit tool.Pipeline patterns
```python pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95), HistGradientBoostingClassifier()) # tune n_components jointly # viz from sklearn.manifold import TSNE Z = TSNE(n_components=2, perplexity=30).fit_transform(X_sample) ```
A closer look
PCA is linear - Swiss rolls need LLE/UMAP/AE. Sign flips break frozen models. 95% variance ≠ 95% task signal. t-SNE is stochastic and not a transform you `transform()` new points through reliably for training features (embedding fits the batch you give it). JL guarantees are on distances, not on classifier accuracy.
Exercise themes
Motivations/drawbacks. Curse definition. Inverse possible? (PCA yes with error; t-SNE no clean inverse.) PCA on nonlinear data - limited. 95% variance → d unknown until fit. When regular vs randomized vs incremental vs random projection. Chain two DR methods? MNIST t-SNE map of digits.
Worked MNIST numbers
784 → 154 dims at 95% variance (~5:1 compression). Randomised search with RF found ~57 PCs enough for that model class; SGD would keep more. Reconstruction shows blur, not label flips, at 95%.
Centering and scaling
PCA centers; it does not standardise variances. On mixed-scale tabular features, `StandardScaler` before PCA or you let high-variance dollars dominate PCs. For images with comparable pixel scales, centering may suffice.
Kernel PCA note
Book edition lean; sklearn `KernelPCA` (RBF etc.) lifts nonlinear structure via kernels then PCA in feature space. Costly; try before deep AEs on medium data.
Incremental / memmap ops
```python inc = IncrementalPCA(n_components=154) for batch in np.array_split(X_train, 100): inc.partial_fit(batch) ``` Flush memmap after writes. Monitor `explained_variance_ratio_` stability across batch orders.
Random projection density
Sparse RP density default ~1/√n. Nonzeros ±1/√(d r). Keep sparse inputs sparse unless `dense_output=True`.
LLE knobs
`n_neighbors` (k), `n_components`, `reg` for weight stability. Too small k → brittle; too large → shorts the manifold. Not for millions of rows without approx NN.
Visualisation discipline
1. PCA/UMAP/t-SNE on a sample, not the full billion-row log. 2. Colour by label or segment - never invent clusters from the projection alone. 3. Do not train classifiers on t-SNE coordinates. 4. Freeze PCA loadings in the model card when used in prod.
Bank vignette
A risk desk builds PCA on 120 correlated market factors; five PCs explain 90% for stress scenarios. Loadings show PC1 ≈ equity beta, PC2 ≈ rates. They publish the loadings and forbid silent refits mid-quarter.
EA vignette
Forty service SLIs → PCA to 3 components for an exec heatmap. Alerts still fire on raw SLIs; PCA is narrative, not paging.
Pitfalls
DR pitfalls.
1. PCA without scaling on mixed units. 2. Shipping t-SNE coordinates as features. 3. Forgetting to refit downstream after PCA flip. 4. 99% variance with no task metric check. 5. Dense Gaussian RP on huge n (use sparse). 6. LLE on large m without budget. 7. Treating UMAP clusters as ground truth segments.
Test yourself
What is the curse? Projection vs manifold? Why max variance? How does SVD give PCs? `n_components=0.95` meaning? Randomized vs incremental PCA? JL lemma dependence? LLE two steps? t-SNE for training features?
Choosing among PCA family
| Situation | Pick | |
|---|---|---|
| Medium dense, fit in RAM | PCA default/auto | d ≪ n, need speed |
| Randomized PCA | Out of core / streams | IncrementalPCA |
| Ultra-wide sparse | SparseRandomProjection | Nonlinear unroll, small m |
| LLE / UMAP | Class separation | LDA |
PCA family chooser.
Reconstruction error monitoring
Track MSE(X, inverse(transform(X))) on a holdout. Spikes after a data schema change mean the subspace moved - retrain and A/B.
Curse of dimensionality, worked
We live in three dimensions, so a tesseract (Figure 7-1) already breaks intuition. Géron's unit-hypercube facts are worth memorising for interviews: in 2D, a random point is almost never within 0.001 of the border; in 10,000-D, that "near border" chance exceeds 99.999999%. Average pairwise distance grows with dimension - about 0.52 in the unit square, about 0.66 in the unit cube, about 408 in a million-dimensional unit hypercube. High-D data is sparse. Distance-based learners lose traction. Models extrapolate farther from any neighbour. Noise becomes easier to fit than signal. Interpretability collapses because every feature can claim a tiny slice of explanation.
Curse of dimensionality.
The theoretical fix - add enough rows to restore density - fails in practice. With 100 features on [0,1], uniform packing to average spacing 0.1 already demands more points than atoms in the observable universe. That is the curse in one sentence: volume grows so fast that data cannot fill it.
When reduction helps, and when it hurts
MNIST border pixels are nearly constant white; neighbouring pixels are correlated. Dropping or merging them removes redundancy. Random forests in Chapter 6 already ranked those border pixels near zero importance. JPEG-style warning still applies: compress too hard and you delete the stroke that separates a 3 from an 8. Neural nets can learn task-specific compression inside the first layers, so a blind PCA front-end sometimes costs accuracy. Use reduction when the table is wide, noisy, and correlated; skip it when a deep model already sees enough data to learn its own subspace.
Visualisation remains a clean win. Two or three dimensions let you show clusters to a credit committee. That plot is a communication tool, not a feature store.
Projection story on the 3D cloud
Figure 7-2 scatters points that already lie near a plane. Orthogonal projection onto that plane yields Figure 7-3 with new axes z1, z2. You did not invent nonlinear magic; you found a linear subspace that holds most of the cloud. PCA automates that search by variance.
Manifold story on the Swiss roll
Figure 7-4 twists a 2D sheet in 3D. Dropping x3 squashes layers (Figure 7-5 left). Unrolling recovers the sheet (right). A d-manifold is a d-dimensional sheet bent inside n-space. Handwritten digits live on such a manifold: random 28×28 noise almost never looks like a digit, so the effective degrees of freedom are far below 784. Manifold learning tries to recover coordinates on that sheet.
Figure 7-6 is the honesty check. Top row: unrolling makes classification trivial. Bottom row: the true boundary was a simple plane in 3D and becomes four segments on the unroll. Simpler geometry in ambient space can become harder on the manifold. Always measure the downstream metric.
PCA as variance-maximising projection
Figure 7-7 shows three candidate axes. The solid axis keeps the most spread; the dotted axis flattens the cloud into a pancake of almost no variance. Max variance ≡ min reconstruction MSE for linear projections. That 1901 idea is still the default linear compressor in sklearn.
Principal components are ordered orthogonal directions: PC1 max variance, PC2 max among axes ⊥ PC1, and so on. After projection, z1 tracks PC1. Each PC is a unit vector; signs can flip between fits. Pipelines that freeze a classifier on top of PCA must refit end-to-end when PCA is refit, or a flipped PC will look like a different world.
SVD mechanics
Center X (sklearn does this inside `PCA`). Factor X = U Σ V⊺. Rows of V⊺ (columns of V) are PC directions. Take the first d rows of V⊺, transpose to W_d, and compute X_d = X_centered W_d. NumPy: `U, s, Vt = np.linalg.svd(X_c); X_d = X_c @ Vt[:d].T`. sklearn: `PCA(n_components=d).fit_transform(X)` with `components_` available for loadings.
Explained variance operations
`explained_variance_ratio_` for the 3D toy might read [0.82, 0.11]; the third PC holds ~7% and is a candidate to drop. For MNIST, fit full PCA, `cumsum = np.cumsum(ratio)`, `d = argmax(cumsum >= 0.95)+1` → 154. Cleaner: `PCA(n_components=0.95)` and read `n_components_`. Figure 7-8's elbow often sits near 100 dims for MNIST - another operating point if you value speed over the last points of variance.
Choosing PCA dimensions.
Hyperparameter search beats superstition. A pipeline `PCA + RandomForest` with `RandomizedSearchCV` in the book landed near 57 PCs for the forest. A linear `SGDClassifier` would demand more. Task-aware d beats a universal 95% rule.
Compression, inverse, reconstruction
154/784 ≈ 19.6% of the columns at 95% variance. `inverse_transform` implements X_hat = X_d W_d⊺ (plus re-centering). Reconstruction error is MSE(X, X_hat). Figure 7-9 shows digits that remain readable with soft blur - the visual companion to a 5% variance sacrifice.
Randomized and incremental PCA
Full SVD costs O(m n²)+O(n³). Randomized PCA targets the top d components in O(m d²)+O(d³) - mandatory when d ≪ n. `svd_solver="randomized"` or `"auto"`.
Incremental PCA streams mini-batches via `partial_fit`, or rides `np.memmap` when the array lives on disk. Same MNIST 154-D target works batch-wise. Use this for online PCA on event streams in fraud feature shops.
Random projection and JL
Draw a Gaussian matrix P with variance 1/d and multiply. Johnson-Lindenstrauss: to keep squared distances within (1±ε) with high probability, you need d = O(log m / ε²). Example: m=5000, ε=0.1 → d≥7300 via `johnson_lindenstrauss_min_dim`. Notice n disappeared - the bound cares about how many points you must separate, not how wide they were.
Random projection path.
`GaussianRandomProjection` and `SparseRandomProjection` implement the idea. Sparse wins on memory (tens of MB vs ~1 GB in the book example) and often on speed; density defaults near 1/√n. Inverse needs a pseudoinverse - expensive; usually you project only forward for a downstream model. Fruit-fly olfaction aside in the book is optional colour; the engineering point is locality-sensitive random maps.
LLE algorithm
For each x^(i), find k neighbours; solve for weights w_ij that reconstruct x^(i) from those neighbours (Equation 7-4) with sum-to-one constraints. Then find low-D z^(i) that obey the same weights (Equation 7-5). `LocallyLinearEmbedding` unrolls the Swiss roll (Figure 7-10) with local distances respected and global shape somewhat stretched. Complexity: NN search plus a heavy eigenproblem - fine for thousands of rows, painful for millions.
LLE two-step idea.
MDS, Isomap, t-SNE, LDA, UMAP
MDS preserves pairwise distances (better on already-low-D data). Isomap preserves geodesic distances on a NN graph - strong on smooth single manifolds. t-SNE keeps neighbours close for visualisation; it amplifies clusters and is a poor preprocessing step for classifiers. LDA uses labels to find separating axes - supervised reduction before another classifier. UMAP (umap-learn) preserves local and more global structure and scales better than t-SNE; still primarily a viz tool unless you freeze a transform carefully. Figure 7-11 compares MDS/Isomap/t-SNE on the Swiss roll: each distorts differently; pick from the downstream need, not from pretty pictures alone.
Bank and EA applications
Bank. Orthogonal macroeconomic factors via PCA on standardised series; publish loadings each quarter; forbid silent refits. Sparse random projection on ultra-wide transactional indicator vectors before a linear SVM. LDA as a quick supervised projection before a small logistic lab model. Never put t-SNE axes into a scorecard.
Applied DR splits.
EA. PCA on correlated cloud metrics for an exec two-axis heatmap ("load" vs "errors"). Keep paging on raw SLIs. UMAP for a one-off incident postmortem slide, then delete the coordinates from any feature store.
A closer look and Oct 2026
PCA remains linear. KernelPCA exists in sklearn for RBF-style nonlinear lifts at a cost. Deep autoencoders and foundation embeddings (Chapter 18 / modern practice) often beat classical DR on unstructured inputs - still start with PCA as the debug baseline. sklearn 1.9.x APIs for PCA/IPCA/random projection/LLE/TSNE match the book; UMAP stays third-party. Sign flips and column reorderings after refit are still silent production killers - version the transformer with the model.
Exercise themes with answer sketches
Motivations: speed, noise, viz; drawbacks: information loss, possible worse metrics. Curse: sparse high-D geometry. Inverse: PCA yes with error; t-SNE no stable inverse. PCA on nonlinear data: limited, use LLE/UMAP/AE. 95% variance: d unknown until fit (154 on MNIST). Choose regular/randomized/incremental/random projection from RAM, d≪n, streaming, ultra-wide sparse. Chaining DR: sometimes PCA then UMAP for viz; rare in prod features. MNIST t-SNE: expect digit clusters with messy boundaries.
Operational checklist
1. Scale features before PCA on tabular mixes. 2. Pick d by variance, elbow, or CV - record which. 3. Log reconstruction MSE and supervised metric. 4. Freeze and version the transformer with the model. 5. Ban t-SNE/UMAP coordinates as training features unless you have a transform that applies to new rows. 6. Prefer SparseRandomProjection on wide sparse inputs. 7. Re-run loadings review when the feature schema changes.
Pitfalls
1. PCA without scaling on mixed units. 2. Shipping t-SNE coordinates as features. 3. Forgetting to refit downstream after PCA flip. 4. 99% variance with no task metric check. 5. Dense Gaussian RP on huge n. 6. LLE on large m without budget. 7. Treating UMAP clusters as ground-truth segments. 8. Using PCA compression as a security anonymiser (it is not).
Test yourself
Define the curse with one numeric fact. Projection vs manifold with Swiss roll. Why max variance? SVD role? `n_components=0.95`? Randomized vs incremental? JL dependence on m not n? LLE's two optimisations? Why t-SNE is for slides?
KernelPCA and friends
`KernelPCA(kernel="rbf", gamma=...)` approximates nonlinear PCA. Tune gamma and n_components with the supervised metric. On medium Swiss-roll-like data it can rival LLE; on large data, budget the kernel matrix or use approximate methods.
End-to-end DR lab
1. MNIST PCA 95% → 154 dims; measure RF accuracy vs full. 2. RandomizedPCA same d; compare wall time. 3. IncrementalPCA mini-batches; match explained variance. 4. SparseRandomProjection with JL d; kNN accuracy vs PCA. 5. LLE on Swiss roll; plot. 6. t-SNE sample of 5k MNIST for a slide - do not train on coordinates.
Production PCA card
Record: scaler type, n_components or variance target, svd_solver, reconstruction MSE, supervised metric delta vs baseline, loadings plot hash, refit policy.
Explained variance versus task metric table
Log a table: d in {20,50,100,154,300}, cumulative variance, RF accuracy, linear SGD accuracy, train time. The d that maximises RF may differ from the d that maximises SGD. Ship the column that matches the production model class.
Ch7 deep panel 100
This stretch centers on Ch7 ideas. Source cue: problems involve thousands even millions features for each training instance. Not only all these features make training extremely slow, but.
If the book shows code here, retype it from memory and diff.
This stretch centers on PCA. Source cue: Reducing dimensionality can also drop some useful information, just like compressing image JPEG can degrade its quality: can make your.
If the book shows code here, retype it from memory and diff.
This stretch centers on PCA. Source cue: what goes high-dimensional space. Then will consider the two main approaches dimensionality reduction projection and manifold learning and will through.
If the book shows code here, retype it from memory and diff.
Ch7 deep panel 101
This stretch centers on Ch7 ideas. Source cue: any dimension But 10,000-dimensional unit hypercube, this probability greater than 99.999999 Most points high-dimensional hypercube are very close the border..
If the book shows code here, retype it from memory and diff.
Closing
Dimensionality reduction is a tool, not a religion. Project, plot, then decide whether the axes earn their keep.
8. Unsupervised Learning Techniques
Labels are optional. Géron surveys clustering (k-means, DBSCAN, and cousins), Gaussian mixtures for density and anomalies, and a toolbox of isolation forest, LOF, and one-class SVM - with image segmentation, semi-supervised learning, and active learning as the applied bridges.
What unsupervised work is for
Study loop.
Clustering groups similar instances without labels. Density estimation learns P(x). Downstream uses in the book: customer segmentation, data analysis, dimensionality reduction, feature engineering, semi-supervised learning, search, image segmentation. Figure 8-1 contrasts labelled Iris (classification) with unlabelled Iris (clustering).
k-means
Lloyd-Forgy: assign points to nearest centroid; move centroids to mean of assignees; repeat. `KMeans(n_clusters=k).fit_predict(X)` on blob data (Figure 8-2). You must pick k. Labels_ are cluster indices, not class labels. `cluster_centers_` holds centroids; `transform(X)` returns distances to each centroid - a useful feature map.
k-means loop.
Hard clustering assigns one cluster; soft scores come later via GMM responsibilities or distance transforms.
Initialization and inertia
Ops contrast.
Bad init → bad local minima (Figure 8-5). Options: hand-picked `init=array`, or multiple random restarts (`n_init`) keeping lowest inertia (Equation 8-1: sum of squared distances to assigned centroid). `inertia_` and `score()` (= −inertia) expose it. k-means++ (`init="k-means++"`, the default) spreads initial centroids and usually needs fewer restarts.
Accelerated and mini-batch
Practice hub.
Elkan's triangle-inequality accelerations help. `MiniBatchKMeans` uses small random batches - faster on large m, slightly higher inertia. Prefer mini-batch when full k-means will not finish in your budget.
Choosing k
Fit discipline.
Inertia always falls with k - useless alone. Elbow plot (Figure 8-7): look for the bend. Better: silhouette score (mean silhouette coefficient) - near +1 well clustered, 0 on boundary, −1 in the wrong cluster. Figures 8-8/8-9 style: silhouette diagrams per k. Book often lands near k=5 for the five-blob toy when silhouettes agree.
Choosing k metrics.
Limits of k-means
Error parts.
Assumes similar-scale spherical-ish clusters. Fails on stretched ellipses, unequal sizes, non-blob manifolds (moons). Scale features first. For odd shapes, DBSCAN/GMM/spectral methods.
Image segmentation
Train vs serve.
Cluster pixels in colour space; replace each pixel by its centroid colour. Low k → posterisation. Classic compression/segmentation demo; modern vision uses CNNs (Chapter 12), but k-means remains a teaching and baseline tool.
Semi-supervised and active learning
Model promotion.
Cluster then propagate labels from a few annotated representatives per cluster - label efficiency. Active learning: repeatedly label the instances the model is least sure about (or nearest the boundary). Clustering can propose diverse candidates to annotate.
DBSCAN
Study loop.
Density-based: core points have ≥ `min_samples` neighbours within `eps`; borders touch cores; the rest are anomalies (−1). Finds arbitrary shapes; no k. Fails when densities vary wildly. `DBSCAN(eps=0.05, min_samples=5)` on moons separates the arcs. `fit_predict`; no `predict` for new points in classic DBSCAN - use the learned cores or switch to a classifier on top.
DBSCAN ideas.
Other clustering algorithms
| Algorithm | Idea | Notes | |
|---|---|---|---|
| Agglomerative | Hierarchical merges | Dendrograms; connectivity constraints | Mean-shift |
| Mode seeking | Bandwidth; can leave outliers | Affinity propagation | Message passing |
| Chooses count of clusters | Spectral clustering | Graph Laplacian embedding then k-means | Good on manifolds |
Gaussian mixtures
Ops contrast.
Data ~ mixture of k Gaussians with weights ϕ, means μ, covariances Σ. EM estimates parameters. `GaussianMixture(n_components=3, n_init=10)`. Soft responsibilities; ellipsoidal clusters (Figure 8-10). `score_samples` = log density; `predict` = hard component.
GMM EM loop.
Covariance types: full, tied, diag, spherical - bias/variance knobs for shape.
GMM anomaly detection
Practice hub.
Low-density instances = anomalies. Set threshold from a percentile (e.g. 2% defect rate): `densities = gm.score_samples(X); anomalies = X[densities < np.percentile(densities, 2)]` (Figure 8-17). Novelty detection trains on clean data; anomaly detection allows contamination. Too many outliers bias GMM - clean or use robust methods.
Anomaly threshold path.
Selecting components
AIC/BIC vs k (Figures in book): minimise information criteria. BayesianGaussianMixture can drive unused weights ≈ 0 when `n_components` is an upper bound (demo: 10 → 3). Moons still break ellipsoidal GMMs (Figure 8-20) - density may be OK for anomalies even when cluster count is wrong.
Likelihood reminder
Max likelihood for GMM via EM; singularities if a component collapses on a point - `reg_covar` and multiple `n_init` help.
Isolation forest, LOF, one-class SVM
IsolationForest: random trees isolate points; anomalies need fewer splits. Strong in high-D. LOF: local density vs neighbours. OneClassSVM: novelty; separates data from origin in kernel space; tune ν (outlier fraction) and kernel.
Two anomaly tools.
k-means step-by-step on blobs
1. Sample five blob centres with `make_blobs`. 2. Fit `KMeans(n_clusters=5, n_init=10, random_state=42)`. 3. Plot assignments and `cluster_centers_`. 4. Call `transform` to get a 5-D distance feature space - feed to a linear model. 5. Compare inertia across `n_init` seeds; confirm k-means++ stability.
Voronoi cell boundaries in feature space are the decision surfaces. New points map to the nearest centroid with `predict`.
Inertia versus silhouette decision rule
Plot inertia vs k and silhouette vs k side by side. Prefer the k that maximises silhouette among elbow candidates. If business needs a fixed segment count (marketing wants 6 personas), constrain k and report silhouette as quality, not as a chooser.
Silhouette for one sample i: (b−a)/max(a,b) where a is mean intra-cluster distance and b is mean distance to the nearest other cluster. Mean silhouette is the chart title number.
Mini-batch trade-offs
On large card-transaction feature matrices, `MiniBatchKMeans(batch_size=1024)` finishes when full KMeans does not. Expect a small inertia penalty. Re-run full KMeans on a sample to sanity-check segment meaning.
Image segmentation recipe
```python X = image.reshape(-1, 3) # RGB km = KMeans(n_clusters=8, n_init=5).fit(X) seg = km.cluster_centers_[km.labels_].reshape(image.shape) ``` Use for demos and crude compression. For production vision, prefer Chapter 12 models.
Semi-supervised label propagation sketch
Cluster with k-means; for each cluster, label the instance closest to the centroid (or a random subset). Propagate that label to the cluster, or train a classifier on the labelled subset and score the rest. Active learning loop: query labels where the classifier's margin is smallest or where density is high but labels are missing.
DBSCAN parameter dance
`eps` too small → everyone is noise. Too large → one giant cluster. `min_samples` too large → only dense cores survive. Grid `eps` via k-distance plots (sorted distance to k-th neighbour; look for the knee). Moons and circles are the unit tests; equal-density blobs with thin bridges can still confuse it.
No `predict` for new points: options include (a) assign to nearest core sample's cluster, (b) fit a classifier on DBSCAN labels, (c) refit periodically on a rolling window.
HDBSCAN note
Variable-density successor to DBSCAN; common in industry stacks via the `hdbscan` package. Mention in model reviews when DBSCAN's single `eps` fails. Not required by Géron's chapter, but critical-lens relevant.
GMM EM intuition
E-step: estimate responsibilities (soft assignment probabilities). M-step: update weights, means, covariances given responsibilities. Repeat until log-likelihood plateaus. Multiple `n_init` fights label switching and bad inits. `reg_covar` stops singular covariances when a point owns a component.
Covariance type chooser
| Type | Freedom | Risk | |
|---|---|---|---|
| full | Each cluster own full Σ | Overfit small m | tied |
| Shared Σ | Stable, less flexible | diag | Axis-aligned ellipses |
| Fast | spherical | Scaled identity | Closest to k-means |
Start diag or full with enough data; use tied when clusters should share shape.
AIC/BIC practice
```python for k in range(1, 9): gm = GaussianMixture(k, n_init=5).fit(X) print(k, gm.bic(X), gm.aic(X)) ``` Pick k at minimum BIC unless domain knowledge overrides. BayesianGaussianMixture with a high `n_components` ceiling is the lazy alternative.
Anomaly threshold ops
Map defect rate or fraud rate to a percentile threshold on `score_samples`. Track precision/recall on a labelled audit set. If GMM fits outliers into their own tiny components, density scores can fail - prefer IsolationForest when contamination is high.
Isolation forest details
Anomaly score from mean path length (shorter → more anomalous). `IsolationForest(contamination=0.02)`. Works with many features; scale less critical than for k-means. Great first fraud screen.
LOF and one-class SVM
LOF: `LocalOutlierFactor(n_neighbors=20)` - local density ratio; good when normal density varies by region. `novelty=True` for predict on new data. One-class SVM: `OneClassSVM(kernel="rbf", nu=0.05)` - novelty on cleaner training sets; tune γ and ν carefully; scale features.
Bank playbook
1. Segment merchants with MiniBatchKMeans on behaviour features; silhouette + SME review. 2. IsolationForest on account-day vectors for AML typologies. 3. GMM density on login telemetry for account takeover. 4. DBSCAN on geo-time points for collusive rings. 5. Never auto-mail customers from a cluster ID without a policy layer.
Bank unsupervised map.
EA playbook
Cluster microservices by SREs' metric fingerprints; page when a service leaves its cluster's density under a GMM or IF score. Use active learning to label incident types on the uncertain boundary.
A closer look
k-means is overused on non-blob data. Silhouette can favour wrong k on odd geometries. GMM soft assignments are not calibrated probabilities of "business segment membership." IsolationForest `contamination` is a prior, not magic truth. Cluster labels shuffle between refits - map via Hungarian matching before comparing runs.
Exercise themes
k-means vs GMM on ellipses. Elbow vs silhouette. DBSCAN on moons. Semi-supervised label propagation accuracy vs #labels. GMM anomaly percentile vs IF. BIC curve. Bayesian GMM weight collapse. Compare LOF and IF on a contaminated Gaussian.
Full API cheat sheet
`KMeans`, `MiniBatchKMeans`, `DBSCAN`, `AgglomerativeClustering`, `MeanShift`, `AffinityPropagation`, `SpectralClustering`, `GaussianMixture`, `BayesianGaussianMixture`, `IsolationForest`, `LocalOutlierFactor`, `OneClassSVM`. Metrics: `silhouette_score`, `calinski_harabasz_score`, `davies_bouldin_score`.
Scaling and pipelines
```python pipe = make_pipeline(StandardScaler(), KMeans(n_clusters=5, n_init=10)) # GMM gm = make_pipeline(StandardScaler(), GaussianMixture(n_components=5, n_init=5)) # IF often OK without scaling; still document feature units ```
Soft cluster features
Use `transform` distances (k-means) or `predict_proba` responsibilities (GMM) as inputs to supervised models - classic unsupervised feature engineering from the chapter's motivation list.
Pitfalls
Unsupervised pitfalls.
1. k-means on unscaled features. 2. Picking k by inertia alone. 3. DBSCAN eps from thin air. 4. GMM anomalies with heavy contamination. 5. Treating cluster IDs as stable across refits. 6. One-class SVM without scaling. 7. Expecting GMM to find moons. 8. Shipping segments without business names.
Test yourself
Inertia definition? Why elbow? Silhouette range? k-means++ idea? DBSCAN roles of eps and min_samples? GMM generative story? Anomaly vs novelty? IF isolation idea? When Bayesian GMM?
k-means worked narrative
Géron opens the clustering half with five Gaussian blobs (Figure 8-2). You can see the groups; the algorithm must rediscover them from coordinates alone. After `fit_predict`, every row carries a cluster index in `labels_`, and `cluster_centers_` holds the five means. Those indices are not class labels from a supervised problem; they are an invented taxonomy. If you rerun with another seed and another `n_init` draw, indices can permute; match clusters across runs with centroid distance or the Hungarian algorithm before you compare.
`transform(X)` returns the matrix of distances to all centroids. That matrix is a nonlinear feature map: a linear model on distance-to-centroid features often beats a linear model on raw X for blob-like structure. This is the feature-engineering use case listed in the chapter introduction.
Inertia, score, and multiple inits
Equation 8-1 defines inertia as the sum of squared distances from each point to its assigned centroid. Lower is better; `score` returns the negative so that greater-is-better APIs stay consistent. Figure 8-5 shows how a bad init lands in a terrible local minimum with inertia near 600 versus near 212 for a good solution. `n_init` reruns and keeps the best. k-means++ picks the first centroid uniformly, then picks subsequent centroids with probability proportional to squared distance from the nearest already-chosen centroid, spreading seeds. It is the default for a reason; do not switch to pure random without a benchmark.
Accelerated algorithms and mini-batches
Elkan-style accelerations use triangle inequalities to skip redundant distance computations. You rarely configure them directly; sklearn enables what it can. `MiniBatchKMeans` pulls small batches, updates centroids incrementally, and finishes on datasets where full k-means would thrash. Expect slightly worse inertia and slightly different boundaries. For overnight batch segment jobs on tens of millions of rows, mini-batch is the practical default.
Elbow, silhouette, and business k
Figure 8-6 shows k=3 merging distinct blobs and k=8 splitting clean ones. Figure 8-7 inertia curve bends at the elbow. Because inertia keeps falling, the elbow is a heuristic, not a proof. Silhouette coefficients diagnose separation: for each sample, compare mean intra-cluster distance a to mean distance b toward the nearest other cluster; coefficient (b-a)/max(a,b). Near +1 means crisp assignment; near 0 means on the fence; negative means probably mis-assigned. Plot mean silhouette versus k; also inspect per-cluster silhouette bars to catch a bad cluster hiding inside a decent average.
Marketing sometimes demands exactly six personas. Then k is a constraint. Report silhouette and SME review scores as quality metrics instead of pretending the data chose six.
When k-means is the wrong shape
Stretched ellipses, rings, moons, and unequal blob sizes break the equal-spherical assumption. Scaling helps when units differ, but it will not turn a moon into a blob. Switch to DBSCAN, spectral clustering, or GMM with full covariances. The chapter honesty about limits is the part practitioners skip.
Image segmentation as clustering
Reshape an RGB image to N by 3, run k-means, replace each pixel with its centroid colour, reshape back. Small k posterises; large k approaches the original. It teaches the algorithm more than it competes with modern segmentation networks. Still useful for thumbnail compression demos and for clustering in other handcrafted colour spaces.
Semi-supervised learning bridge
Label a few representative points, often those nearest centroids after clustering, and propagate. Accuracy can jump versus the same label budget spent at random because clusters diversify the labelled set. Active learning extends the idea: ask humans to label the points the current model finds most uncertain, or the densest unlabelled regions. Banks feel that pain on AML typologies and document classes.
DBSCAN in practice
Core point: at least `min_samples` points within distance `eps` (counting itself). Density-reachable chains of cores form clusters; border points attach; the rest are noise labelled -1. Arbitrary shapes work; you never set k. The failure mode is variable density: one `eps` cannot describe both a tight core and a diffuse halo. k-distance plots help pick `eps`: sort each point distance to its k-th neighbour and look for the knee.
Classic sklearn DBSCAN has no `predict` for new rows. Production patterns: nearest-core assignment, a downstream classifier trained on DBSCAN labels, or periodic full refits on a rolling window. For variable density, evaluate HDBSCAN from the broader ecosystem even though Geron text stays with DBSCAN.
Clustering algorithm menu
Agglomerative clustering builds a hierarchy with dendrograms for taxonomy-minded stakeholders; connectivity constraints respect map adjacency. Mean-shift seeks density modes via bandwidth; it can leave outliers unassigned. Affinity propagation exchanges messages and invents its own cluster count until it invents too many. Spectral clustering embeds the Laplacian of a similarity graph then runs k-means; strong on manifolds, costlier on large m.
GMM generative story
Pick component j with probability phi_j; sample x from Normal(mu_j, Sigma_j). Soft responsibilities replace hard k-means assignments. Ellipsoidal clusters with different sizes and orientations appear (Figure 8-10). EM alternates E-step responsibilities and M-step closed-form updates. `n_init` and `reg_covar` are the practical stabilisers. `score_samples` returns per-instance log-likelihoods under the mixture, the backbone of density-based anomaly cuts.
Anomaly and novelty
Choose a density threshold from a business rate (2% defects maps to the 2nd percentile of scores). Figure 8-17 marks low-density stars. Anomaly detection fits contaminated data; novelty detection assumes clean training. GMM tries to model everything; heavy contamination warps normality. IsolationForest and LOF often survive contamination better.
BIC, AIC, Bayesian GMM
Information criteria penalise extra components. Plot BIC versus k; take the minimum. `BayesianGaussianMixture` with a generous `n_components` can drive unused weights to about 0 (10 down to 3 in the book demo). Moons still defeat ellipsoidal mixtures (Figure 8-20): the model invents many elongated components along the arcs. Density for anomalies may remain usable; cluster interpretation will not.
Isolation forest, LOF, one-class SVM
Isolation forest chops space with random splits; rare points isolate in few chops; mean path length scores anomalies. Excellent on high-D fraud vectors. LOF compares local density to neighbour densities, good when normal density varies by region. One-class SVM finds a small region covering training mass in kernel space; prefer for novelty on cleaner sets; scale inputs; tune nu and gamma.
Bank and EA extended
Card fraud: IsolationForest on account-day aggregates; GMM on login embeddings; DBSCAN on device-geo trails. Customer value: MiniBatchKMeans segments with silhouette plus human names such as dormant high-balance. EA: cluster services; IF on deploy metric vectors; active learning on incident text clusters.
A closer look
Cluster IDs are unstable. Silhouette is not a business KPI. GMM responsibilities are not segment membership probabilities for marketing automation. `contamination` in IF is a prior. k-means on TF-IDF without understanding sparsity burns cycles; prefer spherical variants or embeddings first.
Exercise answer sketches
k-means versus GMM on ellipses: GMM wins with full Sigma. Elbow can under-estimate versus silhouette. DBSCAN separates moons where k-means fails. Label propagation beats random labels at low budgets. BIC minimum aligns with true k on blob toys. Bayesian weights collapse unused components. IF versus LOF: IF scales higher-D; LOF catches local outliers.
Likelihood and singularities
EM maximises the observed-data log-likelihood of the mixture. If a component collapses onto a single point, variance goes to zero and likelihood goes to infinity: a useless singularity. `reg_covar` adds a ridge to covariances. Multiple random inits plus a cap on `max_iter` keep training sane. When log-likelihood barely moves across inits, you are in a stable basin; when it jumps, keep the best and move on.
Soft assignments as features
GMM `predict_proba` columns are rich unsupervised features for a downstream supervised model. So are k-means distances from `transform`. Stack them beside original features inside a `FeatureUnion` or column-wise concatenate. This pattern is the chapter feature-engineering bullet made concrete, and it often helps when labelled rows are few.
Customer segmentation workflow
1. Choose behaviour and value features; winsorise extremes. 2. Scale. 3. MiniBatchKMeans over a k grid with silhouette. 4. Name segments with SMEs. 5. Stability-check across monthly refits via centroid matching. 6. Attach segment to the feature store with a version. 7. Never trigger customer actions from a fresh unstable clustering without a hold-out policy review.
Density estimation view
Beyond clustering, GMM is a generative density model. You can sample new synthetic rows with `gm.sample`. That helps rare-event simulations when a full deep generative model is overkill. Compare histograms of real versus sampled margins before trusting the samples for stress tests.
Search and image bullets from the intro
Clustering groups similar documents or images for search and navigation. Classic systems used k-means on TF-IDF or on CNN embeddings. In 2026 many stacks cluster embedding vectors from foundation models; the sklearn algorithms still apply on top of those vectors. Image segmentation via colour k-means remains a teaching tool; production segmentation is neural (Chapter 12).
Spectral clustering sketch
Build a similarity graph (RBF or kNN). Compute the Laplacian eigenvectors (the spectral embedding). Run k-means in that embedding. Moons and circles separate when the graph captures the manifold. Cost grows with graph construction; approximate neighbours keep it feasible.
Agglomerative linkage choices
Ward minimises within-cluster variance (blob-friendly). Complete linkage looks at farthest pairs; single linkage at nearest (can chain). Average linkage sits between. Dendrograms cut at a height or at a target cluster count. Connectivity matrices stop merges across known barriers (e.g., rivers on a map).
Mean-shift and affinity propagation notes
Mean-shift bandwidth is the main knob; too small invents tiny clusters; too large merges everything. Affinity propagation damping and preference control how many exemplars appear. Both are useful on medium data and finicky on large noisy streams.
Isolation forest scoring detail
Average path length across trees is compared to the expected path length in an unbalanced binary tree; scores near 1 are more anomalous, near 0 more normal (sklearn decision_function sign conventions differ from some papers: always check the docs for the version you pin). Set `contamination` to the expected outlier fraction if you need a hard predict; or ignore it and threshold `decision_function` on a validation audit set.
One-class SVM geometry
In kernel space the algorithm separates mapped points from the origin with a maximum-margin hyperplane. Back in input space that becomes a contour around the data mass. `nu` is both an upper bound on the fraction of training errors and a lower bound on the fraction of support vectors. Too-small gamma underfits the contour; too-large gamma islands every point.
Validation without labels
Internal metrics: silhouette, Calinski-Harabasz, Davies-Bouldin. External metrics when you have a proxy label: adjusted rand index, mutual information. Business metrics beat all of them: lift in campaign response by segment, fraud precision at fixed review budget, mean time to detect incidents.
Pitfalls expanded
1. k-means on unscaled features. 2. Picking k by inertia alone. 3. DBSCAN eps from thin air. 4. GMM anomalies with heavy contamination. 5. Treating cluster IDs as stable across refits. 6. One-class SVM without scaling. 7. Expecting GMM to find moons. 8. Shipping segments without business names. 9. Using t-SNE clusters as if they were k-means segments. 10. Fitting IF on data that already dropped the fraud you hope to find.
Test yourself
Inertia definition? Why elbow? Silhouette range? k-means++ idea? DBSCAN roles of eps and min_samples? GMM generative story? Anomaly versus novelty? IF isolation idea? When Bayesian GMM? Linkage choices?
Figure inventory for Chapter 8
Figure 8-1 labelled versus unlabelled Iris. 8-2 five blobs. 8-3 through 8-5 centroid journeys and bad inits. 8-6 wrong k. 8-7 elbow. 8-8 and 8-9 silhouette diagrams. 8-10 GMM ellipsoids. Later figures cover DBSCAN moons, GMM densities, anomaly stars (8-17), Bayesian weight collapse, and failed moons GMM (8-20).
Hard versus soft clustering
k-means and DBSCAN (except noise) emit hard labels. GMM emits soft responsibilities that sum to one across components. Soft outputs are better when points sit between modes, such as customers who belong partly to two personas. Harden with `predict` only at the last mile when a single segment ID is required for a downstream rule.
Distance feature pipeline example
```python from sklearn.cluster import KMeans from sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler pipe = make_pipeline(StandardScaler(), KMeans(n_clusters=8, n_init=10), LogisticRegression(max_iter=1000)) # Note: KMeans.transform provides distances if you build a custom transformer; # default pipeline would pass labels. Prefer a dedicated DistanceToCentroid transformer in production. ``` The pedagogical point: unsupervised geometry can feed supervised learners when labels are scarce.
Mini-batch convergence ops
Watch `inertia_` across partial fits. If it oscillates without trend, raise `batch_size` or lower the learning-rate-like stepsize knobs your version exposes.
Silhouette failure modes
Silhouette assumes roughly convex clusters and comparable densities. On moons it can prefer wrong k. Pair it with a domain plot and, when labels exist for a slice, adjusted rand index. Never automate k selection from silhouette alone inside a regulated credit segmenter.
DBSCAN on geo data
Use haversine distances or project to local metres before Euclidean DBSCAN. `eps` in degrees is a common foot-gun. For ATM abuse rings, density in space-time may need a joint feature (x, y, minutes) with careful scaling so time is not ignored.
GMM sampling for stress tests
`X_new, y_new = gm.sample(10000)` builds synthetic traffic. Review marginals and pairwise correlations against real data. For fraud, do not treat samples as true fraud; treat them as stress on the normal model.
Isolation forest versus rules
Rules catch known typologies; IF catches novel sparse combinations. Run both. When IF flags a case, demand feature contributions (path-based or SHAP on a surrogate) before case managers trust the queue.
Novelty detection deployment
Train OneClassSVM or IF on a vetted clean window. Freeze. Score new days. When the world shifts (new product, new geo), rebuild the clean window or you will flag the innovation as novel forever. Document the rebuild trigger in the model card.
Semi-supervised accuracy curve
Plot accuracy versus number of labelled points for random labels versus cluster-based labels. The gap is the value of unsupervised structure. Show that curve to budget holders who decide annotation spend.
Active learning loop
1. Fit model on labelled set. 2. Score unlabelled pool by uncertainty (margin, entropy) or by density times uncertainty. 3. Send top-q to annotators. 4. Refit. 5. Stop when the validation curve flattens or the budget ends. Clustering can diversify each batch so you do not query five near-duplicates.
End-to-end lab
Blobs: recover k with silhouette. Moons: DBSCAN versus k-means. Ellipses: GMM full versus spherical. Contaminated Gaussian: IF versus GMM percentile. Label budget: compare random versus centroid labelling. Log metrics in one table; ship the simplest winner.
Covariance estimation sample size
Full covariance in d dimensions has d(d+1)/2 free parameters per component. With small m and large d, prefer diag or tied, or reduce dimensions first (Chapter 7). This is a common reason GMM fails on wide tabular fraud features without a PCA front-end.
Label switching
EM can swap component indices across runs. When comparing models or tracking a segment over time, align components by mean proximity or by maximising overlap of responsibilities. Store a canonical order by descending weight after each fit.
DBSCAN memory
Naive neighbour search is costly. sklearn can use ball trees or kd-trees for Euclidean metrics. For very large m, sample to tune eps, then fit on shards, or move to approximate HDBSCAN-style systems.
Production segment governance
Version the feature list, scaler, k, seed, and training window. Require a human-readable segment dictionary. Diff segment sizes after each refit; alert if a segment shrinks by more than a set percent. That governance matters more than a 0.02 silhouette gain.
Adjusted rand and mutual information
When a subset of rows carries a proxy label (product line, fraud tag), score clusterings with adjusted rand index and normalised mutual information. These external metrics catch pretty silhouette lies.
Image segmentation colour spaces
Clustering in RGB is simple; HSV or lab space sometimes separates perceptual groups better. Scale channels. For production vision, use Chapter 12 models; keep colour k-means for teaching and crude masks.
Semi-supervised math sketch
After clustering, let S_j be the labelled seed in cluster j. Assign all points in cluster j the label of S_j, or train a classifier on seeds and predict the rest. Error concentrates in mixed clusters; silhouette per cluster predicts which segments are safe to auto-label.
Ch8 deep panel 100
This stretch centers on LeCun. Source cue: LeCun, Turing Award winner and Meta Chief Scientist, famously said that intelligence was cake, unsupervised learning would the cake, supervised.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch8 ideas. Source cue: predict whether item defective not, you will need label every single picture defective normal This will generally require human experts.
If the book shows code here, retype it from memory and diff.
This stretch centers on DBSCAN. Source cue: systems, search engines, image segmentation, semi-supervised learning, dimensionality reduction, and more. Anomaly detection also called outlier detection The objective learn.
If the book shows code here, retype it from memory and diff.
Ch8 deep panel 101
This stretch centers on DBSCAN. Source cue: for density estimation, clustering, and anomaly detection. Clustering Algorithms: k-means and DBSCAN you enjoy hike the mountains, you stumble upon.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch8 ideas. Source cue: class represented with different marker. labeled dataset, for which classification algorithms such logistic regression, SVMs, random forest classifiers are well.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch8 ideas. Source cue: Clustering used wide variety applications, including: Customer segmentation You can cluster your customers based their purchases and their activity your.
If the book shows code here, retype it from memory and diff.
Ch8 deep panel 102
This stretch centers on Ch8 ideas. Source cue: preserve enough information for further processing. Feature engineering The cluster affinities can often useful extra features. For example, used k-means.
If the book shows code here, retype it from memory and diff.
This stretch centers on DBSCAN. Source cue: algorithm all the images your database; similar images would end the same cluster. Then when user provides reference image, all.
If the book shows code here, retype it from memory and diff.
This stretch centers on DBSCAN. Source cue: hierarchical, looking for clusters clusters. And the list goes on. this section, will look two popular clustering algorithms, k-means and.
If the book shows code here, retype it from memory and diff.
Ch8 deep panel 103
This stretch centers on Ch8 ideas. Source cue: sklearn.cluster import KMeans from sklearn.datasets import make blobs make blobs ... make the blobs: contains the cluster IDs, but will.
If the book shows code here, retype it from memory and diff.
Closing
Unsupervised tools find structure without labels. Dimensionality reduction and clustering feed representation learning later.
9. Introduction to Artificial Neural Networks with Keras… and PyTorch Ahead
From biological neurons to the perceptron, then MLPs trained by backpropagation (reverse-mode autodiff). Géron still shows sklearn MLPRegressor/Classifier tables for regression and classification heads, then a practical hyperparameter field guide. Chapter 10 rebuilds the stack in PyTorch.
MLP stack.
From biology to artificial neurons
Study loop.
Biological neurons fire spikes along axons into synapses; rates encode intensity. McCulloch - Pitts (1943) showed networks of binary neurons can compute logical propositions. Artificial neurons take weighted inputs, add bias, apply an activation. Early excitement, AI winters, then deep learning after data + GPUs + better training arrived.
Logical computations. Networks of threshold units can implement AND, OR, NOT when weights and thresholds are set by hand (Figure 9-3 theme). Learning those weights from data is the rest of the story.
The perceptron
Rosenblatt (1957): TLU computes z = w⊺x + b, then Heaviside (or sign) step (Equation 9-1, Figure 9-4). Almost logistic regression, but with a hard step. A perceptron is a dense layer of TLUs (Figure 9-5): multilabel binary outputs possible.
Perceptron / TLU.
Perceptron cycle.
Perceptron learning rule. When a prediction is wrong, push weights toward/away from the input: w ← w + η (y − ŷ) x (and bias similarly). Guaranteed to converge if data are linearly separable (perceptron convergence theorem). XOR kills a single layer (Minsky & Papert) - the historical crisis that MLPs later solved (Figure 9-6).
`sklearn.linear_model.Perceptron` is SGD with a hinge-like path; useful pedagogy, rarely a 2026 champion.
MLPs and backpropagation
Ops contrast.
Stack input → hidden layer(s) → output (Figure 9-7). Feedforward only. Deep stacks = DNNs. XOR falls to a small MLP with nonlinear hidden units.
MLP stack.
Backprop loop.
Training history. Gradients needed for GD; Linnainmaa reverse-mode autodiff (1970) computes all parameter gradients in one forward + one backward pass. Popularised for nets in the 1980s (Rumelhart, Hinton, Williams). Forward: compute activations and loss. Backward: chain rule through each op. Update weights with GD/SGD/Adam.
Activations. Step functions kill gradients. Sigmoid/tanh saturated; ReLU and friends dominate hidden layers. Softmax + cross-entropy for multiclass; sigmoid + BCE for multilabel; linear + MSE/Huber for regression.
Universal approximation, practical depth
Practice hub.
One wide hidden layer can approximate continuous functions on compact sets in theory; deep nets reuse hierarchical features with far fewer parameters. Transfer learning: reuse lower layers (edges → parts → objects). Start with 1 - 2 hidden layers for tabular; go deeper for perception tasks (later chapters).
Building MLPs with Scikit-Learn
Fit discipline.
`MLPRegressor` / `MLPClassifier`: convenient, not the production DL path. Still good for small tabular experiments and for matching the book's tables.
Regression MLP defaults (book table theme). Hidden layers 1 - 5; neurons/layer often 10 - 100; hidden activation ReLU; output activation identity; loss MSE or Huber if outliers.
Classification heads.
| Task | Outputs | Activation | Loss | |
|---|---|---|---|---|
| Binary | 1 | Sigmoid | BCE / x-entropy | Multilabel |
| #labels | Sigmoid | BCE | Multiclass | #classes |
Classification heads.
Hyperparameter guidelines
Error parts.
Width. Too narrow underfits; too wide overfits and slows training. Pyramid or constant width are common; tune.
Learning rate. Most important optimizer knob. Too high diverges; too low crawls. Use schedules or adaptive methods (Adam).
Batch size. Small = noisy gradients, often better generalisation; large = hardware efficiency. Match LR when you scale batch (linear scaling rules with care).
Optimizer. SGD+momentum still strong; Adam/AdamW defaults for many nets. Chapter 11 goes deeper.
Activations. ReLU/GELU/SiLU in hidden layers; task-specific outputs as in the table.
Regularisation. Early stopping, weight decay, dropout (Chapter 11). sklearn MLP exposes `alpha` (L2) and `early_stopping`.
torch.compile, Ampere/Hopper kernels, and modern optimisers. Prefer AdamW + cosine decay over decades-old MLP defaults when you leave sklearn.Backprop mental model
Train vs serve.
1. Forward: cache pre-activations z and activations a. 2. Loss L at the end. 3. Backward: δ = ∂L/∂z for the last layer; propagate delta = (W.T @ delta) * activation_grad(z) for hidden layers. 4. ∂L/∂W = δ a_prev⊺; ∂L/∂b = δ. 5. Step parameters.
Backprop loop.
Appendix A expands autodiff. You need the shape intuition even when PyTorch hides the tape.
Perceptron vs logistic regression
Model promotion.
Same linear score; different nonlinearity and loss. Logistic gives probabilities and cross-entropy gradients everywhere; perceptron step gives no probability and sparse updates. Prefer logistic/softmax unless teaching history.
XOR lesson for architecture
Study loop.
Linear separators fail on XOR. One hidden layer with nonlinear activations creates features whose linear readout unravels XOR. That is the entire pitch for deep feature learning in one cartoon.
sklearn MLP gotchas
- Needs scaled inputs.
- `max_iter` often too low; watch convergence warnings.
- `solver='lbfgs'` for tiny data; `adam`/`sgd` for larger.
- Not GPU-first.
- Reproducibility: `random_state` plus careful version pins.
Hyperparameter search pattern
```python from sklearn.neural_network import MLPClassifier from sklearn.model_selection import RandomizedSearchCV param = { "hidden_layer_sizes": [(64,), (128,), (64,64), (128,64)], "alpha": [1e-4, 1e-3, 1e-2], "learning_rate_init": [1e-3, 3e-3, 1e-2], } ``` Pair with `StandardScaler` in a pipeline. For serious DL, move the search to PyTorch + modern tools (Chapter 10 - 11).
Transfer learning preview
Lower layers learn generic features; upper layers specialise. Freeze lower layers when fine-tuning on small target data. Full treatment arrives with pretrained vision and language models later in the book.
A closer look
Backprop through step functions is impossible; historical perceptrons used different rules.
Exercise themes
Why XOR breaks perceptrons. Sketch backprop for a one-hidden-layer net. Choose output activations for regression/binary/multiclass. Learning rate failure modes. Compare MLPClassifier to HGB on a tabular set.
Biological analogy limits
Spikes, timing codes, and glial cells do not map cleanly onto dense ReLU layers. Use biology as motivation, not as a constraint on architecture search.
Dense layer matrix form
A = f(X W + b) with X shape (batch, in_features), W (in_features, out_features). Broadcasting bias. This is the atom of Chapters 10 - 11.
Loss chooser card
| Problem | Last activation | Loss | |
|---|---|---|---|
| Regression | Identity | MSE / Huber / MAE | Binary |
| Sigmoid | BCE | Multilabel | Sigmoid |
| BCE | Multiclass | Softmax | Cross-entropy |
Early stopping with sklearn
`early_stopping=True`, `validation_fraction=0.1`, `n_iter_no_change=10`. Record best `n_iter_`. Same discipline you will use with PyTorch callbacks later.
Bank vignette
A team tries `MLPClassifier` on credit features after scaling. It ties HGB within noise but trains slower and explains nothing. They keep HGB as champion and reserve PyTorch MLPs for transaction sequences (Chapter 13+).
EA vignette
A three-layer MLP predicts incident severity from numeric service features. It beats logistic regression by 3 points PR-AUC. They stop adding layers when valid gain flattens and export ONNX for the pager service.
Pitfalls
MLP pitfalls.
1. Unscaled inputs into MLP. 2. Softmax + MSE (use cross-entropy). 3. Too-high LR without warmup/schedule. 4. Treating sklearn MLP as a deep learning platform. 5. Ignoring early stopping. 6. Width explosion without regularisation. 7. Expecting perceptrons to solve XOR.
Test yourself
TLU equation? Perceptron update? Why XOR fails? What is reverse-mode autodiff? ReLU vs sigmoid? Output activations by task? Key LR failure modes?
Biological neurons, carefully
A biological neuron receives signals on dendrites, sums influences near the soma, and may emit an action potential down the axon. Synaptic strengths change with learning. Artificial neurons borrow the weighted-sum-plus-nonlinearity cartoon and ignore timing codes, neuromodulators, and energy constraints. That cartoon is enough to build useful engines; it is not a brain model.
McCulloch-Pitts and logic
With binary inputs and carefully chosen weights, threshold neurons implement AND, OR, and NOT. Networks of them implement any boolean function. The historical point: computation is possible. The engineering point: hand-set weights do not scale; learning must set them.
Perceptron learning, worked
For a single TLU, prediction ŷ = step(w·x + b). If y = ŷ, do nothing. If y=1 and ŷ=0, add ηx to w and η to b. If y=0 and ŷ=1, subtract. Linearly separable data admit finite convergence. Nonseparable data cause perpetual churn unless you stop early. Multi-output perceptrons train each TLU independently on its binary target.
Why the step function blocked backprop
Derivative of Heaviside is zero almost everywhere. Gradient descent cannot nudge weights through that flatness. Sigmoid/tanh reopen gradients but saturate at the tails. ReLU passes gradient on the positive side and zeroes it on the negative, a cheap compromise that unlocked deep stacks.
Reverse-mode autodiff in one paragraph
Build a computation graph of elementary ops. Forward: compute values. Backward: each op knows how to multiply an incoming upstream gradient by its local Jacobian-vector product. Sum contributions when a node fans out. Complexity is a small constant times the forward cost, not times the number of parameters. That is why millions of weights are trainable. Appendix A drills the mechanics.
MLP forward equations
h¹ = f¹(W¹x + b¹); h² = f²(W²h¹ + b²); …; ŷ = f_out(W_out h_L + b_out). Choose f_out from the loss table. Vectorise over a batch dimension. This is exactly what `nn.Linear` and activations implement in Chapter 10.
Regression MLP details
Single continuous target: one linear output. Multioutput: one linear unit per target. Huber loss dampens outlier gradients versus MSE. Standardise y sometimes for stability; invert after predict. sklearn `MLPRegressor` exposes `hidden_layer_sizes`, `activation`, `alpha`, `learning_rate_init`, `early_stopping`.
Classification MLP details
Binary: one logit + sigmoid, or two-logit softmax with care. Multilabel: independent sigmoids. Multiclass: softmax over exclusive classes. Never apply softmax and then MSE; use cross-entropy on logits when the framework allows (PyTorch `CrossEntropyLoss` expects logits).
Depth versus width, expanded
A shallow net can approximate many functions but may need exponentially many units.
Learning rate schedules preview
Constant LR works for sklearn toys. Real training uses warmup, cosine decay, or reduce-on-plateau. If loss explodes, cut LR. If loss plateaus early, raise LR or check normalisation. Chapter 11 treats this as a first-class topic.
Batch size and generalisation
Tiny batches regularise via noise but underuse GPUs. Huge batches need larger LR and careful schedules to match small-batch generalisation. Measure valid metrics, not only tokens/sec.
Regularisation starter pack
L2 (`alpha` in sklearn; weight decay in AdamW), early stopping, dropout (Chapter 11), data augmentation (vision/audio later), label smoothing for classification. Pick two; measure.
sklearn versus PyTorch decision
Stay in sklearn when the net is a shallow baseline beside HGB and you want Pipelines. Move to PyTorch when you need GPUs, custom losses, sequence models, or production DL tooling. Do not fight sklearn MLP into a transformer.
Where to implement.
Hyperparameter table memory
Hidden layers: 1-5 tabular. Neurons: 10-100s tabular. Hidden activation: ReLU family. Output: identity / sigmoid / softmax. Loss: MSE-Huber / BCE / x-entropy. LR: 1e-3 order for Adam. Batch: 32-256 common starts. Optimiser: AdamW default in modern PyTorch stacks.
Critical API notes
Some older Géron printings leaned Keras/TF; your edition path is PyTorch. Code names differ; ideas do not. `MLPClassifier` `best_loss_` and early stopping attributes help debugging. Watch for non-convergence warnings and raise `max_iter`.
Exercise sketches
Draw XOR MLP weights that work. Derive ∂L/∂W for a one-layer sigmoid net with BCE. Explain why softmax+MSE is frowned upon. Tune an MLPClassifier versus HistGradientBoostingClassifier on moons or credit; report which wins on PR-AUC and latency.
Bank and EA extras
Bank: neural nets shine on sequential spend and documents; shallow MLPs rarely beat tree boosters on static bureau tables.
Pitfalls
1. Unscaled inputs into MLP. 2. Softmax + MSE. 3. Too-high LR. 4. sklearn MLP as a DL platform. 5. Ignoring early stopping. 6. Width explosion without regularisation. 7. Expecting perceptrons to solve XOR. 8. Confusing multilabel sigmoid with multiclass softmax.
Test yourself
TLU equation? Perceptron update? Why XOR fails? Reverse-mode autodiff cost? ReLU vs sigmoid? Output activations by task? When to leave sklearn for PyTorch?
Heaviside and sign
Equation 9-1 lists Heaviside and sign. Both are discontinuous at 0. Implementations must pick a convention at exactly zero; it rarely matters for real-valued scores that almost never hit zero after training noise.
Dense layer as matrix multiply
Batch X times W plus bias is one GEMM. GPUs exist to crush GEMMs. Chapter 10 will wrap this in `nn.Linear`. Understanding the shapes prevents the majority of beginner bugs.
Initialization preview
Break symmetry: random weights so hidden units learn different features. Modern schemes (He/Xavier) match activation variances. sklearn MLP uses its own defaults; PyTorch modules use theirs. Chapter 11 treats vanishing/exploding gradients.
Momentum intuition
SGD with momentum accumulates a velocity vector so updates keep going through small valleys. Adam maintains per-parameter adaptive rates. For sklearn, `solver='adam'` is the usual pick; for PyTorch, AdamW is the 2026 default in many codebases.
Validation discipline
Split train/valid before any fit. Early-stop on valid loss. Report test once. Neural nets will memorise if you peek test while tuning.
Huber versus MSE
Huber acts like MSE near zero and like MAE in the tails. Use when label noise or heavy tails would let a few residuals dominate the gradient.
Multilabel versus multiclass mistakes
Multilabel: an email can be both "urgent" and "billing". Multiclass: a digit is exactly one of 0-9. Wrong head choice silently ruins metrics.
From biology to perceptrons
Biological neurons accumulate synaptic inputs and fire. Early AI used this metaphor for logical computations with threshold units. The Perceptron (Rosenblatt) computes a weighted sum plus bias, applies a step, and updates with the perceptron rule when it errs.
Perceptrons converge on linearly separable data; they fail on XOR with one layer - the historical crisis that delayed multilayer work until backpropagation matured.
MLPs and backpropagation
Multilayer Perceptron: input layer, one or more hidden layers, output layer. Forward pass computes activations; backpropagation applies the chain rule to get gradients of the loss w.r.t. every weight; an optimiser steps.
Regression MLPs: linear output, MSE/Huber. Classification: sigmoid + BCE for binary; softmax + cross-entropy for multiclass. Hidden activations: historically sigmoid/tanh; modern default ReLU and variants (Chapter 11).
Hyperparameter guidelines
Batch size, activation, init, regularisation, and optimiser interact - tune systematically (Optuna in Chapter 10).
Figure and exercise themes
Figures 9-1… cover biological neuron, perceptron, XOR, MLP diagrams, and training curves.
Bank and EA takes
Bank: MLP on engineered bureau features as a challenger to logistic scorecards; calibrate probabilities. EA: small MLP on packed numeric SLIs before jumping to transformers for text.
Perceptron math
The perceptron computes a hard threshold of a weighted sum plus bias. The update adds or subtracts the input vector when the prediction is wrong, scaled by a learning rate in some variants. Linear separability guarantees convergence for the classic rule. XOR is not linearly separable, so a single layer fails; multilayer nets with nonlinearities succeed.
Backprop chain rule
Forward computes activations layer by layer. Loss compares outputs to targets. Backward applies the chain rule: each layer receives a gradient from above, multiplies by local Jacobian, and passes a gradient below. Weight updates use outer products of inputs and local deltas in the fully connected case. Modern libraries hide the Jacobians; you still need the story.
Activation zoo preview
Sigmoid and tanh saturate and can starve deep gradients. ReLU is piecewise linear, cheap, and sparsifies activations, with a dead-ReLU risk when units output zero forever. Chapter 11 covers ELU, SELU, GELU, Swish, SwiGLU, and Mish. For this chapter, ReLU plus careful init is enough to train shallow MLPs.
Hyperparameter playbook
Sweep learning rate first on a log grid. Then width. Then regularisation. Change one family at a time when you are learning; use Optuna when you are shipping. Batch size interacts with learning rate; linear scaling rules are a starting heuristic, not a law.
Document every run with config and seed. A notebook without configs is an anecdote, not an experiment.
Ch9 deep panel 100
This stretch centers on YouTube, DeepMind, AlphaFold. Source cue: Networks Birds inspired fly, burdock plants inspired Velcro, and nature has inspired countless more inventions. seems only logical, then, look.
If the book shows code here, retype it from memory and diff.
This stretch centers on YouTube, DeepMind, AlphaFold, PyTorch, McCulloch. Source cue: Claude recommending the best videos watch hundreds millions users every day e.g., YouTube learning how proteins fold DeepMind AlphaFold This.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch9 ideas. Source cue: might work together animal brains perform complex computations using propositional logic. This was the first artificial neural network architecture. Since.
If the book shows code here, retype it from memory and diff.
Ch9 deep panel 101
This stretch centers on GPU. Source cue: the previous ones did Well, here are few good reasons believe that this time different and that the renewed interest.
If the book shows code here, retype it from memory and diff.
This stretch centers on Transformer. Source cue: this power accessible everyone. The training algorithms have been improved. fair they are only slightly different from the ones used.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch9 ideas. Source cue: many different tasks, possibly with bit fine-tuning that transfer learning just prompting the model the right way that in-context learning,.
If the book shows code here, retype it from memory and diff.
Ch9 deep panel 102
This stretch centers on Ch9 ideas. Source cue: cell mostly found animal brains. composed cell body containing the nucleus and most the cell complex components, many branching extensions.
If the book shows code here, retype it from memory and diff.
This stretch centers on McCulloch. Source cue: Figure 9-1. biological neuron Thus, individual biological neurons seem behave simple way, but they organized vast network billions, with each.
If the book shows code here, retype it from memory and diff.
This stretch centers on McCulloch. Source cue: inputs are active. their paper, McCulloch and Pitts showed that even with such simplified model possible build network artificial neurons.
If the book shows code here, retype it from memory and diff.
Ch9 deep panel 103
This stretch centers on Ch9 ideas. Source cue: activated either neuron neuron activated both Finally, suppose that input connection can inhibit the neuron activity which the case with.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch9 ideas. Source cue: with weight. The TLU first computes linear function its inputs: Then applies step function the result: step almost like logistic.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch9 ideas. Source cue: classification. computes linear function its inputs, and the result exceeds threshold, outputs the positive class. Otherwise, outputs the negative class..
If the book shows code here, retype it from memory and diff.
Ch9 deep panel 104
This stretch centers on Ch9 ideas. Source cue: inputs and three outputs represented Figure 9-5. Architecture perceptron with two inputs and three.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch9 ideas. Source cue: the activation function: when the artificial neurons are TLUs, step function will discuss other activation functions shortly Note mathematics, the.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch9 ideas. Source cue: the catchy phrase, Cells that fire together, wire together that is, the connection weight between two neurons tends increase when.
If the book shows code here, retype it from memory and diff.
Ch9 deep panel 105
This stretch centers on Ch9 ideas. Source cue: current training instance. the output the jth output neuron for the current training instance. the target output the jth output.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch9 ideas. Source cue: may have noticed that the perceptron learning algorithm strongly resembles stochastic gradient descent introduced Chapter fact, Scikit-Learn Perceptron class equivalent.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch9 ideas. Source cue: classifiers but researchers had expected much more from perceptrons, and some were disappointed that they dropped neural networks altogether favor.
If the book shows code here, retype it from memory and diff.
Ch9 deep panel 106
This stretch centers on Ch9 ideas. Source cue: input layer are usually called the lower layers, and the ones close the outputs are usually called the upper layers..
If the book shows code here, retype it from memory and diff.
Closing
Chapter 9 gives the conceptual MLP; Chapter 10 puts it on PyTorch rails.
10. Building Neural Networks with PyTorch
Tensors and autograd first, then the full training loop: linear regression by hand, a regression MLP, DataLoader mini-batches, evaluation, custom nn.Module graphs, a TorchVision classifier, Optuna, save/load, and torch.compile.
Compile discipline.
Optuna trial flow.
Ch10 panel 1
Study loop.
This stretch centers on PyTorch, NumPy, TensorFlow. Source cue: Building Neural Networks with PyTorch PyTorch powerful open source deep learning library developed Facebook Research lab FAIR, now called Meta.
If the book shows code here, retype it from memory and diff.
Save formats.
This stretch centers on PyTorch, autograd, Optuna. Source cue: known define-by-run making intuitive and easy debug. PyTorch also beautifully coded and documented, and focuses its core task: making easy.
If the book shows code here, retype it from memory and diff.
This stretch centers on Optuna, PyTorch, GPU, NumPy. Source cue: model multilayer neural network, first for regression, then for classification. Along the way, will see how build custom neural networks.
If the book shows code here, retype it from memory and diff.
Ch10 panel 2
This stretch centers on NumPy, PyTorch. Source cue: now will input and output tensors much like Scikit-Learn models input and output NumPy arrays let start looking how create.
If the book shows code here, retype it from memory and diff.
This stretch centers on NumPy, PyTorch. Source cue: .
If the book shows code here, retype it from memory and diff.
This stretch centers on NumPy, PyTorch, FloatTensor. Source cue: numpy method, and create tensor from NumPy array:.
If the book shows code here, retype it from memory and diff.
Ch10 panel 3
Ops contrast.
This stretch centers on NumPy, PyTorch, ReLU, GPU. Source cue: data directly, without copying it. But beware: modifying the NumPy array will also modify the tensor, and vice versa. You.
If the book shows code here, retype it from memory and diff.
This stretch centers on PyTorch, GPU, CUDA. Source cue: much faster. Hardware Acceleration PyTorch tensors can copied easily the GPU, assuming your machine has compatible GPU, and you have.
If the book shows code here, retype it from memory and diff.
This stretch centers on GPU, PyTorch. Source cue: AMD Instinct accelerators and AMD Radeon GPUs, through the ROCm software stack, via DirectML Windows. Intel GPUs and CPUs Linux.
If the book shows code here, retype it from memory and diff.
Ch10 panel 4
Practice hub.
This stretch centers on GPU. Source cue: the CPU, then copy the GPU using the method:.
If the book shows code here, retype it from memory and diff.
This stretch centers on GPU. Source cue: lives the GPU. This means can perform multiple operations the GPU without having transfer data back and forth between the.
If the book shows code here, retype it from memory and diff.
This stretch centers on GPU, PyTorch. Source cue: .
If the book shows code here, retype it from memory and diff.
Ch10 panel 5
Fit discipline.
This stretch centers on GPU, PyTorch, autograd. Source cue: now that seen what tensors are and how use them the CPU the GPU, let look PyTorch auto-differentiation feature. Autograd.
Autograd cycle.
If the book shows code here, retype it from memory and diff.
This stretch centers on PyTorch. Source cue: constant specifying requires grad True. Knowing this, PyTorch will automatically keep track all operations involving this needed because PyTorch must.
If the book shows code here, retype it from memory and diff.
This stretch centers on PyTorch, TensorFlow. Source cue: computed during backprop: this gives the derivative with regard Ta-da PyTorch creates new computation graph the fly during each forward.
If the book shows code here, retype it from memory and diff.
Ch10 panel 6
Error parts.
This stretch centers on PyTorch. Source cue: running gradient descent will gradually push toward since that the value that minimizes x2. gradient descent step, you must temporarily.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch10 ideas. Source cue: also modifies The detach method can handy when you need run some computation tensor without affecting the gradients e.g., for.
If the book shows code here, retype it from memory and diff.
This stretch centers on autograd, PyTorch, RuntimeError. Source cue: 0.1 torch.tensor 5.0, requires grad True for iteration range 100 forward pass f.backward backward pass with torch.no grad learning rate.
If the book shows code here, retype it from memory and diff.
Ch10 panel 7
Train vs serve.
This stretch centers on RuntimeError. Source cue: complaining that one the variables needed for gradient computation has been modified in-place operation Indeed, the intermediate result t.exp was.
If the book shows code here, retype it from memory and diff.
This stretch centers on RuntimeError. Source cue: backward pass. This means that you must not modify such operation output place, you will get error during the backward.
If the book shows code here, retype it from memory and diff.
This stretch centers on RuntimeError, PyTorch, GPU, autograd, StandardScaler. Source cue: counterparts. Just make sure that your model still outputs the same result for given input, and also make sure you.
If the book shows code here, retype it from memory and diff.
Ch10 panel 8
Model promotion.
This stretch centers on StandardScaler, FloatTensor, NumPy. Source cue: and train validation set valid and valid and test set test and test using sklearn.model selection.train test split Next, let.
If the book shows code here, retype it from memory and diff.
This stretch centers on PyTorch, GPU. Source cue: parameters our linear regression model: torch.manual seed features train.shape there are input features torch.randn features, requires grad True torch.tensor 0.,.
If the book shows code here, retype it from memory and diff.
This stretch centers on GPU, PyTorch. Source cue: GPU splits each operation into multiple chunks and runs them parallel, the order which these chunks finish may vary across.
If the book shows code here, retype it from memory and diff.
Ch10 panel 9
Study loop.
This stretch centers on autograd. Source cue: learning rate b.grad learning rate w.grad b.grad.zero w.grad.zero print Epoch epoch epochs Loss: loss.item Let walk through this code: First.
If the book shows code here, retype it from memory and diff.
This stretch centers on PyTorch. Source cue: epoch number and the current loss each epoch. The item method extracts the value scalar tensor. And that it; you.
If the book shows code here, retype it from memory and diff.
This stretch centers on PyTorch. Source cue: low-level API wasn too hard, but using this approach for more complex models would get really messy and difficult. PyTorch.
If the book shows code here, retype it from memory and diff.
Ch10 panel 10
This stretch centers on Ch10 ideas. Source cue: and weight matrix with one row per neuron and one column per input dimension, which the transpose the weight matrix.
If the book shows code here, retype it from memory and diff.
This stretch centers on PyTorch, autograd. Source cue: with requires grad True. That the main difference between regular tensor and Parameter:.
If the book shows code here, retype it from memory and diff.
This stretch centers on SGD, PyTorch, MSE. Source cue: method, will called automatically every time the module itself called. This particularly handy for logging debugging. remove hook, just call.
If the book shows code here, retype it from memory and diff.
Ch10 panel 11
Ops contrast.
This stretch centers on MSE, autograd, PyTorch. Source cue: this also module, can use like function, giving the predictions and the targets, and will compute the MSE. The module.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch10 ideas. Source cue: the optimizer.zero grad line corresponds the two lines that zeroed out b.grad and w.grad. Notice that don need use with.
If the book shows code here, retype it from memory and diff.
This stretch centers on PyTorch, Sequential, ReLU. Source cue: .
If the book shows code here, retype it from memory and diff.
Ch10 panel 12
Practice hub.
This stretch centers on PyTorch, Sequential, ReLU. Source cue: that: simple stack modules two hidden layers and one output layer. let build using the nn.Sequential module: torch.manual seed model.
If the book shows code here, retype it from memory and diff.
This stretch centers on ReLU, SGD, PyTorch, DataLoaders, DataLoader. Source cue: clear that the output one layer must match the input the next layer. Then again, nn.ReLU module implement the second.
Data path to step.
If the book shows code here, retype it from memory and diff.
This stretch centers on DataLoaders, PyTorch, DataLoader, TensorDataset. Source cue: switch mini-batch gradient descent. Implementing Mini-Batch Gradient Descent Using DataLoaders help implement mini-batch GD, PyTorch provides class named DataLoader the.
If the book shows code here, retype it from memory and diff.
Ch10 panel 13
Fit discipline.
This stretch centers on TensorDataset, DataLoader, GPU, Sequential. Source cue: DataLoader train dataset TensorDataset train, train train loader DataLoader train dataset, batch size 32, shuffle True Now that have larger.
If the book shows code here, retype it from memory and diff.
This stretch centers on GPU, Dropout, PyTorch. Source cue: moved the model the GPU. Now let create train function implement mini-batch GD: def train model, optimizer, criterion, train loader,.
If the book shows code here, retype it from memory and diff.
This stretch centers on PyTorch, GPU, TensorFlow, CUDA. Source cue: mode running model.eval Note that model.training holds boolean that indicates the current mode. Tip PyTorch itself does not provide training.
If the book shows code here, retype it from memory and diff.
Ch10 panel 14
Error parts.
This stretch centers on CUDA, GPU. Source cue: noticed that each epoch was much slower. There are two easy tweaks you can make considerably speed training: you are.
If the book shows code here, retype it from memory and diff.
This stretch centers on GPU, PyTorch, autograd, DataLoader. Source cue: GPU still working the current batch. For this, set the data loader num workers argument the number processes you want.
If the book shows code here, retype it from memory and diff.
This stretch centers on DataLoader, TensorDataset, MSE, RMSE, PyTorch. Source cue: DataLoader for the dataset that want evaluate the model on, well function compute the metric for given batch, and lastly.
If the book shows code here, retype it from memory and diff.
Ch10 panel 15
Train vs serve.
This stretch centers on RMSE, MSE, TorchMetrics, PyTorch. Source cue: .
If the book shows code here, retype it from memory and diff.
This stretch centers on TorchMetrics, RMSE, GPU, MeanSquaredError. Source cue: updated one batch time. The TorchMetrics library not preinstalled Colab, have run pip install torchmetrics, then can implement the evaluate.
If the book shows code here, retype it from memory and diff.
This stretch centers on TensorBoard, PyTorch, ReLU. Source cue: probably overfitting the training set, there bug, such data mismatch between the training set and the validation set. This easier.
If the book shows code here, retype it from memory and diff.
Ch10 panel 16
Model promotion.
This stretch centers on WideAndDeep, Sequential, ReLU. Source cue: the output layer, shown Figure 10-1. This architecture makes possible for the neural network learn both deep patterns using the.
If the book shows code here, retype it from memory and diff.
This stretch centers on WideAndDeep, Sequential, ReLU, ModuleList, ModuleDict. Source cue: init self, features super init self.deep stack nn.Sequential nn.Linear features, nn.ReLU nn.Linear 50, nn.ReLU self.output layer nn.Linear features, def forward.
If the book shows code here, retype it from memory and diff.
This stretch centers on ModuleList, ModuleDict, ParameterList, ParameterDict, GPU. Source cue: has variable number submodules, you should store them nn.ModuleList nn.ModuleDict, which are returned the children and named children methods opposed.
If the book shows code here, retype it from memory and diff.
Ch10 panel 17
Study loop.
This stretch centers on TensorDataset, DataLoader. Source cue: ... same constructor earlier, except with adjusted input sizes def forward self, wide deep deep output self.deep stack deep wide.
If the book shows code here, retype it from memory and diff.
This stretch centers on TensorDataset, DataLoader. Source cue: train train train train loader DataLoader train data wd, batch size 32, shuffle True ... same for the validation set.
If the book shows code here, retype it from memory and diff.
This stretch centers on WideAndDeepDataset, DataLoader. Source cue: the inputs, which can lead hard-to-debug issues. avoid this, can good idea name each input. For this, you can define.
If the book shows code here, retype it from memory and diff.
Ch10 panel 18
This stretch centers on Ch10 ideas. Source cue: unpack all the tensors the inputs dictionary and pass them named arguments the model: pred model inputs Now that you.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch10 ideas. Source cue: the person facial expression smiling, surprised, etc. and another output identify whether they are wearing glasses not. Another use case.
If the book shows code here, retype it from memory and diff.
This stretch centers on Ch10 ideas. Source cue: and return both the main output and the auxiliary output: class WideAndDeepV4 nn.Module def init self, features ... same earlier.
If the book shows code here, retype it from memory and diff.
Ch10 panel 19
This stretch centers on PyTorch. Source cue: them into single loss that will minimized gradient descent. general, this final loss just weighted sum all the output losses..
If the book shows code here, retype it from memory and diff.
This stretch centers on PyTorch, TorchVision, ToImage, ToDtype. Source cue: have only tackled regression task, let now turn classification. Building Image Classifier with PyTorch Chapter will tackle the Fashion MNIST.
If the book shows code here, retype it from memory and diff.
This stretch centers on PyTorch, ToImage, ToDtype, TorchVision, NumPy. Source cue: from the training set for validation, using PyTorch random split function: import torchvision import torchvision.transforms.v2 toTensor T.Compose T.ToImage T.ToDtype torch.float32,.
If the book shows code here, retype it from memory and diff.
Ch10 panel 20
This stretch centers on ToImage, ToDtype, NumPy, TorchVision, DataLoader. Source cue: formats including PIL images, NumPy arrays, and tensors TorchVision Image class, which subclass Tensor. The ToDtype transform converts the data.
If the book shows code here, retype it from memory and diff.
This stretch centers on DataLoader, PyTorch, TensorFlow, ToImage. Source cue: data loaders: train loader DataLoader train data, batch size 32, shuffle True valid loader DataLoader valid data, batch size test.
If the book shows code here, retype it from memory and diff.
This stretch centers on ToImage, ImageClassifier, Sequential, ReLU, CrossEntropyLoss. Source cue: right place, depending the library you are using. ToImage already took care moving the channel dimension the first position, otherwise.
If the book shows code here, retype it from memory and diff.
Ch10 panel 21
This stretch centers on Sequential, ReLU, CrossEntropyLoss. Source cue: module. did not have create custom module; could have written model nn.Sequential ... instead, but generally preferable wrap your models.
If the book shows code here, retype it from memory and diff.
torch.compile, AdamW, AMP, transforms v2, weights_only=True.Canonical loop
Place model, create optimizer after `.to`, train mode, zero_grad, forward, loss, backward, step, eval under no_grad.
Pitfalls
CPU/CUDA mismatch; optimizer before to; forgotten zero_grad; Softmax+CE; metrics in train mode; whole pickle; test leakage; in-place on tracked tensors; BN batch size 1; compile without measuring.
Test yourself
Autograd storage; disable grads; zero_grad; optimizer after to; DataLoader flags; CE vs BCEWithLogits; train vs eval; state_dict; trace vs script; compile.
Closing
Chapter 10 is the PyTorch operating system for the rest of the book. Chapter 11 stabilises deep training.
PyTorch stack at a glance.
11. Training Deep Neural Networks
Vanishing/exploding gradients, Glorot/He init, better activations, Batch/Layer Norm, clipping, transfer learning, faster optimisers, LR schedules, and regularisation including MC Dropout.
Norm choice.
Vanishing grads path.
Ch11 panel 1
Study loop.
This stretch centers on PyTorch. Source cue: Deep Neural Networks Chapter you built, trained, and fine-tuned several artificial neural networks using PyTorch. But they were.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: there are not enough training instances they are too noisy. this chapter will through each these problems and.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: the backpropagation algorithm second phase works going from the output layer the input layer, propagating the error gradient.
Retype any code from memory and diff.
Ch11 panel 2
This stretch centers on Glorot. Source cue: ago, and was one the reasons deep neural networks were mostly abandoned the early 2000s. wasn clear what.
Retype any code from memory and diff.
This stretch centers on Glorot. Source cue: slightly better than the sigmoid function deep networks Looking the sigmoid activation function see Figure 11-1 you can.
Retype any code from memory and diff.
This stretch centers on Glorot, LeCun. Source cue: flow properly, the authors argued that need the variance the outputs each layer equal the variance its inputs,.
Retype any code from memory and diff.
Ch11 panel 3
Ops contrast.
This stretch centers on LeCun, Glorot, ReLU, SELU. Source cue: 3fanavg you replace fanavg with fanin Equation 11-1, you get initialization strategy that Yann LeCun proposed the 1990s..
Retype any code from memory and diff.
This stretch centers on SELU, LeCun, Glorot, ReLU, GELU. Source cue: For SELU, use Yann LeCun initialization method, preferably with normal distribution. will cover all these activation functions shortly..
Retype any code from memory and diff.
This stretch centers on LeCun, Sequential, ReLU. Source cue: for LeCun init torch.zero layer.bias.data This works, but clearer and less error-prone use one the initialization functions available.
Retype any code from memory and diff.
Ch11 panel 4
Practice hub.
This stretch centers on Ch11 ideas. Source cue: orthogonal function which initializes the weights using random orthogonal matrix, proposed 2014 paper Andrew Saxe al. Orthogonal matrices.
Retype any code from memory and diff.
This stretch centers on Glorot, ReLU. Source cue: layer during initialization e.g., factor Indeed, this will result smaller logits the beginning training, which means they will.
Retype any code from memory and diff.
This stretch centers on ReLU. Source cue: activation function, mostly because does not saturate for positive values, and also because very fast compute. Unfortunately, the.
Retype any code from memory and diff.
Ch11 panel 5
Fit discipline.
This stretch centers on ReLU. Source cue: activation function defined LeakyReLU max see Figure 11-2 The hyperparameter defines how much the function leaks the slope.
Retype any code from memory and diff.
This stretch centers on ReLU, PyTorch, Sequential. Source cue: ReLU PReLU where authorized learned during training: instead being hyperparameter, becomes parameter that can modified backpropagation like any.
Retype any code from memory and diff.
This stretch centers on ReLU, SELU. Source cue: fact that they are not smooth functions: their slopes abruptly change saw Chapter when discussed lasso, this sort.
Retype any code from memory and diff.
Ch11 panel 6
Error parts.
This stretch centers on PyTorch, ReLU, SELU. Source cue: allows the unit have average output closer and helps alleviate the vanishing gradients problem. The hyperparameter defines the.
Retype any code from memory and diff.
This stretch centers on ReLU, SELU, PyTorch, LeCun. Source cue: computation, but still, test time ELU network will bit slower than ReLU network.
Retype any code from memory and diff.
This stretch centers on LeCun, SELU, GELU, Swish, SwiGLU. Source cue: justification The input features must standardized: mean and standard deviation Every hidden layer weights must initialized using LeCun.
Retype any code from memory and diff.
Ch11 panel 7
Train vs serve.
This stretch centers on ReLU, GELU. Source cue: definition given Equation 11-3, where the standard Gaussian cumulative distribution function CDF corresponds the probability that value sampled.
Retype any code from memory and diff.
This stretch centers on GELU, Swish, Mish. Source cue: complex patterns. practice, often outperforms every other activation function discussed far. However, bit more computationally intensive, and the.
Retype any code from memory and diff.
This stretch centers on Swish, GELU, SwiGLU. Source cue: sigmoid function input. The generalized Swish function Swish GELU approximately equal the generalized Swish function using 1.702. You.
Retype any code from memory and diff.
Ch11 panel 8
Model promotion.
This stretch centers on SwiGLU, GELU, Mish, Swish, ReLU. Source cue: model more expressive power, allowing learn when turn off i.e., multiply amplify specific features: this called gating mechanism..
Retype any code from memory and diff.
This stretch centers on ReLU, Swish, Mish, SwiGLU, PyTorch. Source cue: activation function: ReLU2. its name suggests, simply ReLU squared: ReLU2 max has all the qualities ReLU simplicity, computational.
Retype any code from memory and diff.
This stretch centers on ReLU, PyTorch, GELU, Mish, Swish. Source cue: runtime latency, then you may prefer leaky ReLU, parametrized leaky ReLU for complex tasks, even ReLU2, especially for.
Retype any code from memory and diff.
Ch11 panel 9
Study loop.
This stretch centers on StandardScaler. Source cue: beginning training, doesn guarantee that they won come back during training. 2015 paper, Sergey Ioffe and Christian Szegedy.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: inputs, the algorithm needs estimate each input mean and standard deviation. does evaluating the mean and standard deviation.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: standard deviation per input the vector zero-centered and normalized inputs for instance tiny number that avoids division zero.
Retype any code from memory and diff.
Ch11 panel 10
This stretch centers on PyTorch. Source cue: deviation. Moreover, even have batch instances, may too small, the instances may not independent and identically distributed, computing.
Retype any code from memory and diff.
This stretch centers on ImageNet. Source cue: backpropagation, and the final input mean vector and the final input variance vector are estimated using exponential moving.
Retype any code from memory and diff.
This stretch centers on ImageNet. Source cue: same accuracy with times fewer training steps, and beats the original model significant margin. Using ensemble batch-normalized networks,.
Retype any code from memory and diff.
Ch11 panel 11
Ops contrast.
This stretch centers on PyTorch. Source cue: layer computes then the layer will compute ignoring the smoothing term the denominator define and the equation simplifies.
Retype any code from memory and diff.
This stretch centers on PyTorch, Sequential, ReLU. Source cue: with most things with PyTorch, implementing batch norm straightforward and intuitive. Just add nn.BatchNorm1d layer before after each.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: differently during training and during evaluation, critical switch training mode during training using model.train and switch evaluation mode.
Retype any code from memory and diff.
Ch11 panel 12
Practice hub.
This stretch centers on Sequential, ReLU. Source cue: you move the layers before the activation functions, you can also remove the bias term from the previous.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: good momentum value typically close for example, 0.01 0.001. You want more for smaller mini-batches, and fewer for.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: nn.Flatten layer. However, you could use nn.BatchNorm2d layer before the nn.Flatten layer: indeed, expects its inputs image batches.
Retype any code from memory and diff.
Ch11 panel 13
Fit discipline.
This stretch centers on Ch11 ideas. Source cue: For example, suppose you work particle physics and you have dataset particle trajectories, where each trajectory composed sequence.
Retype any code from memory and diff.
This stretch centers on diffusion, PyTorch, LayerNorm. Source cue: samples batch, which may reduce performance especially for small batch sizes Moreover, struggles with some architectures, such recurrent.
Retype any code from memory and diff.
This stretch centers on PyTorch, LayerNorm. Source cue: training set, like does. Lastly, learns scale and offset parameter for each input feature, just like does. PyTorch.
Retype any code from memory and diff.
Ch11 panel 14
Error parts.
This stretch centers on LayerNorm, PyTorch. Source cue: result shape: 32, 100, 200 However, most computer vision architectures that use normalize over all channels once. For.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: all them model.parameters The clip grad norm function clips each gradient vector norm exceeds the given max norm.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: the result would 0.00899964, 0.9999595 this would preserve the vector orientation, but almost eliminate the first component. The.
Retype any code from memory and diff.
Ch11 panel 15
Train vs serve.
This stretch centers on Ch11 ideas. Reusing.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: that are most useful for the new task may differ significantly from the ones that were most useful.
Retype any code from memory and diff.
This stretch centers on PyTorch, Sequential. Source cue: performance, and you have little training data, try dropping the top hidden layer and freezing all the remaining.
Retype any code from memory and diff.
Ch11 panel 16
Model promotion.
This stretch centers on Sequential, ReLU. Source cue: test accuracy. While drinking your morning coffee, you realize that your task quite similar task perhaps transfer learning.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: since task binary classification. You could start training model for task now, but since the new output layer.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: can unfreeze the reused layers setting param.requires grad True for all parameters reduce the learning rate, and continue.
Retype any code from memory and diff.
Ch11 panel 17
Study loop.
This stretch centers on Transformer. Source cue: not malicious, but part the reason why many results science can never reproduced. But why did cheat turns.
Retype any code from memory and diff.
This stretch centers on diffusion. Source cue: examples, but expensive label them. you can gather plenty unlabeled training data, you can try use train unsupervised.
Retype any code from memory and diff.
This stretch centers on diffusion. Source cue: supervised learning. Unsupervised pretraining today typically using autoencoders diffusion models rather than RBMs still good option when you.
Retype any code from memory and diff.
Ch11 panel 18
This stretch centers on Ch11 ideas. Source cue: training data, one last option train first neural network auxiliary task for which you can easily obtain generate.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: pictures random people from the web, but this would probably illegal. Firstly, photos are usually copyrighted their creators,.
Retype any code from memory and diff.
This stretch centers on Nesterov, AdaGrad, Adam. Source cue: fine-tuned, will see Chapter Note Self-supervised learning when you automatically generate the labels from the data itself, the.
Retype any code from memory and diff.
Ch11 panel 19
Ops contrast.
This stretch centers on Ch11 ideas. Source cue: eventually reaches terminal velocity there some friction air resistance This the core idea behind momentum optimization, proposed Boris.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: this momentum vector see Equation 11-5 other words, the gradient used force learning acceleration, not speed. simulate some.
Retype any code from memory and diff.
This stretch centers on PyTorch, SGD. Source cue: the gradients are not constant, the speedup not always dramatic, but momentum optimization can escape from plateaus much.
Retype any code from memory and diff.
Ch11 panel 20
Practice hub.
This stretch centers on PyTorch, SGD, Nesterov. Source cue: the system: reduces these oscillations and thus speeds convergence. Implementing momentum optimization PyTorch no-brainer: just use the SGD.
Retype any code from memory and diff.
This stretch centers on Nesterov, SGD, AdaGrad. Source cue: vector will pointing the right direction i.e., toward the optimum will slightly more accurate use the gradient measured.
Retype any code from memory and diff.
This stretch centers on SGD, AdaGrad. Source cue: bottom the valley. This helps reduce oscillations and thus NAG converges faster. use NAG, simply set nesterov True.
Retype any code from memory and diff.
Ch11 panel 21
Fit discipline.
This stretch centers on AdaGrad. Source cue: This vectorized form equivalent computing for each element the vector other words, each accumulates the squares the partial.
Retype any code from memory and diff.
This stretch centers on AdaGrad, PyTorch. Source cue: the global optimum see Figure 11-8 One additional benefit that requires much less tuning the learning rate hyperparameter.
Retype any code from memory and diff.
This stretch centers on PyTorch, AdaGrad, Adam. Source cue: using exponential decay the first step see Equation 11-8 Equation 11-8. RMSProp algorithm The decay rate typically set.
Retype any code from memory and diff.
Ch11 panel 22
Error parts.
This stretch centers on Adam. Source cue: RMSProp, keeps track exponentially decaying average past squared gradients see Equation 11-9 These are estimations the mean and.
Retype any code from memory and diff.
This stretch centers on Adam, PyTorch, AdaGrad. Source cue: and corresponds RMSProp. The only difference that step computes exponentially decaying average rather than exponentially decaying sum, but.
Retype any code from memory and diff.
This stretch centers on AdaGrad, Adam, AdaMax, AdamW. Source cue: requires less tuning the learning rate hyperparameter You can often use the default value 0.001, making Adam even.
Retype any code from memory and diff.
Ch11 panel 23
This stretch centers on AdaMax, Adam, Nesterov, AdamW. Source cue: the norm with the norm fancy way saying the max Specifically, replaces step Equation 11-9 with max 2s,.
Retype any code from memory and diff.
This stretch centers on SGD, Adam, AdamW, AdaMax, PyTorch. Source cue: model weights each training iteration multiplying them decay factor such 0.99. This may remind you regularization introduced Chapter.
Retype any code from memory and diff.
This stretch centers on AdaMax, AdamW, PyTorch, Adam. Source cue: replace torch.optim.Adam with torch.optim.NAdam, torch.optim.Adamax, torch.optim.AdamW. For AdamW, you probably want tune the weight decay hyperparameter. All the.
Retype any code from memory and diff.
Ch11 panel 24
This stretch centers on Adam, PyTorch. Source cue: information. One such algorithm Shampoo, which uses accumulated gradient information approximate the second-order terms, similar how Adam accumulates.
Retype any code from memory and diff.
This stretch centers on SGD, Adam, AdaMax, AdamW. Source cue: out many weights can see Lasso Regression You can also try scaling down random weights during initialization encourage.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: rate, you can end with excellent model, but training will generally bit too slow. Luckily, you can better.
Retype any code from memory and diff.
Ch11 panel 25
This stretch centers on SGD. Source cue: epoch, the learning rate will equal the initial learning rate times gamma the power This factor gamma yet.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: remains unchanged scheduler.step Tip you interrupt training and you later want resume where you left off, you should.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: includes the CosineAnnealingLR scheduler, which you can create follows max Tmax and eta min min You can then.
Retype any code from memory and diff.
Ch11 panel 26
This stretch centers on Ch11 ideas. Source cue: information the other hyperparameters mode the tracked metric must maximized such the validation accuracy then you must set.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: each epoch this example, are using the evaluate function that defined Chapter and must then pass the result.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: can then make quick progress using high learning rate. Why does this work Well, the loss surface sometimes.
Retype any code from memory and diff.
Ch11 panel 27
This stretch centers on Ch11 ideas. Source cue: during the third epoch, and 100 after that warmup scheduler torch.optim.lr scheduler.LinearLR optimizer, start factor 0.1, end factor.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: are there any cases where you may want tweak the learning rate the middle training Well yes, there.
Retype any code from memory and diff.
This stretch centers on PyTorch, CosineAnnealingWarmRestarts. Source cue: But much simpler option use the cosine annealing with warm restarts learning schedule. Let look now. Cosine Annealing.
Retype any code from memory and diff.
Ch11 panel 28
This stretch centers on CosineAnnealingWarmRestarts. Source cue: may also set mult you want double Tmax each round the default meaning Tmax stays constant and all.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: training e.g., down 0.85, linearly and then bring back the maximum value e.g., 0.95 during the second half.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: PyTorch offers few extra schedulers e.g., polynomial scheduler, cyclic scheduler, scheduler that makes easy chain other schedulers, and.
Retype any code from memory and diff.
Ch11 panel 29
This stretch centers on AdamW, SGD, PyTorch, Adam. Source cue: Moreover, even though batch-norm and layer-norm were designed solve the unstable gradients problems, they also act like pretty.
Retype any code from memory and diff.
This stretch centers on Adam, AdamW, SGD, PyTorch. Source cue: normally during training instead you are using Adam optimizer, you should switch AdamW and set the weight decay.
Retype any code from memory and diff.
This stretch centers on PyTorch, SGD. Source cue: for batch, batch train loader: ... the rest the training loop unchanged main loss loss pred, batch loss.
Retype any code from memory and diff.
Ch11 panel 30
This stretch centers on SGD, PyTorch, Dropout. Source cue: parameters, and will not use weight decay all. params bias and param for name, param model.named parameters bias.
Retype any code from memory and diff.
This stretch centers on Dropout. Source cue: Dropout was proposed paper Geoffrey Hinton al. 2012 and further detailed 2014 paper Nitish Srivastava al., and has.
Retype any code from memory and diff.
This stretch centers on Ch11 ideas. Source cue: training, neurons don get dropped anymore. And that all except for technical detail will discuss shortly surprising first.
Retype any code from memory and diff.
Ch11 panel 31
This stretch centers on Ch11 ideas. Source cue: They also cannot rely excessively just few input neurons; they must pay attention all their input neurons. They.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: neural networks, each with just one training instance. These neural networks are obviously not independent because they share.
Retype any code from memory and diff.
This stretch centers on PyTorch, Dropout, Sequential, ReLU. Source cue: inputs during training, which the same dividing them More generally, need divide the inputs the keep probability during.
Retype any code from memory and diff.
torch.compile and AMP (App B).Closing
Deep nets train when signals stay alive. Init, activation, norm, optimiser, schedule, regularise - then measure.
12. Deep Computer Vision Using CNNs
Convolution, pooling, CNN architectures (ResNet and friends), pretrained models, object detection (SSD/FCOS), and segmentation - the vision stack before transformers steal the show.
Conv mechanism.
Ch12 panel 1
Study loop.
This stretch centers on Ch12 ideas. Source cue: Chapter 12. Deep Computer Vision Using Convolutional Neural Networks Although IBM Deep Blue supercomputer beat the chess world.
Retype any code from memory and diff.
This stretch centers on PyTorch, CNN. Source cue: neural networks CNNs emerged from the study the brain visual cortex, and they have been used computer image.
Retype any code from memory and diff.
This stretch centers on Ch12 ideas. Source cue: each pixel according the class the object belongs The Architecture the Visual Cortex David Hubel and Torsten Wiesel.
Retype any code from memory and diff.
Ch12 panel 2
This stretch centers on LeCun, LeNet. Source cue: more complex patterns larger receptive fields Moreover, the authors showed that some neurons react only images horizontal lines,.
Retype any code from memory and diff.
This stretch centers on LeNet, CNN. Source cue: recognize handwritten digits checks. This architecture has some building blocks that you already know, such fully connected layers.
Retype any code from memory and diff.
This stretch centers on CNN. Source cue: the input image like they were the layers discussed previous chapters but only pixels their receptive fields see.
Retype any code from memory and diff.
Ch12 panel 3
Ops contrast.
This stretch centers on Ch12 ideas. Source cue: their corresponding inputs. neuron located row column given layer connected the outputs the neurons the previous layer located.
Retype any code from memory and diff.
This stretch centers on Ch12 ideas. Source cue: example the stride the same both directions, which generally the case although there are exceptions neuron located row.
Retype any code from memory and diff.
This stretch centers on Ch12 ideas. Source cue: column, which full neurons using these weights will ignore everything their receptive field except for the central vertical.
Retype any code from memory and diff.
Ch12 panel 4
Practice hub.
This stretch centers on Ch12 ideas. Source cue: the top-left image. Notice that the vertical white lines get enhanced while the rest gets blurred. Similarly, the.
Retype any code from memory and diff.
This stretch centers on CNN. Source cue: 12-6. Two convolutional layers with multiple filters each kernels processing color image with three color channels; each convolutional.
Retype any code from memory and diff.
This stretch centers on Ch12 ideas. Source cue: particular location. Input images are also composed multiple sublayers: one per color channel. mentioned Chapter there are typically.
Retype any code from memory and diff.
Ch12 panel 5
Fit discipline.
This stretch centers on Ch12 ideas. Source cue: big mathematical equation: shows how compute the output given neuron convolutional layer. bit ugly due all the different.
Retype any code from memory and diff.
This stretch centers on PyTorch, NumPy. Source cue: previous layer layer the output the neuron located layer row column feature map channel the previous layer the.
Retype any code from memory and diff.
This stretch centers on NumPy, PyTorch, TorchVision, CenterCrop. Source cue: arrays, let stack these images into single NumPy array, then convert 32-bit float tensor, and rescale the pixel.
Retype any code from memory and diff.
Ch12 panel 6
Error parts.
This stretch centers on PyTorch. Source cue: see what comes out. For this, PyTorch provides the nn.Conv2d layer. Under the hood, this layer relies the.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: green, and blue each location, now have the intensity each feature each location. Second, the height and width.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: padded with enough zeros all sides ensure that the output feature maps end with the same size the.
Retype any code from memory and diff.
Ch12 panel 7
Train vs serve.
This stretch centers on PyTorch. Source cue: almost certainly bad idea since would drown your image sea zeros for this reason, PyTorch raises exception you.
Retype any code from memory and diff.
This stretch centers on Ch12 ideas. Source cue: height and width the input images not appear the kernel shape: this because all the neurons the output.
Retype any code from memory and diff.
This stretch centers on ReLU, CNN. Source cue: and width the kernel, and the number input channels. saw Chapter 11, you will generally want reinitialize the.
Retype any code from memory and diff.
Ch12 panel 8
Model promotion.
This stretch centers on Ch12 ideas. Source cue: Their goal subsample i.e., shrink the input image order reduce the computational load, the memory usage, and the.
Retype any code from memory and diff.
This stretch centers on Ch12 ideas. Source cue: input values are and only the max value, propagated the next layer. Because the stride the output image.
Retype any code from memory and diff.
This stretch centers on CNN. Source cue: two pixels the right. you can see, the outputs the max pooling layer for images and are identical..
Retype any code from memory and diff.
Ch12 panel 9
Study loop.
This stretch centers on PyTorch. Source cue: dropping the input values. And some applications, invariance not desirable. Take semantic segmentation the task classifying each pixel.
Retype any code from memory and diff.
This stretch centers on CNN, PyTorch. Source cue: the max. Average pooling layers used very popular, but people mostly use max pooling layers now, they generally.
Retype any code from memory and diff.
This stretch centers on CNN, PyTorch, DepthPool. Source cue: anything: thickness, brightness, skew, color, and on. Depthwise max pooling can help.
Retype any code from memory and diff.
Ch12 panel 10
This stretch centers on Ch12 ideas. Source cue: size and the default stride equal kernel size and padding The forward method starts merging the spatial dimensions,.
Retype any code from memory and diff.
This stretch centers on CNN. Source cue: pooling layer. works very differently: all does compute the mean each entire feature map. Therefore outputs single number.
Retype any code from memory and diff.
This stretch centers on CNN, ReLU. Source cue: nn.AdaptiveAvgPool2d output size output global avg pool cropped images Alternatively, you could just use the torch.mean function get.
Retype any code from memory and diff.
Ch12 panel 11
Ops contrast.
This stretch centers on CNN, Sequential, ReLU, Dropout. Source cue: will use fewer parameters and require fewer computations, and will usually perform better. One exception for the first.
Retype any code from memory and diff.
This stretch centers on Sequential, ReLU, CNN. Source cue: different default arguments: small kernel size and same padding. This avoids having repeat these arguments throughout the model..
Retype any code from memory and diff.
This stretch centers on ReLU, CrossEntropyLoss. Source cue: 256 makes sense for grow, since the number low-level features often fairly low e.g., small circles, horizontal lines.
Retype any code from memory and diff.
Ch12 panel 12
Practice hub.
This stretch centers on RuntimeError, LazyLinear. Source cue: first dense layer, since expects array features for each instance. also add two dropout layers, with dropout rate.
Retype any code from memory and diff.
This stretch centers on ImageNet. Source cue: matrix the first time gets called: can then automatically set the number input features the correct value. Other.
Retype any code from memory and diff.
This stretch centers on LeNet, AlexNet, GoogLeNet, ResNet, ResNeXt. Source cue: predictions did not include the correct answer. The images are fairly large e.g., 256 pixels high and there.
Retype any code from memory and diff.
Ch12 panel 13
Fit discipline.
This stretch centers on ReLU, LeCun, LeNet, AlexNet, CNN. Source cue: tanh Convolution 120 tanh Avg pooling tanh Convolution tanh Avg pooling tanh Convolution tanh Input you can see,.
Retype any code from memory and diff.
This stretch centers on LeNet, AlexNet, CNN, Softmax, ReLU. Source cue: AlexNet CNN architecture won the 2012 ILSVRC challenge large margin: achieved top-five error rate while the second best.
Retype any code from memory and diff.
This stretch centers on ReLU. Source cue: Convolution 256 same ReLU Max pooling valid Convolution valid ReLU Input RGB 227 227 reduce overfitting, the authors.
Retype any code from memory and diff.
Ch12 panel 14
Error parts.
This stretch centers on RandomCrop, RandomRotation. Source cue: tell whether was augmented not. Simply adding white noise will not help; the modifications should learnable white noise.
Retype any code from memory and diff.
This stretch centers on AlexNet, GoogLeNet. Source cue: classes. This called the synthetic minority oversampling technique, SMOTE for short. Lastly, although data augmentation typically used only.
Retype any code from memory and diff.
This stretch centers on AlexNet, GoogLeNet, ReLU. Source cue: with few tweaked hyperparameters number feature maps, kernel size, stride, etc. GoogLeNet The GoogLeNet architecture was developed Christian.
Retype any code from memory and diff.
Ch12 panel 15
Train vs serve.
This stretch centers on Ch12 ideas. Source cue: capture patterns different scales. Also note that every single layer uses stride and same padding even the max.
Retype any code from memory and diff.
This stretch centers on GoogLeNet, CNN. Source cue: they reduce dimensionality. This cuts the computational cost and the number parameters, speeding training and improving generalization. Each.
Retype any code from memory and diff.
This stretch centers on GoogLeNet, ReLU, CNN. Source cue: stack, including nine inception modules the boxes with the spinning tops The six numbers the inception modules represent.
Retype any code from memory and diff.
Ch12 panel 16
Model promotion.
This stretch centers on CNN, GoogLeNet, AlexNet. Source cue: max pooling layer reduces the image height and width again speed computations. Then comes the CNN backbone: tall.
Retype any code from memory and diff.
This stretch centers on GoogLeNet, ResNet, CNN. Source cue: limits the risk overfitting. The last layers are self-explanatory: dropout for regularization, then fully connected layer with 1,000.
Retype any code from memory and diff.
This stretch centers on CNN. Source cue: deep CNN composed 152 layers other variants had 34, 50, and 101 layers confirmed the general trend: computer.
Retype any code from memory and diff.
Ch12 panel 17
Study loop.
This stretch centers on ResNet, GoogLeNet, ReLU. Source cue: skip connection, the resulting network outputs copy its inputs; other words, acts the identity function the start training..
Retype any code from memory and diff.
This stretch centers on ReLU, ResNet. Regular.
Retype any code from memory and diff.
This stretch centers on ResNet, ResNets, GoogLeNet, ImageNet. Source cue: considerably without compromising accuracy. Different variations the architecture.
Retype any code from memory and diff.
Ch12 panel 18
This stretch centers on GoogLeNet, ResNet, ImageNet, CNN. Source cue: 512 maps, whopping RUs with 1,024 maps, and finally RUs with 2,048 maps. Note Google Inception-v4 architecture merged.
Retype any code from memory and diff.
This stretch centers on GoogLeNet. Source cue: convolutional layer makes the strong assumption that spatial patterns and cross-channel patterns can modeled separately see Figure 12-20.
Retype any code from memory and diff.
This stretch centers on GoogLeNet, PyTorch. Source cue: inception modules all. Well, discussed earlier, inception module contains convolutional layers with filters: these look exclusively for cross-channel.
Retype any code from memory and diff.
Ch12 panel 19
Ops contrast.
This stretch centers on ResNets, ResNet. Source cue: that channels and out channels need divisible groups default groups giving you normal convolutional layer, but you set.
Retype any code from memory and diff.
This stretch centers on ResNet, ReLU. Source cue: SENet adds small neural network, called block, every inception module residual unit the original architecture, shown Figure 12-21..
Retype any code from memory and diff.
This stretch centers on ReLU. Source cue: this feature map recalibration will help resolve the ambiguity.
Retype any code from memory and diff.
Ch12 panel 20
Practice hub.
This stretch centers on CNN, ResNeXt, ResNet. Source cue: see this principle action again when discuss autoencoders Chapter Finally, the output layer takes the embedding and outputs.
Retype any code from memory and diff.
This stretch centers on ResNeXt, ResNet, DenseNet, MobileNet, MobileNets. Source cue: dense network with hidden layers and the output layer. used small filters, but had many them. ResNeXt ResNeXt.
Retype any code from memory and diff.
This stretch centers on MobileNet, MobileNets, CNN, SqueezeNet, ShuffleNet. Source cue: For example, layer four block takes input the depthwise concatenation the outputs layers one, two, and three that.
Retype any code from memory and diff.
Ch12 panel 21
Fit discipline.
This stretch centers on ImageNet, EfficientNet, ConvNeXt, ResNet, CNN. Source cue: called compound scaling. They used neural architecture search find good architecture for scaled-down version ImageNet with smaller and.
Retype any code from memory and diff.
This stretch centers on CNN, EfficientNet, TorchVision. Source cue: compute budget doubles, then increases other words, the number floating-point operations available for training proportional Your CNN architecture.
Retype any code from memory and diff.
This stretch centers on TorchVision, ImageNet, EfficientNet, MobileNet, GoogLeNet. Source cue: project: Accuracy Model size e.g., for deployment mobile device Inference speed Energy consumption Table 12-3 lists some the.
Retype any code from memory and diff.
Ch12 panel 22
Error parts.
This stretch centers on DenseNet, EfficientNet, ResNet, ConvNeXt, AlexNet. Source cue: EfficientNet small 84.2 96.9 21.5M 8.4 ResNet 73.3 91.4 21.8M 3.7 Inception 77.3 93.5 27.2M 5.7 ConvNeXt Tiny.
Retype any code from memory and diff.
This stretch centers on Ch12 ideas. Source cue: stride and same padding, processing 150 100 RGB image channels The number parameters 200 15,200 the corresponds the.
Retype any code from memory and diff.
This stretch centers on GPU, CNN. Source cue: that just for one instance training batch contains 100 instances, then this single convolutional layer will use 1.2.
Retype any code from memory and diff.
Ch12 panel 23
This stretch centers on CNN, PyTorch. Source cue: 32-bit floats, distributing the CNN across multiple devices, offloading the most memory-hungry modules the CPU using module.to cpu.
Retype any code from memory and diff.
This stretch centers on RevNets. Source cue: produce the same result you call twice with the same inputs, else the gradients will incorrect. This means.
Retype any code from memory and diff.
This stretch centers on ReLU, CNN, RevNet, RevNets. Source cue: and long they always produce the same output for given input. For example, and can identical modules composed.
Retype any code from memory and diff.
Ch12 panel 24
This stretch centers on RevNets, Transformer, CNN, PyTorch, ResNet. Source cue: very first layer, which reduces the spatial dimensions and increases the number channels: the result can split two.
Retype any code from memory and diff.
This stretch centers on Sequential, ResidualUnit, ResNet, ReLU. Source cue: size stride stride, padding nn.BatchNorm2d out channels else: self.skip connection nn.Identity def forward self, inputs return F.relu self.main.
Retype any code from memory and diff.
This stretch centers on ReLU, ResidualUnit, LazyLinear, Sequential, ImageNet. Source cue: padding bias False nn.BatchNorm2d num features nn.ReLU nn.MaxPool2d kernel size stride padding prev filters for filters 128 256.
Retype any code from memory and diff.
Ch12 panel 25
This stretch centers on ResNet, PyTorch, CNN, TorchVision, GoogLeNet. Source cue: model that won the ILSVRC 2015 challenge This demonstrates both the elegance the ResNet model and the expressiveness.
Retype any code from memory and diff.
This stretch centers on ConvNeXt, TorchVision, CenterCrop. Source cue: small, base, and large and this code loads the base variant: weights torchvision.models.ConvNeXt Base Weights.IMAGENET1K model torchvision.models.convnext base.
Retype any code from memory and diff.
This stretch centers on TorchVision, CenterCrop, ImageNet, GPU, autograd. Source cue: could this using TorchVision CenterCrop and/or Resize transform, but much easier and safer use the transforms returned weights.transforms.
Retype any code from memory and diff.
Ch12 panel 26
This stretch centers on ImageNet, TorchVision. Source cue: each image i.e., the class with the maximum logit.
Retype any code from memory and diff.
This stretch centers on TorchVision, ImageNet, ConvNeXt. Source cue: probabilities for each these classes:.
Retype any code from memory and diff.
This stretch centers on DataLoader, TorchVision. Source cue: and there are 102 classes total the name indicates you try train model from scratch, you will really.
Retype any code from memory and diff.
Ch12 panel 27
This stretch centers on DataLoader, TorchVision, ConvNeXt, ImageNet, Sequential. Source cue: set and the validation set. then create the data loaders, usual: from torch.utils.data import DataLoader train loader DataLoader.
Retype any code from memory and diff.
This stretch centers on Sequential. Source cue: features module the main part the model, which includes all layers except for the global average pooling layer.
Retype any code from memory and diff.
This stretch centers on ImageNet, RandomHorizontalFlip, RandomRotation, RandomResizedCrop, ColorJitter. Source cue: can train this model for few epochs, and you will already reach about accuracy just training the new.
Retype any code from memory and diff.
Ch12 panel 28
This stretch centers on ImageNet, TorchGeo, TorchVision. Source cue: generalizes well many other image datasets, and also offers predefined settings for two other datasets: CIFAR10 and the.
Retype any code from memory and diff.
This stretch centers on ImageNet, TorchGeo, TorchVision, ConvNeXt, FlowerLocator. Source cue: satellite images, aerial images, even raster data such digital elevation models DEM then models pretrained ImageNet won help.
Retype any code from memory and diff.
This stretch centers on ConvNeXt, FlowerLocator, Sequential. Source cue: predict. does not require much change the ConvNeXt model; just need add second dense output layer with four.
Retype any code from memory and diff.
Ch12 panel 29
This stretch centers on CrossEntropyLoss, OpenLabeler, ImgLab, LabelBox, RectLabel. Source cue: preproc images ... batch preprocessed images pred logits, pred bbox locator model preprocessed images.to device But how can.
Retype any code from memory and diff.
This stretch centers on LabelBox, RectLabel, TorchVision, BoundingBoxes. Source cue: Supervisely, Roboflow, RectLabel. Many these are now assisted, greatly speeding the annotation task. You may also want consider.
Retype any code from memory and diff.
This stretch centers on TorchVision, BoundingBoxes, BoundingBoxFormat. Source cue: class that represents list bounding boxes. For example, the following code creates bounding box for the largest flower.
Retype any code from memory and diff.
Ch12 panel 30
This stretch centers on BoundingBoxes, BoundingBoxFormat, MSE. Source cue: .
Retype any code from memory and diff.
This stretch centers on MSE. Source cue: When using the MSE, 10-pixel error for large bounding box will penalized just much 10-pixel error for small.
Retype any code from memory and diff.
This stretch centers on Ch12 ideas. Source cue: not great for training because equal zero whenever and have overlap, regardless the distance between them their shapes:.
Retype any code from memory and diff.
Ch12 panel 31
This stretch centers on MSE, CNN. Source cue: torchvision.ops.generalized box iou loss function. IoU metric for bounding boxes Another important.
Retype any code from memory and diff.
This stretch centers on CNN. Source cue: common approach was take CNN that was trained classify and locate single object roughly centered the image, then.
Retype any code from memory and diff.
This stretch centers on CNN. Source cue: sliding across all regions and making predictions each step. Detecting multiple objects.
Retype any code from memory and diff.
Ch12 panel 32
This stretch centers on CNN. Source cue: next region, one grid cell the right centered the shaded blue square did not detect any flower centered.
Retype any code from memory and diff.
This stretch centers on CNN. Source cue: regions well, capture smaller and larger objects. This technique fairly straightforward, but you can see will often detect.
Retype any code from memory and diff.
This stretch centers on CNN. Source cue: get rid although this example would already have been removed the previous step Repeat step until there are.
Retype any code from memory and diff.
Ch12 panel 33
This stretch centers on Ch12 ideas. Source cue: size this the feature map size, not the kernel size Each neuron will compute weighted sum all 100.
Retype any code from memory and diff.
This stretch centers on CNN, CrossEntropyLoss. Source cue: dense layer convolutional layer, the number filters the convolutional layer must equal the number units the dense layer,.
Retype any code from memory and diff.
This stretch centers on CrossEntropyLoss, CNN. Source cue: 224 224 images, and outputs 107 values per image: The classification head outputs 102 class logits one per.
Retype any code from memory and diff.
Ch12 panel 34
This stretch centers on CNN. Source cue: FCN 448 448 image see the righthand side Figure 12-26 the bottleneck layer will now output feature maps..
Retype any code from memory and diff.
This stretch centers on YOLO. Source cue: possible locations for the window, hence predictions. However, the FCN approach much more efficient, since the network only.
Retype any code from memory and diff.
This stretch centers on YOLO. Source cue: means the bottom right. However, the bounding box height and width may extend well beyond the cell. outputs.
Retype any code from memory and diff.
Ch12 panel 35
This stretch centers on YOLO, PyTorch. Source cue: large horizontal one for the car, and smaller vertical one for the person. These bounding boxes may have.
Retype any code from memory and diff.
This stretch centers on CNN. Source cue: speed and accuracy, using variety techniques; for example, YOLOv3 boosted accuracy part thanks anchor priors, exploiting the fact.
Retype any code from memory and diff.
Closing
CNNs exploit locality and weight sharing. Know conv arithmetic before you abandon it for ViTs.
13. Processing Sequences Using RNNs and CNNs
Sequence hygiene.
Sequences for time series and text precursors: RNNs, LSTM/GRU, sequence-to-sequence, and convolutional alternatives over time.
Cell choice.
Ch13 panel 1
Study loop.
This stretch centers on RNN. Source cue: 13. Processing Sequences Using RNNs and CNNs Predicting the future something you all the time, whether you are.
Retype any code from memory and diff.
This stretch centers on LSTM, GRU, WaveNet, CNN. Source cue: fundamental concepts underlying RNNs and how train them using backpropagation through time. Then, will use them forecast time.
Retype any code from memory and diff.
This stretch centers on WaveNet, CNN, Attention, RNN. Source cue: architecture capable handling sequences tens thousands time steps. But can even better Chapter 14, will apply RNNs natural.
Retype any code from memory and diff.
Ch13 panel 2
This stretch centers on Ch13 ideas. Source cue: neuron receives the inputs well its own output from the previous time step, Since there previous output the.
Retype any code from memory and diff.
This stretch centers on ReLU. Source cue: outputs the previous time step, Let call these weight vectors and consider the whole recurrent layer instead just.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Source cue: containing the layer outputs time step for each instance the mini-batch the number instances the mini-batch, and nneurons.
Retype any code from memory and diff.
Ch13 panel 3
Ops contrast.
This stretch centers on RNN. Source cue: 13-2 The notation represents the horizontal concatenation the matrices and Notice that function and which function and which.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Source cue: time steps, you could say has form memory. part neural network that preserves some state across time steps.
Retype any code from memory and diff.
This stretch centers on RNN. Source cue: basic cells have discussed far, the output just equal the state, but more complex cells this not always.
Retype any code from memory and diff.
Ch13 panel 4
Practice hub.
This stretch centers on CNN. Source cue: and ignore all outputs except for the last one see the top-right network Figure 13-4 This sequence-to-vector network..
Retype any code from memory and diff.
This stretch centers on RNN. Source cue: You would feed the network sentence one language, the encoder would convert this sentence into single vector representation,.
Retype any code from memory and diff.
This stretch centers on RNN, PyTorch. Source cue: regular backpropagation, there first forward pass through the unrolled network represented the dashed arrows Then the output sequence.
Retype any code from memory and diff.
Ch13 panel 5
Fit discipline.
This stretch centers on PyTorch. Source cue: backward phase complete and all the gradients have been computed, BPTT can perform gradient descent step update the.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Source cue: path, parse dates service date df.columns date day type bus rail total shorter names df.sort values date .set.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Source cue: both the start and end month the range, this plots the data from the 1st March all the.
Retype any code from memory and diff.
Ch13 panel 6
Error parts.
This stretch centers on Ch13 ideas. Source cue: results. This called naive forecasting: simply copying past value make our forecast. Naive forecasting often great baseline, and.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Time series overlaid with 7-day lagged time series top.
Retype any code from memory and diff.
This stretch centers on MSE. Source cue: hard tell glance how good bad this is, let put the forecast errors into perspective dividing them the.
Retype any code from memory and diff.
Ch13 panel 7
Train vs serve.
This stretch centers on MSE. Source cue: metric depends the task. For example, your project suffers quadratically more from large errors than from small ones,.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Source cue: taking long-term patterns into account. For example, daily bus ridership dropped about 2,500 October 2017, which represents about.
Retype any code from memory and diff.
Ch13 panel 8
Model promotion.
This stretch centers on Ch13 ideas. Source cue: component computed using weighted sum the last few forecast errors. Equation 13-3 shows how the model makes its.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Source cue: step will produce approximation the derivative the time series: indeed, will give the slope the series each time.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Source cue: the central contribution the autoregressive integrated moving average ARIMA model, introduced 1970 George Box and Gwilym Jenkins their.
Retype any code from memory and diff.
Ch13 panel 9
Study loop.
This stretch centers on Ch13 ideas. Source cue: the time series 2s, 3s, etc. Let see how fit SARIMA model the rail time series, and use.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Source cue: already daily, but without this the ARIMA class would have guess the frequency, and would display warning. Next,.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Source cue: same code loop make forecasts for every day March, April, and May, and compute the MAE over that.
Retype any code from memory and diff.
Ch13 panel 10
This stretch centers on Ch13 ideas. Source cue: each model you want evaluate i.e., each hyperparameter combination you can run the preceding code example, changing only.
Retype any code from memory and diff.
This stretch centers on TimeSeriesDataset, IndexError. Source cue: that have two baselines, naive forecasting and SARIMA, let try use the machine learning models covered far forecast.
Retype any code from memory and diff.
This stretch centers on TimeSeriesDataset, IndexError, RNN, DataLoader. Source cue: class TimeSeriesDataset torch.utils.data.Dataset def init self, series, window length self.series series self.window length window length def len self.
Retype any code from memory and diff.
Ch13 panel 11
Ops contrast.
This stretch centers on TimeSeriesDataset, DataLoader. Source cue: .
Retype any code from memory and diff.
This stretch centers on FloatTensor. Source cue: models, let ahead and prepare our ridership dataset. First, need split our data into training period, validation period,.
Retype any code from memory and diff.
This stretch centers on TimeSeriesDataset, DataLoader. Source cue: 2001 2019, you might able split this data across the different companies. very likely that many these companies.
Retype any code from memory and diff.
Ch13 panel 12
Practice hub.
This stretch centers on Sequential, HuberLoss, SGD, MeanAbsoluteError, PyTorch. Source cue: Using Linear Model Let try basic linear model first. will use the Huber loss, which usually works better.
Retype any code from memory and diff.
This stretch centers on SimpleRnnModel, Sequential. Source cue: the last hidden state input and output the model forecast: class SimpleRnnModel nn.Module def init self, input size,.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Source cue: per time step: sequential module composed nn.Linear layer and the tanh activation function. You can use another activation.
Retype any code from memory and diff.
Ch13 panel 13
Fit discipline.
This stretch centers on Ch13 ideas. Source cue: window, there one zero per recurrent neuron, the hidden state shape batch size, hidden size Next, iterate over.
Retype any code from memory and diff.
This stretch centers on GPU, PyTorch, RNN, SimpleRnnModel. Source cue: memory cell both the current inputs and the last hidden state, which gives the new hidden state. repeats.
Retype any code from memory and diff.
This stretch centers on PyTorch, RNN, SimpleRnnModel. Source cue: much about preprocessing. However, get the best performance, you may want try making the time series more stationary,.
Retype any code from memory and diff.
Ch13 panel 14
Error parts.
This stretch centers on RNN. Source cue: constructor also creates output layer, exactly like our previous implementation. the forward method, pass the input batch directly.
Retype any code from memory and diff.
This stretch centers on RNN, GPU. Source cue: state each recurrent layer after the very last time step. Its shape number layers, batch size, hidden size.
Retype any code from memory and diff.
This stretch centers on RNN, PyTorch. Source cue: results before. Forecasting Using Deep RNN quite common stack multiple layers cells, shown Figure 13-10. This gives you.
Retype any code from memory and diff.
Ch13 panel 15
Train vs serve.
This stretch centers on DataFrame, FloatTensor, PyTorch, TimeSeriesDataset. Source cue: fact, let also throw the day type Since can always know advance whether tomorrow going weekday, weekend, holiday,.
Retype any code from memory and diff.
This stretch centers on FloatTensor, PyTorch, TimeSeriesDataset, MulvarTimeSeriesDataset, DataLoader. Source cue: torch.FloatTensor mulvar 2019-01 2019-05 .values 1e6 mulvar test torch.FloatTensor mulvar 2019-06 .values 1e6 Then need create the PyTorch.
Retype any code from memory and diff.
This stretch centers on RNN, SimpleRnnModel, MulvarTimeSeriesDataset. Source cue: the target shape 32, Perfect can finally create the RNN: torch.manual seed mulvar model SimpleRnnModel input size hidden.
Retype any code from memory and diff.
Ch13 panel 16
Model promotion.
This stretch centers on RNN. Source cue: task, and this particular case the multitask RNN that forecasts both the bus and the rail ridership doesn.
Retype any code from memory and diff.
This stretch centers on GPU. Source cue: the predicted value had actually occurred. then use the model again predict the following value, and on, the.
Retype any code from memory and diff.
This stretch centers on RNN, ForecastAheadDataset, TimeSeriesDataset. Source cue: are the last values The resulting forecasts are plotted Figure 13-11. Warning the model makes error one time.
Retype any code from memory and diff.
Ch13 panel 17
Study loop.
This stretch centers on TimeSeriesDataset, ForecastAheadDataset, DataLoader, RNN, SimpleRnnModel. Source cue: return window, target Tip hardcoded the number 14, but real project you should make this configurable e.g., just.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Source cue: This approach works quite well. Its forecasts for the next day are obviously better than its forecasts for.
Retype any code from memory and diff.
This stretch centers on RNN. Source cue: words, the targets are sequences consecutive windows, shifted one time step each time step. The target for each.
Retype any code from memory and diff.
Ch13 panel 18
This stretch centers on RNN, ForecastAheadDataset. Source cue: And you right: after training, you can actually ignore all outputs except for the very last time step..
Retype any code from memory and diff.
This stretch centers on ForecastAheadDataset, DataLoader, SimpleRnnModel. Source cue: getitem self, idx end idx self.window length 1st index after window window self.series idx end target period self.series.
Retype any code from memory and diff.
This stretch centers on DataLoader, SimpleRnnModel. Source cue: seq train loader DataLoader seq train set, batch size 32, shuffle True ... create the datasets and data.
Retype any code from memory and diff.
Ch13 panel 19
This stretch centers on Ch13 ideas. Source cue: relies torch.matmul for matrix multiplication. This function efficiently supports multiplying arrays more than two dimensions. For example, you.
Retype any code from memory and diff.
This stretch centers on Ch13 ideas. Source cue: you pass input nn.Linear layer. Tip Another way get the exact same result replace the nn.Linear output layer.
Retype any code from memory and diff.
This stretch centers on RNN, ReLU. Source cue: good. course, the model not accurate for more distant forecasts. For example, the MAE for 35,315. Simple RNNs.
Retype any code from memory and diff.
Ch13 panel 20
This stretch centers on ReLU, RNN. Source cue: can also used for RNNs: good parameter initialization, faster optimizers, dropout, and on. However, nonsaturating activation functions e.g.,.
Retype any code from memory and diff.
This stretch centers on PyTorch, RNN. Source cue: normalization cannot used efficiently with RNNs with deep feedforward nets. fact, you cannot use between time steps, only.
Retype any code from memory and diff.
This stretch centers on PyTorch, RNN, LayerNorm, SimpleRnnModel, Sequential. Source cue: Layer norm introduced Chapter tends work bit better than within recurrent layers. usually applied just before the activation.
Retype any code from memory and diff.
Ch13 panel 21
This stretch centers on RNN, LSTM. Source cue: you must write custom RNN since the nn.RNN module does not support that. However, does support adding dropout.
Retype any code from memory and diff.
This stretch centers on LSTM, RNN. Source cue: problem, various types cells with long-term memory have been introduced. They have proven successful that the basic cells.
Retype any code from memory and diff.
This stretch centers on LSTM. Source cue: long-term state. LSTM cell Now let open the box The key idea.
Retype any code from memory and diff.
Ch13 panel 22
This stretch centers on RNN, LSTM. Source cue: how the gates work. First, the current input vector and the previous short-term state are fed four different.
Retype any code from memory and diff.
This stretch centers on LSTM. Source cue: should erased. The input gate controlled controls which parts should added the long-term state. Finally, the output gate.
Retype any code from memory and diff.
This stretch centers on LSTM, RNN. Source cue: computations t-1 t-1 t-1 tanh t-1 t-1 tanh this equation: Wxi, Wxf, Wxo, and Wxg are the weight.
Retype any code from memory and diff.
Ch13 panel 23
This stretch centers on RNN, LSTM, LstmModel, SimpleRnnModel, GRU. Source cue: the four layers. Tip Try replacing nn.RNN with nn.LSTM the previous models and see what performance you can.
Retype any code from memory and diff.
This stretch centers on LSTM, GRU. Source cue: two parts: the short-term and the long-term There are several variants the LSTM cell. One particularly popular variant.
Retype any code from memory and diff.
This stretch centers on GRU, PyTorch, RNN, LSTM. Source cue: full state vector output every time step. However, there new gate controller that controls which part the previous.
Retype any code from memory and diff.
Ch13 panel 24
This stretch centers on RNN, GRU, LSTM. Source cue: based GRU cell just replace nn.LSTMCell with nn.GRUCell the previous example, and get rid LSTM and GRU are.
Retype any code from memory and diff.
This stretch centers on GRU. Source cue: kernel size you use kernels, then the layer output will composed sequences all the same length equivalently you.
Retype any code from memory and diff.
Closing
Sequence models need causal hygiene. Do not shuffle across time; do not leak the future into features.
14. NLP with RNNs and Attention
Attention steps.
From text tensors to embeddings, RNN language models, attention, and the bridge toward transformers.
Ch14 panel 1
Study loop.
This stretch centers on Attention. Source cue: Chapter 14. Natural Language Processing with RNNs and Attention When Alan Turing imagined his famous Turing test 1950,.
Retype any code from memory and diff.
This stretch centers on Transformer, RNN. Source cue: the fact that mastering language arguably Homo sapiens greatest cognitive ability. Until recently, state-of-the-art natural language processing NLP.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: original text. the second section, will turn text classification, and more specifically sentiment analysis, which aims predict how.
Retype any code from memory and diff.
Ch14 panel 2
This stretch centers on RNN. Source cue: final boss this chapter will neural machine translation NMT the topic the third and last section: will build.
Retype any code from memory and diff.
This stretch centers on RNN. Source cue: sample the text generated char-RNN model after was trained all Shakespeare works: PANDARUS: Alas, think shall come approached.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: text download shakespeare text Let print the first few lines:.
Retype any code from memory and diff.
Ch14 panel 3
Ops contrast.
This stretch centers on RNN. Source cue: .
Retype any code from memory and diff.
This stretch centers on DataLoader, CharDataset, IndexError, GPU. Source cue: without the final and the corresponding target sequence character IDs representing the text not with the final but.
Retype any code from memory and diff.
This stretch centers on CharDataset, DataLoader, RNN. Source cue: 000:1 060 000 window length test set CharDataset shakespeare text 060 000: window length train loader DataLoader train.
Retype any code from memory and diff.
Ch14 panel 4
Practice hub.
This stretch centers on Ch14 ideas. Source cue: inputs represent similar things; unfortunately, similar IDs may represent totally unrelated tokens, and conversely, distant IDs may represent.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: comparatively small dense vector; for example, with just 300 dimensions. Tip The embedding size hyperparameter you can tune..
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: away from the INLAND category embedding see Figure 14-2 Indeed, the better the representation, the easier will for.
Retype any code from memory and diff.
Ch14 panel 5
Fit discipline.
This stretch centers on Ch14 ideas. Source cue: 2013, when Tom Mikolov and other Google researchers published paper describing efficient technique learn word embeddings using neural.
Retype any code from memory and diff.
This stretch centers on PyTorch, Embedding. Source cue: also encoded the embeddings. Word embeddings similar words tend close, and some.
Retype any code from memory and diff.
This stretch centers on Embedding. Source cue: row for each token the vocabulary and one column per embedding dimension. The embedding dimensionality hyperparameter you can.
Retype any code from memory and diff.
Ch14 panel 6
Error parts.
This stretch centers on Embedding, RNN, GRU. Source cue: these encodings are just random. Note that embedding layer mathematically equivalent one-hot encoding followed linear layer with bias.
Retype any code from memory and diff.
This stretch centers on RNN, GRU, ShakespeareModel, Embedding. Source cue: than simple RNN with few recurrent neurons. Let build and train model with two-layer nn.GRU module introduced Chapter.
Retype any code from memory and diff.
This stretch centers on Embedding, GRU, CrossEntropyLoss. Source cue: outputs the nn.Embedding layer will float tensors shape batch size, window length, embedding size The nn.GRU layer has.
Retype any code from memory and diff.
Ch14 panel 7
Train vs serve.
This stretch centers on GRU, CrossEntropyLoss, GPU. Source cue: why must permute the last two dimensions the nn.Linear layer output. Note that the nn.GRU layer also returns.
Retype any code from memory and diff.
This stretch centers on GPU, RNN. Source cue: Lastly, convert the token character, and that our prediction. The model correctly predicts great Now let use this.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: next character randomly, using the model estimated probability distribution: the model estimates probability for given token, then this.
Retype any code from memory and diff.
Ch14 panel 8
Model promotion.
This stretch centers on Ch14 ideas. Source cue: when generating more diverse and creative text. Let write next char helper function that will use this approach.
Retype any code from memory and diff.
This stretch centers on GRU, LSTM, RNN. Source cue: you may gone. romeo: that.
Retype any code from memory and diff.
This stretch centers on LSTM, GRU, RNN. Source cue: but would also make training harder, and even LSTM and GRU cells cannot handle very long sequences. Interestingly,.
Retype any code from memory and diff.
Ch14 panel 9
Study loop.
This stretch centers on Ch14 ideas. Source cue: Analysis Using Hugging Face Libraries One the most common applications NLP text classification especially sentiment analysis. image classification.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: original training set for training, and the remaining for validation, using the train test split method split the.
Retype any code from memory and diff.
This stretch centers on RNN. Source cue: review much harder classify: contains detailed description the movie, sprinkled with both positive and negative comments. Luckily, the.
Retype any code from memory and diff.
Ch14 panel 10
This stretch centers on BpeTrainer. Source cue: encoding BPE introduced Philip Gage 1994 initially for data compression BPE works splitting the whole training set into.
Retype any code from memory and diff.
This stretch centers on BpeTrainer. Source cue: replaced with the unk token. then create Tokenizer based the BPE model. The Tokenizers library lets you specify.
Retype any code from memory and diff.
This stretch centers on BpeTrainer. Source cue: and unk will token Next create list all the text the IMBd training set. Lastly, train the tokenizer.
Retype any code from memory and diff.
Ch14 panel 11
Ops contrast.
This stretch centers on Ch14 ideas. Source cue: unknown token unk The tokenizer provides get vocab method which returns dictionary mapping each token its ID. You.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: three reviews, must first ensure that they all have the same number tokens, which not the case right.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: padded with 0s, which our padding token ID. Each Encoding object also includes attention mask attribute containing for.
Retype any code from memory and diff.
Ch14 panel 12
Practice hub.
This stretch centers on ByteLevel. Source cue: .
Retype any code from memory and diff.
This stretch centers on ByteLevel, WordPiece. Source cue: its name suggests, the ByteLevel pre-tokenizer allows the BPE model work the byte level, rather than the character.
Retype any code from memory and diff.
This stretch centers on WordPiece, BpeTrainer, WordPieceTrainer. Source cue: pair when the individual tokens are themselves frequent. This normalization tends favor more useful and meaningful tokens than.
Retype any code from memory and diff.
Ch14 panel 13
Fit discipline.
This stretch centers on Ch14 ideas. Source cue: Google. This technique bit different than the previous ones: starts out with very large vocabulary containing every frequent.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: This means that the word them has chance being sampled the single token them 0.1 chance being sampled.
Retype any code from memory and diff.
This stretch centers on SentencePiece, WordPiece. Source cue: not use spaces between words, Vietnamese uses spaces even within words, and German often attaches multiple words together,.
Retype any code from memory and diff.
Ch14 panel 14
Error parts.
This stretch centers on SentencePiece, WordPiece, GPT, BERT. Source cue: open source implementation. This project described 2018 paper Taku Kudo and John Richardson. Table 14-1 summarizes the three.
Retype any code from memory and diff.
This stretch centers on Transformer, GPT, AutoTokenizer, BatchEncoding. Source cue: tokenizer, will use the Hugging Face Transformers library. This library provides many popular models for NLP, computer vision,.
Retype any code from memory and diff.
This stretch centers on AutoTokenizer, BatchEncoding, WordPiece, BERT, GPT. Source cue: encoded sequences they exceed 500 tokens: import transformers gpt2 tokenizer transformers.AutoTokenizer.from pretrained gpt2 gpt2 encoding gpt2 tokenizer train.
Retype any code from memory and diff.
Ch14 panel 15
Train vs serve.
This stretch centers on PyTorch, AutoTokenizer, BERT. Source cue: reviews: usual, the shortest texts will padded the length the longest one using the padding token. This allows.
Retype any code from memory and diff.
This stretch centers on BERT, AutoTokenizer, PreTrainedTokenizerFast. Source cue: 1019, 1013, 1019, 102.
Retype any code from memory and diff.
This stretch centers on PreTrainedTokenizerFast, DataLoader, BatchEncoding, PyTorch, WordPiece. Source cue: ... The Transformers library also provides object that can wrap your own tokenizer from the Tokenizers library and.
Retype any code from memory and diff.
Ch14 panel 16
Model promotion.
This stretch centers on BatchEncoding, PyTorch, WordPiece, DataLoader, SentimentAnalysisModel. Source cue: PyTorch tensors for the token IDs and attention masks, along with another tensor containing the labels. For tokenization,.
Retype any code from memory and diff.
This stretch centers on GRU, BatchEncoding. Source cue: zero vector reduce the impact padding tokens the loss. Since this sequence-to-vector model, not sequence-to-sequence model, only need.
Retype any code from memory and diff.
This stretch centers on GRU, PackedSequence. Source cue: ambiguous, classifying them feels like flipping coin. One problem with our model the fact that are not fully.
Retype any code from memory and diff.
Ch14 panel 17
Study loop.
This stretch centers on PackedSequence, PyTorch, GRU, Embedding. Source cue: .
Retype any code from memory and diff.
This stretch centers on Embedding, GRU, RNN. Source cue: sequences without being bothered any padding tokens. You don actually need set padding idx anymore when creating the.
Retype any code from memory and diff.
This stretch centers on GRU, RNN, LSTM. Source cue: model. But for tasks like text classification, the encoder seq2seq model, often preferable look ahead the next words.
Retype any code from memory and diff.
Ch14 panel 18
This stretch centers on GRU. Source cue: layer, since the hidden states will double size: self.output nn.Linear hidden dim, must also concatenate the forward and.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: and backward states. One way this permute the first two dimensions the top hidden states using permute get.
Retype any code from memory and diff.
This stretch centers on BERT, AutoModel, Embedding, PyTorch. Source cue: something different: using pretrained embeddings. Reusing Pretrained Embeddings and Language Models Our model was able learn useful embeddings.
Text to vectors.
Retype any code from memory and diff.
Ch14 panel 19
Ops contrast.
This stretch centers on BERT, PyTorch, Embedding, SentimentAnalysisModelPreEmbeds, GloVe. Source cue: PyTorch, and contains regular nn.Embedding layer. could just replace our model nn.Embedding layer with this one and retrain.
Retype any code from memory and diff.
This stretch centers on GloVe, FastText, RNN, LSTM. Source cue: embeddings 2016 and more. However, this approach has its limits. particular, word has single representation, matter the context..
Retype any code from memory and diff.
This stretch centers on BERT, RNN, SentimentAnalysisModelBert. Source cue: classification tasks large margin reducing the error rate most cases Moreover, the authors showed that pretrained model fine-tuned.
Retype any code from memory and diff.
Ch14 panel 20
Practice hub.
This stretch centers on BERT, RNN, SentimentAnalysisModelBert, AutoModel, GRU. Source cue: time step BERT transformer, not RNN This last hidden state tensor has shape batch size, max sequence length,.
Retype any code from memory and diff.
This stretch centers on BERT, GRU, BertForSequenceClassification. Source cue: learned summarize the most important features the text into this embedding. This simplifies our model quite bit, since.
Retype any code from memory and diff.
This stretch centers on BERT, BertForSequenceClassification, BertForQuestionAnswering, RobertaForSequenceClassification, AutoModelForSequenceClassification. Source cue: binary classification task using BERT, can use the BertForSequenceClassification class provided the Transformers library. just BERT model plus.
Retype any code from memory and diff.
Ch14 panel 21
Fit discipline.
This stretch centers on CrossEntropyLoss, ModelOutput, BERT, BertForSequenceClassification. Source cue: classification exactly like multiclass classification, this model will output two logits instead one, and must trained using the.
Retype any code from memory and diff.
This stretch centers on ModelOutput, CrossEntropyLoss, LogSoftmax. Source cue: other model from the Transformers library then also computes the loss and returns the ModelOutput object. For example:.
Retype any code from memory and diff.
This stretch centers on PyTorch, TorchMetrics. Source cue: IMDb model. The Trainer API works directly with dataset objects, not data loaders, but expects the datasets contain.
Retype any code from memory and diff.
Ch14 panel 22
Error parts.
This stretch centers on TorchMetrics, TrainingArguments, TensorBoard. Source cue: provided the Hugging Face Evaluate library: they are designed work nicely with the Transformers library. Alternatively, although the.
Retype any code from memory and diff.
This stretch centers on TensorBoard, DataCollatorWithPadding, BERT. Source cue: logs to, such TensorBoard Weights Biases This can useful visualize the learning curves. For simplicity, set report none.
Retype any code from memory and diff.
This stretch centers on BERT. Source cue: pipelines for various tasks. Each pipeline contains pretrained model along with its corresponding preprocessing and post-processing modules. For.
Retype any code from memory and diff.
Ch14 panel 23
This stretch centers on BERT, GPU. Source cue: default model for the chosen task. For sentiment analysis, the time writing, the model chose: DistilBERT model scaled.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: latter classified very negative. The model also positive about Thailand but negative about Vietnam. You can try this.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: probably pause and think before you rush deploy model production. For example, you train model score resumes, you.
Retype any code from memory and diff.
Ch14 panel 24
This stretch centers on Ch14 ideas. Source cue: model chose well-suited for general-purpose sentiment analysis, such movie reviews, but other models are better suited for specific.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: example, model fine-tuned the multi-genre natural language inference MultiNLI dataset can classify pair texts each ending with separation.
Retype any code from memory and diff.
This stretch centers on RNN. Source cue: https://huggingface.co/tasks. Warning Before you download model, make sure you trust the hosting platform e.g., the Hugging Face Hub.
Retype any code from memory and diff.
Ch14 panel 25
This stretch centers on Embedding. Source cue: Vamos Encoder-Decoder Network for Neural Machine Translation Let begin with relatively simple sequence-to-sequence NMT model that will translate.
Retype any code from memory and diff.
This stretch centers on Embedding, CrossEntropyLoss, RNN. Source cue: nn.Embedding layer returns the token embedding. These token embeddings are then fed the encoder and the decoder. each.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: 1406.png Figure 14-6. inference time, the decoder fed input the word just output the previous time step Tip.
Retype any code from memory and diff.
Ch14 panel 26
This stretch centers on BpeTrainer. Source cue: test set load dataset path ageron/tatoeba train name eng-spa split validation test split nmt original valid set.train test.
Retype any code from memory and diff.
This stretch centers on GPU, NmtPair, NmtPairBase, DataLoader. Source cue: tokenizer trainer Let test this tokenizer:.
Retype any code from memory and diff.
This stretch centers on NmtPair, DataLoader, Embedding, GRU. Source cue: enc tgt encodings inputs NmtPair src token ids, src mask, tgt token ids :-1 tgt mask :-1 labels.
Retype any code from memory and diff.
Ch14 panel 27
This stretch centers on NmtPair, Embedding, GRU, NmtModel. Source cue: create the data loaders usual. And now are ready build our translation model. just like Figure 14-5, except.
Retype any code from memory and diff.
This stretch centers on GRU, CrossEntropyLoss, BLEU, TorchMetric. Source cue: last hidden states across all nn.GRU layers Lastly, the decoder outputs are passed through the output nn.Linear layer,.
Retype any code from memory and diff.
This stretch centers on CrossEntropyLoss. Source cue: the output tokens when the targets are padding tokens, like this: xentropy nn.CrossEntropyLoss ignore index ignore pad tokens.
Retype any code from memory and diff.
Ch14 panel 28
This stretch centers on CrossEntropyLoss. Source cue: model, src text, max length 20, pad eos tgt text token ids for index range max length batch,.
Retype any code from memory and diff.
This stretch centers on PyTorch, AdaptiveLogSoftmaxWithLoss. Source cue: use sampled softmax, introduced 2015 bastien Jean al. Instead computing the softmax over all the logits, computes approximation.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: weights these two layers, and was first proposed 2016 paper Ofir Press and Lior Wolf. implement it, just.
Retype any code from memory and diff.
Ch14 panel 29
This stretch centers on GRU. Source cue: play around with our translation model, you will find that often works reasonably well short text, but really.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: Unfortunately, this means that when the model makes one mistake, stuck with for the rest the translation, which.
Retype any code from memory and diff.
This stretch centers on Ch14 ideas. Source cue: One the most common solutions beam search: keeps track short list the most promising output sequences say, the.
Retype any code from memory and diff.
Ch14 panel 30
This stretch centers on Ch14 ideas. Source cue: Note that these are actually conditional probabilities, given that the sentence starts with For the second sentence the.
Retype any code from memory and diff.
This stretch centers on GenerationMixin. Source cue: winning, but gusta has not been eliminated. Beam search, with beam width.
Retype any code from memory and diff.
This stretch centers on RNN, Attention. Source cue: from the Transformers library get their generate method: accepts num beams argument which you can set the desired.
Retype any code from memory and diff.
Closing
Attention routes information without stuffing everything into one hidden state. Transformers come next.
15. Transformers for NLP and Chatbots
Chatbot pieces.
Transformer architecture, pretrained LMs, fine-tuning, instruction tuning, and chatbot patterns - the NLP core of the modern stack.
Ch15 panel 1
Study loop.
This stretch centers on Attention, Transformer. Source cue: Chapter 15. Transformers for Natural Language Processing and Chatbots landmark 2017 paper titled Attention All You Need team.
Retype any code from memory and diff.
This stretch centers on Transformer, ImageNet, GPT. Source cue: full translation the target fed the decoder step starting with start-of-sequence token and steps and are not needed..
Retype any code from memory and diff.
This stretch centers on ImageNet, GPT, Transformer, BERT, ViT. Source cue: language tasks. The year 2018 was even called the ImageNet moment for NLP June 2018, OpenAI released the.
Retype any code from memory and diff.
Ch15 panel 2
This stretch centers on ViT, Transformer. Source cue: fact, some transformers can even handle multiple modalities once e.g., text image these are called multimodal models. This.
Retype any code from memory and diff.
This stretch centers on GPT, DeepSeek. Source cue: 15-2. Some the most influential transformers released since 2018; see larger online Then, November 2022, OpenAI released ChatGPT,.
Retype any code from memory and diff.
This stretch centers on GPT, DeepSeek, Transformer, BERT. Source cue: Anthropic Claude models, and Google Gemini models and they can only used via web UI, app, API: you.
Retype any code from memory and diff.
Ch15 panel 3
Ops contrast.
This stretch centers on BERT, GPT. Source cue: like BERT, learn how they are pretrained, and see how use them for tasks like text classification, semantic.
Retype any code from memory and diff.
This stretch centers on Transformer, Attention. Source cue: will take quick look encoder-decoder models, such and BART, which are great for tasks such translation and summarization..
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: max English sequence length the batch, embedding size and returns tensor the exact same shape. This means that.
Retype any code from memory and diff.
Ch15 panel 4
Practice hub.
This stretch centers on Ch15 ideas. Source cue: the sentence translate like soccer and already called the decoder four times, producing one new token each time:.
Retype any code from memory and diff.
This stretch centers on PyTorch, CrossEntropyLoss. Source cue: Figure 15-3. The original 2017 transformer architecture After going through the decoder, each token representation goes through final.
Retype any code from memory and diff.
This stretch centers on ReLU. Source cue: outputs the whole encoder stack are fed each the decoder blocks. you can see, you are already familiar.
Retype any code from memory and diff.
Ch15 panel 5
Fit discipline.
This stretch centers on Transformer. Source cue: the layer notices the subject infers that like must verb will discuss exactly how this works shortly. The.
Retype any code from memory and diff.
This stretch centers on Transformer. Source cue: sentence. This needed because all layers the Transformer architecture are position-agnostic, meaning they treat all positions equally unlike.
Retype any code from memory and diff.
This stretch centers on Transformer, Embedding, PositionalEmbedding, Dropout. Source cue: more detail, starting with the positional encodings. Positional Encodings positional encoding dense vector that encodes the position token.
Retype any code from memory and diff.
Ch15 panel 6
Error parts.
This stretch centers on Dropout, Transformer, Attention. Source cue: self.dropout self.pos embed :X.size Note The inputs have shape batch size, sequence length, embedding size but are adding.
Retype any code from memory and diff.
This stretch centers on Attention. Source cue: attention MHA layer based scaled dot-product attention, variant dot-product attention introduced Chapter that scales down the similarity scores.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: prevent this matrix from being huge, the input sequences must not too long: this the critical quadratic context.
Retype any code from memory and diff.
Ch15 panel 7
Train vs serve.
This stretch centers on PyTorch. Source cue: can add torch.inf The resulting weights will equal zero. This useful mask padding tokens, well future tokens the.
Retype any code from memory and diff.
This stretch centers on Transformer. Source cue: transformation again, across all tokens But why What the intuition behind this architecture Well, consider once again the.
Retype any code from memory and diff.
This stretch centers on Transformer, MultiheadAttention, Dropout. Source cue: example, check out this fascinating post Anthropic. This why the MHA layer splits the values, keys, and queries.
Retype any code from memory and diff.
Ch15 panel 8
Model promotion.
This stretch centers on MultiheadAttention, Dropout. Source cue: num heads, dropout 0.1 super init self.h num heads self.d embed dim num heads self.q proj nn.Linear embed.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: method used the forward method. splits its input along its last dimension one split per head converting from.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: all the attention outputs weights The forward method starts applying linear transformation the query, key, and value, and.
Retype any code from memory and diff.
Ch15 panel 9
Study loop.
This stretch centers on Ch15 ideas. Source cue: decoder masked self-attention layers must only consider previous tokens when trying predict what the next token else would.
Retype any code from memory and diff.
This stretch centers on PyTorch, MultiheadAttention. Source cue: will zero after the softmax operation tried zero out these weights directly, the remaining weights would not add.
Retype any code from memory and diff.
This stretch centers on Transformer, TransformerEncoderLayer, MultiheadAttention, Dropout. Source cue: this argument False, sometimes allows for some optimizations. When need weights set False, the method returns None instead.
Retype any code from memory and diff.
Ch15 panel 10
This stretch centers on TransformerEncoderLayer, MultiheadAttention, Dropout, LayerNorm, ReLU. Source cue: self-attention layer and after both dense layers the feedforward module: class TransformerEncoderLayer nn.Module def init self, model, nhead,.
Retype any code from memory and diff.
This stretch centers on TransformerDecoderLayer, LayerNorm, Dropout, PyTorch, TransformerEncoderLayer. Source cue: the encoder allows each token attend all tokens, even ones located after it. However, the user expected set.
Retype any code from memory and diff.
This stretch centers on PyTorch, TransformerEncoderLayer, TransformerDecoderLayer, GELU, TransformerEncoder. Source cue: box, with the same arguments, plus few more: most importantly batch first, which you must set True the.
Retype any code from memory and diff.
Ch15 panel 11
Ops contrast.
This stretch centers on Transformer, CrossEntropyLoss, Embedding, PyTorch, NmtTransformer. Source cue: need add final Linear layer and use the nn.CrossEntropyLoss get the full architecture shown Figure 15-3 saw earlier.
Retype any code from memory and diff.
This stretch centers on NmtPair, Attention, Transformer, CrossEntropyLoss. Source cue: modules. The forward method takes NmtPair input this class was defined Chapter The method starts embedding the input.
Retype any code from memory and diff.
This stretch centers on CrossEntropyLoss, RNN, NmtTransformer, GPU. Source cue: Lastly, pass the result through the output Linear layer, and permute the last two dimensions because nn.CrossEntropyLoss expects.
Retype any code from memory and diff.
Ch15 panel 12
Practice hub.
This stretch centers on GPU, CUDA, Transformer, BERT. Source cue: how ChatGPT and its friends can impressive. Tip Before move other models, important clean the GPU RAM, else.
Retype any code from memory and diff.
This stretch centers on BERT, Transformer. Source cue: tasks, just fine-tuning fairly small dataset for each task. Let start looking BERT architecture, then look how was.
Retype any code from memory and diff.
This stretch centers on Transformer, PyTorch, BERT. Source cue: 512 dimensions also uses trainable positional embeddings and supports input sentences 512 tokens. applies layer-norm just before each.
Retype any code from memory and diff.
Ch15 panel 13
Fit discipline.
This stretch centers on BERT. Source cue: representation: segment embedding added all tokens within segment and segment embedding added all tokens within segment theory, could.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: sentence: the loss only computed the mask token outputs. more precise, some the masked tokens are not truly.
Retype any code from memory and diff.
This stretch centers on BERT, BooksCorpus. Source cue: classification task, which the authors chose implement introducing new class token CLS this token inserted the beginning the.
Retype any code from memory and diff.
Ch15 panel 14
Error parts.
This stretch centers on BERT, TransformerEncoder, TransformerEncoderLayer, BertConfig, BertTokenizerFast. Source cue: most later architectures. Chapter 14, saw how use the Transformers library download pretrained BERT model and its tokenizer..
Retype any code from memory and diff.
This stretch centers on BertConfig, BertTokenizerFast, WikiText, TrainingArguments, DataCollatorForLanguageModeling. Source cue: config BertConfig adapt training budget, and dataset size complexity vocab size bert tokenizer.vocab size, hidden size 128, num.
Retype any code from memory and diff.
This stretch centers on TrainingArguments, DataCollatorForLanguageModeling, BERT. Source cue: Trainer, and call its train method: from transformers import Trainer, TrainingArguments from transformers import DataCollatorForLanguageModeling args TrainingArguments output.
Retype any code from memory and diff.
Ch15 panel 15
Train vs serve.
This stretch centers on BERT. Source cue: better off downloading model that was pretrained text corpus close possible yours, then fine-tuning your own dataset. This.
Retype any code from memory and diff.
This stretch centers on BERT. Source cue: new classification head Using the exact same approach, you can tackle other sentence classification tasks. For example, the.
Retype any code from memory and diff.
This stretch centers on BERT. Source cue: subjects, nouns, and verbs this part-of-speech tagging locating questions, statements, and greetings this dialogue act tagging and more..
Retype any code from memory and diff.
Ch15 panel 16
Model promotion.
This stretch centers on CrossEntropyLoss, BERT. Source cue: each answer, the class token output passed through linear layer with single unit, producing score. Once have all.
Retype any code from memory and diff.
This stretch centers on BERT, BertForSequenceClassification, BertForQuestionAnswering. Source cue: determines whether the last. course most tokens are neither, and possible for one token both the answer single.
Retype any code from memory and diff.
This stretch centers on BERT, PyTorch. Source cue: sentences dataset containing sentences, you will need run BERT pairs: this could take hours the dataset large. Instead,.
Retype any code from memory and diff.
Ch15 panel 17
Study loop.
This stretch centers on PCA. Source cue: documents through SBERT obtain their sentence embeddings, then apply clustering algorithm such k-means HDBSCAN see Chapter the embeddings.
Retype any code from memory and diff.
This stretch centers on SentenceTransformer, BERT. Source cue: for documents based their embeddings. More traditional databases, such PostgreSQL MongoDB, also have growing support for embeddings, although.
Retype any code from memory and diff.
This stretch centers on BERT. Source cue: shopping more similar She bought some shoes the cosine similarity 0.6328 than She working 0.5841 Now that examined.
Retype any code from memory and diff.
Ch15 panel 18
This stretch centers on BERT. Source cue: used dynamic masking and dropped NSP, were following RoBERTa pretraining approach. DistilBERT Hugging Face, October 2019 66M This.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: settling somewhere between. result, distillation often works better than training the student from scratch the same dataset the.
Retype any code from memory and diff.
This stretch centers on BERT, GPU. Source cue: training loss also had two more components: the standard MLM loss, well cosine embedding loss which minimizes the.
Retype any code from memory and diff.
Ch15 panel 19
Ops contrast.
This stretch centers on GPU, BERT. Source cue: embeddings reduce the size the embedding layer: BERT-large, the vocabulary size about 30,000, and the embedding size 1,024,.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: two consecutive sentences, the goal predict which one comes first. This much harder task than NSP, and led.
Retype any code from memory and diff.
This stretch centers on BERT. Source cue: discriminator must then try predict since the first and third tokens are the same the original text, but.
Retype any code from memory and diff.
Ch15 panel 20
Practice hub.
This stretch centers on BERT. Source cue: much the ith query token should attend the jth key token, the model has access learned embedding for.
Retype any code from memory and diff.
This stretch centers on BERT. Source cue: for major Indian languages Pretrained cased uncased text Tweaked for specific tasks e.g., BERT for question answering You.
Retype any code from memory and diff.
This stretch centers on BERT, AdapterHub, Transformer. Source cue: distillation, weight-sharing, and/or other techniques such quantization see Appendix Popular examples include: DistilBERT, TinyBERT, MobileBERT, MiniLM available for.
Retype any code from memory and diff.
Ch15 panel 21
Fit discipline.
This stretch centers on Transformer, BERT, GPT. Source cue: scratch, and now looked into encoder-only models like BERT and how they can used for many different NLU.
Retype any code from memory and diff.
This stretch centers on GPT, Transformer. Source cue: chatbots like ChatGPT Claude. The GPT model now known GPT-1 was released June 2018. GPT stands for Generative.
Retype any code from memory and diff.
This stretch centers on GPT. Source cue: perform text classification quite well, but encoder-only models shine this area, they are faster and often provide similar.
Retype any code from memory and diff.
Ch15 panel 22
Error parts.
This stretch centers on GPT, BERT, Transformer, PyTorch, TransformerDecoder. Source cue: answer questions. GPT-1 Architecture and Generative Pretraining During pretraining, GPT-1 was fed batches sequences randomly sampled from the.
Retype any code from memory and diff.
This stretch centers on PyTorch, TransformerDecoder, TransformerEncoder, GPT. Source cue: has attention heads instead That total 117 million parameters. Warning Counterintuitively, you cannot use PyTorch nn.TransformerDecoder module build.
Retype any code from memory and diff.
This stretch centers on BERT, GPT. Source cue: architecture: For text classification tasks, classification head added top the last token output embedding. See the righthand side.
Retype any code from memory and diff.
Ch15 panel 23
Train vs serve.
This stretch centers on GPT, WebText. Source cue: the answer scores are then passed through softmax layer. all cases they added start-of-sequence token and end-of-sequence token..
Retype any code from memory and diff.
This stretch centers on GPT. Source cue: ranked Reddit pages. Most importantly, GPT-2 performed incredibly well many tasks without any fine-tuning: this called zero-shot learning.
Retype any code from memory and diff.
This stretch centers on GPT, WebCrawl. Source cue: was just matter training large enough transformer Note GPT-2 performance was impressive that OpenAI initially chose not release.
Retype any code from memory and diff.
Ch15 panel 24
This stretch centers on GPT. Source cue: just few examples. This called few-shot learning FSL one-shot learning OSL there single example. tackle FSL OSL tasks,.
Retype any code from memory and diff.
This stretch centers on GPT, AutoTokenizer, GPU. Source cue: will play with GPT-3 via the API later this chapter Using GPT-2 Generate Text you might expect, can.
Retype any code from memory and diff.
This stretch centers on GPU, GPT. Source cue: for the model weights, based what available the model checkpoint and your hardware. Typically, loads the model using.
Retype any code from memory and diff.
Ch15 panel 25
This stretch centers on GPT. Source cue: lastly decodes the resulting token IDs return nice string containing the extended text. Since GPT-2 was pretrained without.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: nThe unicorn was found field the northern part the state New Mexico. nThe unicorn was found field Hmm,.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: there weren't solitary unicorn the land. Today, there are around 1,000. Well, that certainly less repetitive get better.
Retype any code from memory and diff.
Ch15 panel 26
This stretch centers on GPT. Source cue: the most likely cities but top-k sampling will only sample from the few most likely ones, ignoring many.
Retype any code from memory and diff.
This stretch centers on GPT. Source cue: that the prompt template includes one example the task help GPT-2 understand what and what format expect: that.
Retype any code from memory and diff.
This stretch centers on Transformer. Source cue: sure, just repeats the country name, roughly the time. This might because several countries have capital city the.
Retype any code from memory and diff.
Ch15 panel 27
This stretch centers on GPU. Source cue: increase its speed and performance. The good news that released under the permissive Apache 2.0 license, and not.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: model, including code snippets and more. For this particular model, you should also see the message asking you.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: gets compromised. You would select the Fine-grained option see Figure 15-15 then scroll down the Repositories permissions section,.
Retype any code from memory and diff.
Ch15 panel 28
This stretch centers on Ch15 ideas. Source cue: old token and replaces with new one, keeping just the token name. These measures are especially important for.
Retype any code from memory and diff.
This stretch centers on GPT, AutoTokenizer. Source cue: column your secret give the current notebook access your secret. This button always deactivated default, you will need.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: find capital cities, did earlier, you will see that finds the correct answer for almost all countries the.
Retype any code from memory and diff.
Ch15 panel 29
This stretch centers on Ch15 ideas. Source cue: until the model reliably behaves you want to. For example, can try adding introduction that should make the.
Retype any code from memory and diff.
This stretch centers on BobTheChatbot, ChatBob. Source cue: then generated the rest the conversation. That not too hard fix; can simply drop anything after Bob first.
Retype any code from memory and diff.
This stretch centers on BobTheChatbot, ChatBob. Source cue: new prompt: class BobTheChatbot: ChatBob you prefer def init self, model, tokenizer, introduction bob introduction, max answer length.
Retype any code from memory and diff.
Ch15 panel 30
This stretch centers on BobTheChatbot. Source cue: stops whenever the conversation goes back Me: when the answer empty becomes way too long. OK, time chat.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: fall into loops. For example, you ask Tell jokes will repeat the same joke five times: What you.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: words, add some context, give few examples, suggest character imitate e.g., You are friendly real-estate expert specify the.
Retype any code from memory and diff.
Ch15 panel 31
This stretch centers on Ch15 ideas. Source cue: improves the quality the result significantly, and can fully automated e.g., generate lessons any other topic related technique.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: several options. Then ask the LLM evaluate each option, and explore the most promising one, continuing recursively until.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: prompt. This called retrieval augmented generation RAG and dramatically improves the LLM reliability and reduces hallucinations will get.
Retype any code from memory and diff.
Ch15 panel 32
This stretch centers on GPT. Source cue: data extraction attack get individual personal information, assuming some was leaked online and ended the base model training.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: both. For example, Mistral-7B-Instruct was fine-tuned starting from Mistral-7B both conversational and follow instructions. Note note terminology: base.
Retype any code from memory and diff.
This stretch centers on GPT, DeepMind, PPO. Source cue: training process just regular supervised learning using next token prediction. However, common compute the loss only the answer.
Retype any code from memory and diff.
Ch15 panel 33
This stretch centers on PPO. Source cue: RLHF involves training reward model predict human preferences, then fine-tuning the LLM using PPO favor answers that the.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: elements: prompt and two possible answers, where one preferred human raters. The goal make the model more likely.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: given the current model parameters and frozen reference model. the prompt, the chosen answer, and the rejected answer..
Retype any code from memory and diff.
Ch15 panel 34
This stretch centers on Ch15 ideas. Source cue: result and run through the model get the output logits. typically this simultaneously for both the correct and.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: last one since doesn have next token next token ids encodings.input ids shape log probas F.log softmax logits,.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: last two dimensions the input tensor, since F.cross entropy expects the class dimension dimension Now let inspect each.
Retype any code from memory and diff.
Ch15 panel 35
This stretch centers on Ch15 ideas. Source cue: correct. Now add the log probabilities all answer tokens e.g., for Buenos and Aires get the estimated log.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: extra for the chosen answer cancels out exactly with the extra for the rejected answer. only need mask.
Retype any code from memory and diff.
This stretch centers on Ch15 ideas. Source cue: full input ref sum log probas ref model, tokenizer, full input return -F.logsigmoid beta ref ref .mean You.
Retype any code from memory and diff.
Ch15 panel 36
This stretch centers on OpenAssistant. Source cue: red, blue, and yellow. you can see, the goal this dataset train the model follow single instruction and.
Retype any code from memory and diff.
This stretch centers on GPT. Source cue: input The text field will used for training, let use the individual components compose new text field and.
Retype any code from memory and diff.
This stretch centers on OpenAssistant. Source cue: sft trainer.model.save pretrained sft model dir Now the DPO phase. need human-preference dataset. can use Anthropic Anthropic/hh-rlhf dataset,.
Retype any code from memory and diff.
Ch15 panel 37
This stretch centers on BobTheChatbot. Source cue: can right ahead with the second phase fine-tuning, using DPO: from trl import DPOConfig, DPOTrainer dpo model dir.
Retype any code from memory and diff.
Closing
Transformers dominate NLP. Own the block (attention, FFN, residual, norm) before you chase agents.
16. Vision and Multimodal Transformers
ViT, multimodal models like CLIP-style encoders, and vision-language patterns that fuse pixels and text.
ViT path.
Ch16 panel 1
Study loop.
This stretch centers on ViTs, Transformer, ViT. Source cue: and Multimodal Transformers the previous chapter, implemented transformer from scratch and turned into translation system, then explored encoder-only.
Retype any code from memory and diff.
This stretch centers on Transformer, ViT, BERT, CLIP. Source cue: Transformer more data-efficient ViT trained scale using distillation. PVT Pyramid Vision Transformer hierarchical model that can produce multiscale.
Retype any code from memory and diff.
This stretch centers on CNN, Transformer, ViT, ViTs, Attention. Source cue: Flamingo Rather than starting from scratch, reuses two large pretrained models one for vision and one for language.
Retype any code from memory and diff.
Ch16 panel 2
This stretch centers on RNN. Source cue: generates the caption, one token time. The decoder uses attention layer each decoding step focus just the right.
Retype any code from memory and diff.
This stretch centers on CNN, ViTs, Transformer. Source cue: the model focused when output the word wolf You may find that was paying attention not only the.
Retype any code from memory and diff.
This stretch centers on CNN, ViTs, Transformer, ViT. Source cue: for visual tasks, usually don consider them ViTs. The detection transformer DETR good example this. DETR: CNN-Transformer Hybrid.
Retype any code from memory and diff.
Ch16 panel 3
Ops contrast.
This stretch centers on CNN, ViT, BERT, ImageNet. Source cue: was simply named the vision transformer ViT The idea surprisingly simple: chop the image into little patches, and.
Retype any code from memory and diff.
This stretch centers on BERT, ImageNet, ViT. Source cue: top the corresponding output i.e., this BERT-style classification And that This model beat the state the art ImageNet.
Retype any code from memory and diff.
This stretch centers on ViT, PyTorch, PatchEmbedding, TransformerEncoder. Source cue: But the implicit assumptions are wrong, then the model may perform poorly even trained large dataset.
Retype any code from memory and diff.
Ch16 panel 4
Practice hub.
This stretch centers on TransformerEncoder, ViT, PatchEmbedding, Dropout, TransformerEncoderLayer. Source cue: dimensions and transpose the last two dimensions ensure the embedding dimension ends last, which what the nn.TransformerEncoder module.
Retype any code from memory and diff.
This stretch centers on PatchEmbedding, Dropout, TransformerEncoder, TransformerEncoderLayer, LayerNorm. Source cue: logits Let through this code: The constructor starts creating the PatchEmbedding module. Then creates the class token trainable.
Retype any code from memory and diff.
This stretch centers on ViT, CrossEntropyLoss, AutoImageProcessor. Source cue: each sequence patch tokens starts with the class token. The rest straightforward: add the positional embeddings, apply some.
Retype any code from memory and diff.
Ch16 panel 5
Fit discipline.
This stretch centers on ViT, AutoImageProcessor, ImageNet, PyTorch. Source cue: First, let download the dataset: from datasets import load dataset pets load dataset timm/oxford-iiit-pet Next, let download the.
Retype any code from memory and diff.
This stretch centers on PyTorch, TrainingArguments, ViT. Source cue: returns dictionary containing pixel values entry equal the preprocessed image. Next, need data collator that will preprocess all.
Retype any code from memory and diff.
This stretch centers on ViT, ViTs, Transformer, ImageNet, CNN. Source cue: else will dropped, including the image attribute. Since the unused attributes are dropped before the collate function called,.
Retype any code from memory and diff.
Ch16 panel 6
Error parts.
This stretch centers on ViT, CNN, ViTs. Source cue: distillation was introduced Chapter The authors used frozen, state-of-the-art CNN the teacher model. During training, they added special.
Retype any code from memory and diff.
This stretch centers on ViT, ViTs, Transformer, SenseTime, CNN. Source cue: 16-4. Data-efficient image transformer DeiT ViT distillation far, have only used ViTs for classification tasks, but what about.
Retype any code from memory and diff.
This stretch centers on CNN, Transformer, ViT. Source cue: with channels RGB first turned into image with channels, then into image with 128 channels, then image with.
Retype any code from memory and diff.
Ch16 panel 7
Train vs serve.
This stretch centers on ViT. Source cue: chopped into grid patches and 3,072 Each patch token represented 64-dimensional vector. The encoder also outputs 3,072 vectors.
Retype any code from memory and diff.
This stretch centers on ViT. Source cue: why the PVT authors introduced spatial reduction attention SRA just like MHA except that the keys and values.
Retype any code from memory and diff.
This stretch centers on CNN, ImageNet. Source cue: tokens representing grid because and only need compute 3,072 147,456 attention scores: that times less computationally expensive. And.
Retype any code from memory and diff.
Ch16 panel 8
Model promotion.
This stretch centers on CNN, Transformer, ViT. Source cue: and add final classification head output one class per pixel. Similarly, can use Mask R-CNN for object detection.
Retype any code from memory and diff.
This stretch centers on Transformer, ViT. Source cue: patches, and these patches are grouped into nonoverlapping windows. the first level the Swin pyramid, the patches are.
Retype any code from memory and diff.
This stretch centers on Transformer. Source cue: result, these models are way too slow for very large images, meaning you must first downsample the image,.
Retype any code from memory and diff.
Ch16 panel 9
Study loop.
This stretch centers on Ch16 ideas. Source cue: now the same window, they can see each other. alternating W-MSA and SW-MSA, information from any part the.
Retype any code from memory and diff.
This stretch centers on ViTs, ViT. Source cue: are within the same window, appropriate attention mask must applied. Overall, Swin harder implement than PVT, but its.
Retype any code from memory and diff.
This stretch centers on Ch16 ideas. Source cue: duplicated during training, with one network acting teacher and the other acting student see Figure 16-7 Gradient descent.
Retype any code from memory and diff.
Ch16 panel 10
This stretch centers on Ch16 ideas. Source cue: strong risk mode collapse. This when both the student and the teacher always output the exact same thing,.
Retype any code from memory and diff.
This stretch centers on ImageNet. Source cue: DINO, self-distillation with labels After training, you can drop the teacher: the student the final DINO model. you.
Retype any code from memory and diff.
This stretch centers on TokenCut, CNN, ViT. Source cue: each attention head seems focus different part the object, you can see Figure 16-8. See the notebook for.
Retype any code from memory and diff.
Ch16 panel 11
Ops contrast.
This stretch centers on ViTs, ImageNet, BERT. Source cue: transformer models and techniques. Other Major Vision Models and Techniques Progress vision transformers has continued steadily this day..
Retype any code from memory and diff.
This stretch centers on Ch16 ideas. Source cue: come from Well, the original image passed through discrete variational autoencoder dVAE, see Chapter which encodes each patch.
Retype any code from memory and diff.
This stretch centers on ViTs, ImageNet, LeCun. Source cue: their weights create new and improved model. This similar ensemble see Chapter except there just one model the.
Retype any code from memory and diff.
Ch16 panel 12
Practice hub.
This stretch centers on EfficientFormer, FocalNet, InternImage, OneFormer. Source cue: predictor: the teacher encoder sees the full input e.g., photo cat while the student encoder sees only part.
Retype any code from memory and diff.
This stretch centers on FocalNet, InternImage, OneFormer, TimeSformer, ToMe. Source cue: for mobile devices Hierarchical transformers like Twins-SVT, FocalNet, MaxViT, and InternImage, often used backbones for dense prediction tasks.
Retype any code from memory and diff.
This stretch centers on Ch16 ideas. Source cue: had transformers that could read and write and chat and now have vision transformers that can see. time.
Retype any code from memory and diff.
Ch16 panel 13
Fit discipline.
This stretch centers on Ch16 ideas. Source cue: their voice while also watching the movement their lips: these two modalities auditory and visual carry overlapping information,.
Retype any code from memory and diff.
This stretch centers on Ch16 ideas. Source cue: have recording speech, well text transcription, and you want find the timestamp each word. you want find the.
Retype any code from memory and diff.
This stretch centers on Transformer, BERT, YouTube. Source cue: ways: summed up, concatenated, passed through fusion encoder, and more. This can take care the fusion problem. And.
Retype any code from memory and diff.
Ch16 panel 14
Error parts.
This stretch centers on YouTube, CNN, LSTM, BERT. Source cue: video training set. This dataset was built from large collection instructional YouTube videos, particularly cooking videos. These videos.
Retype any code from memory and diff.
This stretch centers on YouTube, LSTM, BERT. Source cue: speech-to-text software, after which they dropped the audio tracks from the videos. Then they separated the text into.
Retype any code from memory and diff.
This stretch centers on BERT. Source cue: clusters. This process called hierarchical k-means, and much faster than running k-means just once using 20,736, plus typically.
Retype any code from memory and diff.
Ch16 panel 15
Train vs serve.
This stretch centers on BERT. Source cue: three different training regimes: text-only, video-only, and text plus video. text-only and video-only modes, VideoBERT was fed single.
Retype any code from memory and diff.
This stretch centers on Ch16 ideas. Source cue: later, the authors concatenated random neighboring sentences give the model more context. The authors had few more tricks.
Retype any code from memory and diff.
This stretch centers on BERT. Source cue: and cake boil and egg Video captioning Given video clip, generate caption. this, the authors used the earliest.
Retype any code from memory and diff.
Ch16 panel 16
Model promotion.
This stretch centers on BERT, Transformer, CNN. Source cue: the highest score. The success VideoBERT inspired many other BERT-based multimodal transformers, many which were released August and.
Retype any code from memory and diff.
This stretch centers on CNN, BERT. Source cue: the visual features come from deep CNN, then already have good high-level visual features, whereas the text will.
Retype any code from memory and diff.
This stretch centers on BERT, CNN. Source cue: text encoder are initialized with BERT weights the authors used BERT base, which has layers and co-attention layers.
Retype any code from memory and diff.
Ch16 panel 17
Study loop.
This stretch centers on BERT, CNN. Source cue: are computed like this: each region bounding box encoded vector containing the normalized upper-left and lower-right coordinates, and.
Retype any code from memory and diff.
This stretch centers on CNN. Source cue: the CNN predicts for the given image region this soft target The authors chose this task rather than.
Retype any code from memory and diff.
This stretch centers on CLIP. Source cue: justification. For example, given image waiter serving some pancakes table, along with the question Why person pointing person.
Retype any code from memory and diff.
Ch16 panel 18
This stretch centers on CLIP. Source cue: model learns encode text and images into vector representations that are similar when the text and image match,.
Retype any code from memory and diff.
This stretch centers on CLIP. Source cue: pretrained models are available the Hugging Face Hub, and the Transformers library provides convenient pipeline for zero-shot image.
Retype any code from memory and diff.
This stretch centers on CLIP, ResNet, ViT, GPT, BERT. Source cue: high confidence ignoring the ladybug Impressive how does this magic work Well, CLIP architecture based regular text encoder.
Retype any code from memory and diff.
Ch16 panel 19
Ops contrast.
This stretch centers on GPT, CLIP. Source cue: was chosen: the authors simply had more experience with this model and good training infrastructure already place. Using.
Retype any code from memory and diff.
This stretch centers on Ch16 ideas. Source cue: authors used shockingly large batch size 215 32,768 during training. The model was then pretrained large dataset 400.
Retype any code from memory and diff.
This stretch centers on Ch16 ideas. Source cue: Since the ith image corresponds the ith caption, want the main diagonal this matrix contain similarity scores close.
Retype any code from memory and diff.
Ch16 panel 20
Practice hub.
This stretch centers on CLIP, ResNet, ImageNet. Source cue: score class logit and simply compute the cross-entropy loss for that row with the target. can follow the.
Retype any code from memory and diff.
This stretch centers on ResNet, ImageNet, CLIP, ViTs. Source cue: performed better without any extra training i.e., zero-shot than linear probe trained ResNet-50 features that linear classifier trained.
Retype any code from memory and diff.
This stretch centers on CLIP. Source cue: you can then search for them given either text query image query. For this, just run the query.
Retype any code from memory and diff.
Ch16 panel 21
Fit discipline.
This stretch centers on CLIP. Source cue: must divide them features.norm dim keepdim True see the notebook for code example The features are already normalized,.
Retype any code from memory and diff.
This stretch centers on CLIP, GPT. Source cue: the result through the softmax function nice see that get the same result the pipeline.
Retype any code from memory and diff.
This stretch centers on CLIP, diffusion. Source cue: get the full image. The visual tokens are generated dVAE model, which takes image and outputs sequence tokens.
Retype any code from memory and diff.
Ch16 panel 22
Error parts.
This stretch centers on diffusion, CLIP, GPT, ViTs. Source cue: fully shifted away from its initial openness: there was peer-reviewed paper, code, weights, data. Like the previous version,.
Retype any code from memory and diff.
This stretch centers on ViTs. Source cue: the Perceiver. Perceiver: Bridging High-Resolution Modalities with Latent Spaces Every transformer far has required chopping the inputs into.
Retype any code from memory and diff.
This stretch centers on DeepMind, Transformer. Source cue: performance using unbiased models with very few implicit assumptions. Sure, the model will have figure out its own.
Retype any code from memory and diff.
Ch16 panel 23
This stretch centers on Transformer. Source cue: uses Fourier positional encodings, which are very similar the sinusoidal positional encodings the original Transformer, except they encode.
Retype any code from memory and diff.
This stretch centers on RNN. Source cue: Perceiver dimensionality e.g., 512 The Perceiver architecture itself composed repeated processing blocks e.g., eight where each block composed.
Retype any code from memory and diff.
This stretch centers on Ch16 ideas. Source cue: short sequence, doesn suffer much from the quadratic attention problem. This called the latent bottleneck trick, and the.
Retype any code from memory and diff.
Ch16 panel 24
This stretch centers on ImageNet, AudioSet. Source cue: regular encoder, each self-attention layer would have compute 50,1762 2.5 billion attention scores But since the Perceiver only.
Retype any code from memory and diff.
This stretch centers on Ch16 ideas. Source cue: pixel and each audio frame individually the Perceiver, but this would bit extreme, each 10s video would represented.
Retype any code from memory and diff.
This stretch centers on DeepMind, BERT, StarCraft. Source cue: that the Perceiver able extract useful features from the raw data without any help. Then they simply concatenated.
Retype any code from memory and diff.
Ch16 panel 25
This stretch centers on StarCraft. Source cue: Perceiver the output latent tokens, but the pooling layer and the classification head are replaced very flexible output.
Retype any code from memory and diff.
This stretch centers on Ch16 ideas. Source cue: position. For example, given the masked sentence The dog MASK the MASK the masked tokens are located positions.
Retype any code from memory and diff.
This stretch centers on DeepMind. Source cue: large number output query tokens. The latent bottleneck allows the model scale linearly for both the inputs and.
Retype any code from memory and diff.
Ch16 panel 26
This stretch centers on DeepMind. Source cue: trained using next token prediction. Perceiver didn quite have the same impact Perceiver and Perceiver IO, but got.
Retype any code from memory and diff.
This stretch centers on CLIP. Source cue: few photos along with the corresponding license plate numbers text then add new car photo, and Flamingo will.
Retype any code from memory and diff.
This stretch centers on Transformer. Source cue: fairly short sequence latent representations typically much shorter than the output the vision model This works around the.
Retype any code from memory and diff.
Ch16 panel 27
This stretch centers on DeepMind, OpenFlamingo. Source cue: outputs. the gated xattn-dense module, each text token can only attend visual tokens from the closest image located.
Retype any code from memory and diff.
This stretch centers on DeepMind, OpenFlamingo, CLIP, ViT, AudioFlamingo. Source cue: Flamingo the public. The good news that open source replications and variants are available: OpenFlamingo, created the MLFoundations.
Retype any code from memory and diff.
This stretch centers on AudioFlamingo, OpenFlamingo, CLIP. Source cue: very similar Flamingo but handles audio instead images. Other variants are available, such domain-specific models like Med-Flamingo, OpenFlamingo.
Retype any code from memory and diff.
Multimodal align.
Closing
Patches become tokens; text and images share spaces. Multimodal eval needs both retrieval and generation tests.
17. Speeding Up Transformers
The stub promises KV cache, speculative decoding, parallel decode, sparse/approximate attention, FlashAttention, MoE, PEFT/LoRA, activation checkpointing, sequence packing, gradient accumulation, and parallelism.
LoRA path.
MoE pieces.
Attention kernels.
Why transformers get slow
Study loop.
Self-attention is O(T²) in sequence length for the naive dense form, and autoregressive decode is memory-bandwidth bound: each new token re-reads keys and values for the whole prefix. Training gigantic models is parameter- and activation-memory bound. The stub's agenda attacks all three: decode, attention kernels, sparse compute via MoE, and efficient training via PEFT plus systems tricks.
Measure before you optimise. Prefill (prompt) and decode (token-by-token) have different bottlenecks. A trick that helps decode may not help training throughput. Bank and EA deployments usually care about decode latency at modest batch sizes, not about training MFU on a research cluster.
KV caching for decode
KV cache lifecycle.
During autoregressive generation, the query at step t only needs keys/values for positions 1..t. Recomputing K and V for the whole prefix every step wastes compute. A KV cache stores past K/V per layer and appends the new slice each step. Attention becomes "query the new token against the cached prefix."
Memory cost grows with batch × layers × heads × dim × sequence. Long contexts blow VRAM; remedies include paged KV caches (vLLM-style block tables), quantized KV (8-bit/4-bit caches), multi-query / grouped-query attention that shrinks K/V heads, and sliding-window or sink-style caches that drop far tokens under a policy.
Implementation sketch: on step 1, run full prefill and materialise K/V. On later steps, run a single-token forward that reads cache and writes one new column. Frameworks expose this via `past_key_values` / `use_cache=True` in Hugging Face, or via dedicated kernels in FlashInfer and TensorRT-LLM. Never mix caches across different requests without isolation.
Failure modes: off-by-one positions, wrong causal masks when appending, cache not invalidated on beam changes, and speculative drafts that require KV rollback when tokens are rejected.
Speculative decoding
Ops contrast.
Speculative decode.
Speculative decoding uses a cheap draft model (or draft head) to propose several tokens; the target model verifies them in one parallel forward. Accepted tokens advance; the first rejection falls back to the target's own sample. Correctness matches sampling from the target when verification is exact.
Classic forms: smaller draft LM; EAGLE-style draft heads that predict from target hidden states; Medusa-style multi-head drafts. Oct 2026 practice includes EAGLE-2/3 recipes, tree-structured drafts, and serving stacks that fuse speculative sampling into the engine (FlashInfer lists chain speculative sampling; NeMo AutoModel documents EAGLE recipes for Llama/Qwen/MoE targets).
Speedup depends on acceptance rate. High acceptance needs a draft that matches the target's distribution. MoE targets need careful draft KV alignment and cache rollback. Measure tokens/sec versus baseline decode at your batch size and temperature; synthetic microbenchmarks lie.
Bank/EA caution: speculative decoding is an inference optimisation, not a safety control. Guardrails and retrieval still apply to accepted text.
Parallelising text generation
Practice hub.
Batch requests to amortise kernel launches. Continuous batching / paged attention packs sequences of different lengths. Tensor parallel shards weights across GPUs for large models; pipeline parallel shards layers; sequence/context parallel shards the sequence dimension for long contexts. Speculative tree drafts add another parallelism axis inside one sequence.
For single-user interactive decode, batching across users matters more than micro-optimising one stream. For offline jobs, raise batch until throughput plateaus or VRAM ends.
Making attention cheaper
Fit discipline.
Sparse attention patterns (local windows, strided, block-sparse) cut the O(T²) edges. Approximate attention (low-rank, kernelised, hashing) trades exactness for speed. Sharing projections (multi-query attention, grouped-query attention, shared K/V across heads) cuts KV memory and bandwidth - often the real decode win.
These are architectural choices. You cannot bolt arbitrary sparsity onto a checkpoint trained dense without retraining or careful distillation. Prefer checkpoints that already ship GQA/MQA when decode memory dominates.
FlashAttention and fused kernels
Error parts.
FlashAttention (Dao et al.) fuses attention into tiles that keep softmax stats in SRAM, reducing HBM traffic. Versions progressed through FA2 and FA3; serving stacks also route through cuDNN, CUTLASS, TensorRT-LLM, and FlashInfer backends that pick kernels by GPU generation and workload (prefill vs decode vs mixed).
Since publication FlashInfer advertises paged/ragged KV, FP8/FP4 paths, MLA support, and speculative sampling hooks. Hugging Face diffusers blogs show FA3 + `torch.compile` + TorchAO FP8 recipes for diffusion transformers with LoRA hot-swap caveats. Enable the fused SDPA backends in PyTorch (`scaled_dot_product_attention`) before you write custom CUDA.
A closer look: FlashAttention does not change the math you owe examiners; it changes the memory IO. If your sequence is short and batch is tiny, gains shrink. Always A/B latency and numerics on a golden batch.
Mixture of Experts (MoE)
Train vs serve.
MoE layers route each token to a small set of expert FFNs (top-k gating). Parameter count soars while FLOPs per token stay closer to a dense model of expert-width size. Training needs load-balancing losses so experts do not collapse; serving needs expert parallelism and fused MoE kernels.
Oct 2026: fused MoE kernels and FP8/FP4 expert weights appear in FlashInfer and vendor stacks; QLoRA-on-MoE tooling exists for single-GPU adapter training with expert offload (e.g. 4-bit expert packs). DeepSeek-style MLA and MoE routing variants are in the wild - treat paper names as checkpoints to evaluate, not as permanent APIs.
Ops reality: MoE improves quality-per-FLOP when routing works; it complicates deployment (expert placement, network fabric, tail latency). Do not adopt MoE because a blog said "trillion parameters"; adopt it when eval and cost curves beat your dense baseline.
PEFT and LoRA / QLoRA
Model promotion.
Parameter-efficient fine-tuning (PEFT) updates a small adapter instead of all weights. LoRA freezes W and learns low-rank factors BA on selected projections (often attn q/v, sometimes FFN). Merge adapters for serve or keep them hot-swappable per tenant.
QLoRA freezes 4-bit quantized base weights and trains LoRA in higher precision - the workhorse for single-GPU fine-tunes. Oct 2026: LoRA + FA3 + compile + FP8 recipes exist; hot-swapping adapters can force `torch.compile` recompilation unless you fuse/unfuse carefully or use hot-swap-aware kernels. PEFT library + bitsandbytes / TorchAO remain common entry points - pin versions.
Bank/EA pattern: one base model, many LoRA adapters (product line, locale, risk policy). Store adapters in the model registry; evaluate each on its own valid set; never silently merge untested adapters into the production base.
Rank, alpha, dropout, and target modules are hyperparameters. Start with rank 8-64 on attention projections; expand if underfit. Full fine-tunes still win sometimes on small models or when every layer must move.
Activation checkpointing, packing, accumulation
Study loop.
Use checkpointing when activations dominate (long context, large batch). Use packing when many short prompts waste pads. Use accumulation when you need a larger effective batch for optimiser stability. None of these replace a wrong learning rate.
Parallelism taxonomy
Parallel picks.
Data parallel (DDP/FSDP): shard batches; FSDP shards parameters/gradients/optimizer state. Tensor parallel: shard matmuls inside layers. Pipeline parallel: shard layers across stages with microbatches. Context / sequence parallel: shard long sequences. Expert parallel: shard MoE experts. Real clusters combine these (3D/4D parallel).
Pick the smallest parallelism that fits. A 7B model often needs only DDP or single-GPU QLoRA. A 70B+ model may need FSDP or tensor parallel. Measure communication overhead; bad partitioning spends all day in NCCL.
Quantization pointer
Ops contrast.
The stub correctly defers reduced precision and quantization to Appendix B. In 2026 practice, weight-only INT8/INT4, AWQ/GPTQ-style LLM quant, FP8 training/serve, and quantized KV caches are part of the same speed story. Read App B next; do not pretend BF16 is free forever.
Field card
Practice hub.
| Lever | Helps | Watch out |
|-------|-------|-----------|
| KV cache | Decode | Memory growth, rollback |
| Speculative decode | Tokens/sec | Acceptance rate |
| GQA/MQA | KV bandwidth | Need matching ckpt |
| FlashAttention/SDPA | HBM traffic | Short-seq gains |
| MoE | Quality/FLOP | Serving complexity |
| LoRA/QLoRA | Train cost | Hot-swap/compile |
| FSDP/TP/PP | Fit/scale | Comm overhead |
| Quant (App B) | Memory/latency | Quality floors |
What the stub promised versus what we added
Fit discipline.
Promised and covered: KV cache, speculative decoding, parallel generation notes, sparse/approximate/shared attention, FlashAttention, MoE, PEFT/LoRA, checkpointing, packing, accumulation, parallelism, plus App B pointer.
Added beyond the stub's sentences: paged KV, GQA, EAGLE-style drafts, FlashInfer/FA3 ecosystem, LoRA hot-swap versus compile, MoE quant/offload, bank/EA adapter registries, and Oct 2026 version pins. If Géron's full online chapter appears later with different API names, prefer his worked examples for book exercises and keep this fill for systems context.
Bank and EA takes
Error parts.
Bank: LoRA adapters per product policy; paged-KV serving for chat with retrieval; never speculative-decode your way around content filters. EA: compile + SDPA on internal 7B helpers; QLoRA on a single GPU for runbook fine-tunes; measure p95 latency, not only tokens/sec averages.
Pitfalls
1. Growing KV without paging until OOM. 2. Speculative drafts without cache rollback. 3. Enabling FlashAttention but failing numerics checks. 4. MoE without load balance. 5. LoRA hot-swap thrashing compile caches. 6. FSDP when single-GPU QLoRA would do. 7. Ignoring App B quant. 8. Claiming speedups without a baseline at the same batch and context length.
Test yourself
What does a KV cache store and why does decode need it? How can speculative decoding stay distributionally faithful to the target? FlashAttention's main systems win? MoE's FLOP versus parameter story? LoRA versus full fine-tune? FSDP versus tensor parallel? When packing beats naive padding?
KV cache deep dive
Prefill writes a full K and V tensor per layer for the prompt. Decode reads those tensors and appends one position per new token. The attention score at decode step t is softmax(q_t K_{1:t}^T / sqrt(d)) V_{1:t}. Without a cache you recompute K_{1:t} and V_{1:t} from scratch every step, which is wasteful once T is large.
Shapes: for multi-head attention with H heads and head dim d, cache entries are often stored as (batch, H, T, d) or in a packed paged format with block tables. Grouped-query attention stores fewer K/V heads than Q heads, cutting cache size by the group factor. Multi-query attention is the extreme: one K/V head shared across queries.
Paged KV (vLLM lineage) allocates fixed-size blocks and maps logical token positions to physical blocks. Fragments less than contiguous caches when requests finish at different times. Continuous batching pairs naturally with paging: the engine schedules whatever sequences have work, not a static batch matrix.
Quantized caches store K/V in int8 or fp8 with scales. Quality impact is task-dependent; always measure on your eval suite. Sliding-window attention drops tokens beyond a window; attention sinks keep a few early tokens that models rely on for stability.
API habits: Hugging Face `use_cache=True` returns `past_key_values`. When you implement beams or speculative trees, fork or roll back caches explicitly. A stale cache is a silent correctness bug that looks like nonsense text.
Speculative decoding deep dive
Algorithm: draft proposes γ tokens; target scores those tokens in one forward with a suitably masked batch; accept a prefix until the first rejection; sample the rejection position from the target's adjusted distribution. Expected speedup approximates acceptance rate times draft cost structure.
Draft designs: (1) smaller LM of the same family; (2) EAGLE heads that read target layer features and predict draft tokens without a full second stack; (3) Medusa multi-head drafts on the target trunk; (4) retrieval-inspired or n-gram drafts for narrow domains. Oct 2026 tooling documents EAGLE-1/2/3 recipes across Llama, Qwen, and some MoE targets.
Tree speculative decoding proposes branching candidates and verifies with tree attention masks, raising acceptance opportunities per target forward. Serving engines that fuse this path beat naive Python loops.
Failure modes: mismatched tokenizers between draft and target; temperature or top-p applied inconsistently; KV misalignment after partial accepts; evaluating speedup at batch=1 when production runs batch=16 (or the reverse). Log acceptance rate beside tokens/sec.
Attention kernel deep dive
Naive attention materialises a T×T matrix, blows memory, and streams from HBM repeatedly. FlashAttention tiles Q/K/V, computes softmax in online fashion (running m and l statistics), and writes O without the full score matrix. FA2 improved parallelism; FA3 targets newer NVIDIA generations with still less HBM traffic.
PyTorch `scaled_dot_product_attention` dispatches to fused backends when shapes and dtypes allow. FlashInfer and vendor libraries add paged decode, ragged batches, MLA paths, and low-precision GEMMs. Choose by modality: training prefers stable SDPA; high-QPS serve prefers specialised engines.
Sparse patterns: local window of w tokens is O(T w); block-sparse keeps selected blocks; Linformer-style low-rank projections shrink keys/values along sequence. Approximate methods can change model quality - they are not drop-in free lunches for an already-trained dense checkpoint.
Sharing projections: MQA/GQA cut KV heads. Cross-layer KV sharing is an active research/product area; treat claims as eval homework, not defaults.
MoE deep dive
A dense FFN computes y = W2 φ(W1 x) for every token. An MoE replaces that with E experts and a gate g(x) that picks top-k experts. Output is a weighted mix of expert outputs. Parameters scale with E; FLOPs scale with k × expert size.
Training: auxillary load-balancing losses encourage uniform expert use; without them a few experts win and others die. Capacity factors limit how many tokens an expert may take per step. Dropless MoE variants avoid token drop at extra compute/comms cost.
Serving: expert parallel places experts on ranks; all-to-all moves tokens. Fused kernels reduce launch overhead. Quantised experts (FP8/FP4) and CPU/SSD offload of cold experts appear in 2026 single-GPU fine-tune tooling. Tail latency spikes when routing is imbalanced or fabric is slow.
When to use: quality-per-dollar beats dense on your eval and you can staff the serving complexity. When not to: a 7B-13B dense model already meets the SLO.
LoRA mathematics and practice
For a frozen weight W ∈ R^{d×n}, LoRA learns A ∈ R^{r×n} and B ∈ R^{d×r} with update ΔW = BA (sometimes scaled by α/r). Forward uses Wx + BAx (or merged W'=W+BA at serve). Rank r << min(d,n) keeps trainable params small.
Target modules: query/value projections are the usual start; adding key, out, and FFN can help hard domains. α/r scaling interacts with learning rate - change one at a time. Dropout on the adapter is optional regularisation.
QLoRA stores W in 4-bit (often NormalFloat) with a small quantization computation dtype for matmuls, while adapters stay in bf16/fp16. Double quantization packs scales. Memory win enables 30B-70B class fine-tunes on single GPUs depending on length and rank.
Multi-adapter serve: keep base resident; swap A,B per request or per session. Compile caveat: changing adapter structure can recompile. Fuse for single-tenant max speed; keep unfused for multi-tenant hot-swap.
Evaluation: compare adapter versus base on domain valid sets and on safety/regression suites. A LoRA that wins BLEU but tanks refusal quality is not a win for bank chat.
Systems training bag
Activation checkpointing wraps blocks in `torch.utils.checkpoint.checkpoint` (or HF `gradient_checkpointing_enable`). Expect ~20-30% slowdown for large memory wins - measure. Sequence packing needs attention masks that prevent cross-sample attention; get masks wrong and you leak across packed prompts.
Gradient accumulation: `loss = loss / k; loss.backward()` each micro-step; `opt.step(); opt.zero_grad()` every k. Sync with DDP only on step boundaries when using no_sync contexts. Effective batch = micro_batch × k × world_size.
Most bank/EA fine-tunes should stop at step 1 or 2.
Measurement protocol
Define: model, precision, batch, context length, hardware SKU, engine version. Metrics: time-to-first-token, inter-token latency p50/p95, tokens/sec, VRAM peak, acceptance rate (if speculative), eval quality delta versus baseline. Change one lever at a time. Record in a spreadsheet beside the checkpoint hash.
Microbenchmarks on random tokens miss tokenizer and sampling overhead. Use real prompts from production (redacted) for the final gate.
Worked scenario: 70B bank policy model
QLoRA fine-tune on policy Q&A with retrieval-augmented train rows. Serve with paged KV, GQA checkpoint, INT4 weights, and a draft model for speculative decode if acceptance stays high on policy text. Adapters per jurisdiction. Human eval on adverse-action and hallucination rubrics before any customer traffic.
Sparse and approximate attention catalogue
Local window: each token attends to a neighbourhood - good for long documents with locality. Dilated / strided patterns reach farther with holes. BigBird-style mixes global tokens, windows, and random blocks. Reformer LSH attention hashes queries/keys. Performer kernelises softmax. Longformer popularised sliding windows plus globals for documents.
Practical 2026 stance: for new work, prefer native long-context dense models with GQA and fused kernels, or retrieval that keeps T small, before inventing a sparse pattern. Sparse architectures need training; they are not a free inference patch on a dense checkpoint.
Parallel decode beyond speculation
Non-autoregressive and diffusion language models exist as research/product bets; they change the generation algorithm rather than speeding standard AR decode. Lookahead decoding and Jacobi-style iterative decoding appear in papers - treat as experimental unless your engine supports them productively. For most Insight readers, KV cache + continuous batching + speculative is the boring profitable path.
Adapter ecosystems
besides LoRA: (IA)³, prefix/prompt tuning, adapters (Houlsby), DoRA (weight-decomposed), and full-layer freezes with head-only trains. LoRA won mindshare because it merges cleanly and trains stably. Prompt tuning can win when you cannot touch weights at all. Choose by deployment constraint, not by Twitter.
Version pins: PEFT, transformers, bitsandbytes/TorchAO, and CUDA drivers must cohere. An Oct 2026 pin set that works on one machine can fail on another with an older driver - document the matrix.
Security and tenancy
Hot-swapped LoRAs are a tenancy boundary. Do not let tenant A's adapter remain loaded when serving tenant B. Cache keys must include adapter id. Speculative drafts should not weaken content filters: verify filters on final accepted tokens. Paged KV must not leak blocks across sessions.
Exercise themes for an authored chapter
Implement a toy KV cache on a tiny transformer and prove identical outputs to cacheless decode. Benchmark SDPA versus math attention on fixed shapes. Fine-tune a small model with LoRA and merge weights; confirm logits match unmerged within tolerance. Sketch FSDP wrapping for one block.
Extended oral exam
Derive memory of a KV cache versus T. Explain why GQA reduces bandwidth. Walk speculative accept/reject. What SRAM vs HBM story does FlashAttention tell? Write ΔW=BA. When is MoE the wrong tool? Order parallelism choices for a 7B fine-tune. How does packing corrupt masks if you forget sample boundaries?
Prefill versus decode checklist
Prefill is compute-heavy and parallel across prompt tokens. Decode is bandwidth-heavy and serial across tokens. Optimisations: for prefill, large matmul efficiency, sequence packing, and chunked prefill; for decode, KV cache quality, GQA, quantised cache, speculative accept rate, and continuous batching. A slide that says "we used FlashAttention" without naming which phase is incomplete.
When streaming to users, time-to-first-token is dominated by prefill of the prompt plus system prompt plus retrieved chunks. Shorten prompts before you buy more GPUs. Retrieval that returns 20 chunks of noise hurts more than a slightly slower kernel.
Engine selection notes
Hugging Face `generate` is fine for demos and small loads. vLLM, TensorRT-LLM, Text Generation Inference, and similar engines win for multi-user serve with paged KV and continuous batching. FlashInfer-style kernels show up inside engines more than in training scripts. Choose the engine that matches your ops skill and hardware; switching engines mid-incident is not a plan.
LoRA training recipe (reference)
1. Pick base checkpoint and tokenizer. 2. Freeze base; inject LoRA on target modules. 3. Prepare chat-templated data with loss masks on prompts. 4. Train with AdamW, cosine, modest LR (often 1e-4 to 2e-4 for LoRA). 5. Eval domain plus regression. 6. Merge or store adapter. 7. Red-team. 8. Register. QLoRA adds a 4-bit load step before injection.
MoE routing sketch
Gate logits → softmax or sigmoid → top-k mask → dispatch tokens to experts → combine with gate weights → continue residual stream. Aux loss encourages balance. At serve, monitor per-expert load and p99 latency together; imbalance shows up as tails before averages move.
A closer look box reprise
Géron's stub is honest about being skippable and online-only. Insight fills it because the series requires a complete spine. We may over-weight 2026 serving stack names that will age; the durable ideas are KV reuse, verify-with-target speculation, IO-aware attention, sparse compute via experts, low-rank adaptation, and hierarchical parallelism. Algorithms first, brand names second.
Glossary for the speed stack
- KV cache: stored keys/values for past tokens during decode.
- GQA/MQA: fewer KV heads than query heads.
- Speculative decoding: draft proposes, target verifies.
- FlashAttention / SDPA: IO-aware fused attention kernels.
- MoE: routed experts; many params, fewer FLOPs per token.
- LoRA / QLoRA: low-rank adapters; QLoRA uses 4-bit bases.
- FSDP / TP / PP: parameter shard / matrix shard / layer shard.
- Paged KV: block-table memory manager for concurrent sequences.
One-page decision tree
Latency problem on decode? Check KV, GQA, quant, speculative, engine. VRAM problem on train? Check QLoRA, checkpointing, accumulation, FSDP. Quality problem? Do not expect FlashAttention to fix your data. Cost problem at scale? Consider MoE only after dense baselines and retrieval shrink T.
Prefill versus decode (ops view)
Prefill (prompt processing) is compute-heavy and parallel across the prompt tokens. Decode is memory-bandwidth heavy: each step loads the full KV cache and writes one new column. Throughput SLOs for offline scoring can be prefill-bound. Never quote a single "tokens/s" number without naming the mode.
Paged attention (vLLM-style) and continuous batching keep GPU utilisation high when many short requests share a device. If your bank gateway still does naive static batching, fix serving before you buy another H100.
Speculative decoding and EAGLE-class methods
Speculative decoding drafts k tokens with a cheap model (or draft head), then verifies them in one parallel forward of the target model. Accepted prefixes keep target-quality distribution; rejections fall back. Wall-clock wins need a draft that agrees often and a verify path that is actually fused.
By Since publication, production stacks commonly cite EAGLE / EAGLE-2/3-style draft heads, Medusa-like multi-head drafts, and server integrations in vLLM, TensorRT-LLM, and Hugging Face TGI. Measure acceptance rate on your prompts: code and legalese reject more than chitchat. Speculative decoding does not reduce FLOPs of the target; it reduces serial steps.
FlashAttention, SDPA, and sparse cousins
FlashAttention (and FA3-class kernels) tiles attention so the T×T matrix never fully materialises in HBM. PyTorch SDPA (`scaled_dot_product_attention`) dispatches to flash / mem-efficient / math backends. Prefer SDPA or the vendor kernel; hand-written attention is for teaching.
Sparse / approximate attention (local windows, strided, Linformer-style projections, Reformer hashes) trade exactness for long context. Many 2026 long-context models instead combine exact local attention with retrieval or sliding windows rather than exotic approximations. Sharing K/V projections across heads (MQA/GQA) cuts cache size and is now default in most decode-optimised LLMs.
MoE at serving time
Mixture of Experts routes each token to top-k expert FFNs. Parameter count soars while FLOPs per token stay closer to a dense model of expert-size. The tax is routing imbalance, expert parallelism networking, and tricky latency tails. Aux losses and capacity factors still matter. For EA capacity planning, model expert-parallel degree and all-to-all bandwidth, not just "70B-equivalent dense."
PEFT, LoRA, QLoRA, and multi-adapter serve
LoRA trains low-rank adapters BA on frozen W. QLoRA keeps the base in 4-bit (or similar) while training adapters in higher precision. Rank, alpha, target modules (q/v/o/ffn), and dropout are the knobs. Merge adapters for single-tenant serve; keep them hot-swappable for multi-tenant bank bots with per-desk adapters.
A closer look: LoRA is not magic data efficiency. Garbage instruction data still yields garbage. Prefer eval harnesses (held-out tasks, toxicity, PII) over loss curves alone.
Checkpointing, packing, accumulation, parallelism
Activation checkpointing (recompute) trades compute for memory. Sequence packing fills the batch with concatenated short sequences and careful masking. Gradient accumulation simulates larger batches when HBM is tight. Parallelism ladder: DDP → FSDP/ZeRO → tensor parallel → pipeline parallel → expert parallel. Match the ladder to model size and cluster topology; mis-ordered parallelism wastes NCCL time.
Stub syllabus checklist
Map every stub bullet to a shipped section: KV cache [ok] (authored); speculative + parallel decode [ok]; sparse/approximate + shared projections + FlashAttention [ok]; MoE [ok]; PEFT/LoRA [ok]; activation checkpointing, packing, accumulation, parallelism [ok]; pointer to Appendix B for quant [ok]. The stub's skip advice still stands for brand-new transformer readers: finish Ch15-16 first.
Bank / EA take
Credit and fraud LLM features usually need decode latency, adapter isolation, and audit of which adapter served. Prefer LoRA-per-product with a shared frozen base, KV-cache-aware autoscaling, and speculative decode only after acceptance rates clear a gate on production-like prompts. Quantisation (Appendix B) often beats exotic attention tricks for cost.
Oral
Explain prefill vs decode; why KV cache saves compute but grows memory; how speculative verify preserves distribution; what FlashAttention changes about HBM traffic; MQA/GQA cache math; MoE top-k routing failure modes; LoRA rank/alpha intuition; when to climb DDP→FSDP→TP/PP.
KV cache shapes and memory math
For a decoder layer, cached K and V per token are roughly `2 * n_layers * n_kv_heads * d_head * bytes_per_elem` (times batch). GQA/MQA shrink `n_kv_heads` relative to query heads and cut cache bytes almost linearly. FP16/BF16 caches are common; FP8/INT8 KV caches appear in 2026 serving stacks when quality gates pass. Prefix caching reuses KV for shared system prompts across requests: huge for bank assistants with long policy preambles.
Eviction policies matter when context exceeds GPU RAM: drop oldest turns, summarise, or move cold KV to CPU (slower). Measure time-to-first-token (TTFT) and time-per-output-token (TPOT) separately. TTFT is dominated by prefill; TPOT by decode bandwidth.
Parallel decode beyond speculation
Blockwise parallel decoding and look-ahead methods try to guess several future positions with masks; quality and speed vary. Medusa / multi-head drafts attach small heads to predict multiple future tokens from one hidden state. Treat these as siblings of speculative decoding: always verify with the target model if you need the target distribution. For non-autoregressive translation the book already covered parallel decode differently; do not confuse the two.
Attention implementation checklist
1. Use SDPA or vendor FlashAttention; assert the backend in logs once at startup. 2. Prefer GQA checkpoints for decode-heavy products. 3. Enable torch.compile carefully; validate numerics on a golden batch. 4. For very long context, test sliding-window / local+global hybrids before custom sparse kernels. 5. Profile with real sequence-length histograms from production, not synthetic T=4k only.
A closer look: papers quote best-case speedups on long sequences. Your mean T may be 256. Optimise for the percentile that pays the bill.
MoE training versus MoE serving
Training cares about load-balancing losses and capacity factors so experts learn. Serving cares about tail latency when many tokens route to the same expert. Expert parallelism shards experts across GPUs; the all-to-all can dominate if the network is weak. Start with a dense model unless you truly need trillion-parameter capacity. Distilled dense students often beat poorly tuned MoE on bank hardware.
PEFT catalogue (practical)
Beyond LoRA: (IA)³, prefix/prompt tuning, adapters in residual branches, and DoRA-class magnitude/direction splits. In Oct 2026, LoRA/QLoRA remain the default path in Hugging Face PEFT and most internal platforms. Store adapter weights in the model registry with base commit SHA, rank, targets, and eval report. Never overwrite a production adapter in place.
QLoRA double-quant and NF4 details belong next to Appendix B. End-to-end: quantised base + LoRA train + merge or dynamic load + optional speculative decode.
Systems tricks the stub listed
Activation checkpointing: wrap transformer blocks; expect ~30% slower train for large memory wins. Sequence packing: watch document boundaries and position ids; packing bugs look like mysterious quality drops. Gradient accumulation: scales loss correctly only if you average right; unit-test against a true large batch on a tiny model. Parallelism: debug with one GPU first, then DDP, then FSDP. Tensor parallel needs even head division. Pipeline parallel needs microbatches to hide bubbles.
Worked mental example
Cutting KV heads via GQA by 4× can cut decode bandwidth roughly 4× for the cache traffic. Adding a draft model with 70% acceptance on 4-token chunks can cut serial target steps sharply. Neither helps if your gateway serialises requests on a single worker with cold starts.
Connect to adjacent chapters
Ch15-16 built the architectures you are accelerating. Appendix B shrinks weights and activations. Ch11's training hygiene (AdamW, grad clip, compile) still applies when you fine-tune with LoRA. Ch10's device and DataLoader lessons still apply to packing.
Closing
Speed work is measurement work. Separate prefill from decode, shrink KV, fuse attention, speculate only with verify, route MoE only with networking budget, adapt with LoRA under eval gates, and climb the parallelism ladder in order.
18. Autoencoders, GANs, and Diffusion Models
Diffusion sketch.
GAN pair.
AE loop.
Autoencoders, variational variants, GANs, and diffusion models for generative vision and representation learning.
Ch18 panel 1
Study loop.
This stretch centers on GAN. Source cue: Autoencoders, GANs, and Diffusion Models Autoencoders are artificial neural networks capable learning dense representations the input data, called.
Retype any code from memory and diff.
This stretch centers on GAN, diffusion. Source cue: You can judge for yourself visiting https://thispersondoesnotexist.com, website that shows faces generated GAN architecture called StyleGAN. GANs have.
Retype any code from memory and diff.
This stretch centers on diffusion. Source cue: learn copy their inputs their outputs. This may sound like trivial task, but you will see, constraining the.
Retype any code from memory and diff.
Ch18 panel 2
This stretch centers on diffusion, GAN. Source cue: money, while the discriminator like the police investigator trying tell real money from fake. Diffusion models diffusion model.
Retype any code from memory and diff.
This stretch centers on diffusion. Source cue: Let start with autoencoders Efficient Data Representations Which the following number sequences you find the easiest memorize 40,.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: The fact that hard memorize long sequences what makes useful recognize patterns, and hopefully this clarifies why constraining.
Retype any code from memory and diff.
Ch18 panel 3
Ops contrast.
This stretch centers on Ch18 ideas. Source cue: latent representation, and then capable reconstructing something that hopefully looks very close the inputs. autoencoder always composed two.
Retype any code from memory and diff.
This stretch centers on PCA, MSE, Sequential. Source cue: chess memory experiment left and simple autoencoder right Because the internal representation has lower dimensionality than the input.
Retype any code from memory and diff.
This stretch centers on PCA, Sequential, MSE, DataLoader, TensorDataset. Source cue: find the plane that lies closest the training data, just like PCA would. import torch import torch.nn torch.manual.
Retype any code from memory and diff.
Ch18 panel 4
Practice hub.
This stretch centers on DataLoader, TensorDataset, MeanSquaredError, PCA. Source cue: from torch.utils.data import DataLoader, TensorDataset train ... generate dataset, like Chapter train set TensorDataset train, train the inputs.
Retype any code from memory and diff.
This stretch centers on PCA. Approximate PCA performed undercomplete linear.
Retype any code from memory and diff.
This stretch centers on ReLU, PyTorch, Sequential. Source cue: typically symmetrical with regard the central hidden layer the coding layer put simply, looks like sandwich. For example,.
Retype any code from memory and diff.
Ch18 panel 5
Fit discipline.
This stretch centers on Sequential, ReLU, TorchVision, MSE. Source cue: Let through this code: Just like earlier, split the autoencoder model into two submodels: the encoder and the.
Retype any code from memory and diff.
This stretch centers on TorchVision, MSE. Source cue: split into train data, valid data, and test data just like did Chapter then train the autoencoder exactly.
Retype any code from memory and diff.
This stretch centers on MSE. Source cue: plt.subplot len images len images idx plot image pred idx valid torch.stack for valid data plot reconstructions stacked.
Retype any code from memory and diff.
Ch18 panel 6
Error parts.
This stretch centers on MSE. Source cue: the output will very high. use the model for anomaly detection, you simply need define threshold, then any.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: MNIST. First use the encoder from our stacked autoencoder reduce the dimensionality down 32, then use Scikit-Learn implementation.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: tackling complex supervised task but you not have lot labeled training data, one solution find neural network that.
Retype any code from memory and diff.
Ch18 panel 7
Train vs serve.
This stretch centers on Ch18 ideas. Source cue: Figure 18-7. Unsupervised pretraining using autoencoders Note Having plenty unlabeled data and little labeled data common. Building large.
Retype any code from memory and diff.
This stretch centers on TiedAutoencoder. Source cue: layers the weights the encoder layers. This halves the number weights the model, speeding training and limiting the.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: self, F.relu F.linear self.enc2.weight.t self.dec1 bias F.sigmoid F.linear self.enc1.weight.t self.dec2 bias return Z.view -1, 28, unflatten def forward.
Retype any code from memory and diff.
Ch18 panel 8
Model promotion.
This stretch centers on Ch18 ideas. Source cue: autoencoder, then the decoder layers reverse order This gives the final stacked autoencoder. could easily train more autoencoders.
Retype any code from memory and diff.
This stretch centers on CNN, Sequential, ReLU. Source cue: have seen far will not work well unless the images are very small you saw Chapter 12, convolutional.
Retype any code from memory and diff.
This stretch centers on ReLU, Sequential. Source cue: size padding same nn.ReLU nn.MaxPool2d kernel size output: nn.Conv2d 64, 32, kernel size padding same nn.ReLU nn.AdaptiveAvgPool2d nn.Flatten.
Retype any code from memory and diff.
Ch18 panel 9
Study loop.
This stretch centers on LeCun, Dropout. Source cue: the coding layer just large the inputs, even larger, resulting overcomplete autoencoder. the following sections look few more.
Retype any code from memory and diff.
This stretch centers on Dropout, Sequential, ReLU, BERT. Source cue: autoencoder with additional Dropout layer applied the encoder inputs recall that the Dropout layer only active during training.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: data visualization unsupervised pretraining, like the other autoencoders discussed far, but they can also used quite simply and.
Retype any code from memory and diff.
Ch18 panel 10
This stretch centers on Ch18 ideas. Source cue: coding layer activations. This means adding the norm the codings i.e., the sum their absolute values the loss,.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: 0.3, but the target sparsity 0.1, then this neuron must penalized activate less. One approach could simply adding.
Retype any code from memory and diff.
This stretch centers on PyTorch, SparseAutoencoder, Sequential, ReLU. Source cue: Equation 18-2. Equation 18-2. divergence between the target sparsity and the actual sparsity log log 1-p 1-q implement.
Retype any code from memory and diff.
Ch18 panel 11
Ops contrast.
This stretch centers on MSE, SparseAutoencoder. Source cue: after calling the model. Next, can define the loss function: def mse plus sparsity loss pred, target, target.
Retype any code from memory and diff.
This stretch centers on SparseAutoencoder. Source cue: the model and train using the same train function earlier, from Chapter torch.manual seed sparse SparseAutoencoder .to device.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: even after training opposed denoising autoencoders, which use randomness only during training Most importantly, they are generative autoencoders,.
Retype any code from memory and diff.
Ch18 panel 12
Practice hub.
This stretch centers on Ch18 ideas. Source cue: The final output resembles the training instance. you can see Figure 18-12, although the inputs may have very.
Retype any code from memory and diff.
This stretch centers on MSE. Source cue: the deterministic and stochastic parts the process, allowing the gradients flow back into the encoder through and The.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: latent loss can computed using Equation 18-3 for the full mathematical details, check out the original paper variational.
Retype any code from memory and diff.
Ch18 panel 13
Fit discipline.
This stretch centers on VAE, Sequential, ReLU. Source cue: Fashion MNIST, using the architecture shown Figure 18-12, except using the tweak: VAEOutput namedtuple VAEOutput output codings mean.
VAE path.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: and codings logvar The encoder and decoder architectures strongly resemble the previous autoencoders, but notice that the encoder.
Retype any code from memory and diff.
This stretch centers on VAE, MSE. Source cue: This the reparameterization trick discussed earlier. The decode method simply calls the decoder model produce the reconstructions. The.
Retype any code from memory and diff.
Ch18 panel 14
Error parts.
This stretch centers on VAE. Source cue: over all instances the batch and all 784 pixels: this why divide the latent loss 784 ensure that.
Retype any code from memory and diff.
This stretch centers on VAE. Source cue: using convolutional layers Variational autoencoders make possible perform semantic interpolation: instead interpolating between two images the pixel level,.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: which integer between and where the number possible latent codes. The length the coding vector often denoted For.
Retype any code from memory and diff.
Ch18 panel 15
Train vs serve.
This stretch centers on DeepMind, Sequential. Source cue: but cannot reuse the same for regular VAEs: need one designed for categorical distributions. The most popular one.
Retype any code from memory and diff.
This stretch centers on DeepMind, Sequential, VAE, MSE. Source cue: other from Google, Cambridge University, and Stanford University. Let implement dVAE for Fashion MNIST: DiscreteVAEOutput namedtuple DiscreteVAEOutput output.
Retype any code from memory and diff.
This stretch centers on VAE, MSE. Source cue: training, then gradually reduce during training, down small value such 0.1. The loss function also similar the regular.
Retype any code from memory and diff.
Ch18 panel 16
Model promotion.
This stretch centers on VAE, DeepMind. Source cue: epoch epochs Once the model trained, you can generate new images sampling random codings from uniform distribution, one-hot.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: this operation: this why this trick called the straight-through estimator STE approximation that assumes that the gradients around.
Retype any code from memory and diff.
This stretch centers on Softmax. Source cue: generating sequence codes using the transformer, then passing this sequence the dVAE decoder. This two-stage approach also makes.
Retype any code from memory and diff.
Ch18 panel 17
This stretch centers on Softmax, VAE. Source cue: logits the case Gumbel-Softmax dVAEs After mapping these embeddings the indices the nearest embeddings the codebook after categorical.
Retype any code from memory and diff.
This stretch centers on GAN. Source cue: you manage get them work, they produce pretty amazing images. Generative Adversarial Networks Generative adversarial networks were proposed.
Retype any code from memory and diff.
This stretch centers on GAN. Source cue: you will soon see. Discriminator Takes either fake image from the generator real image from the training set.
Retype any code from memory and diff.
Ch18 panel 18
This stretch centers on GAN. Source cue: loss. Backpropagation only optimizes the weights the discriminator during this phase. Second phase: train the generator first.
Retype any code from memory and diff.
This stretch centers on GAN, Sequential, ReLU. Source cue: First, need build the generator and the discriminator. The generator similar autoencoder decoder takes coding vector input and.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: zeros torch.zeros batch size, device device fake loss criterion pred fake, zeros discriminator loss real loss fake loss.
Retype any code from memory and diff.
Ch18 panel 19
This stretch centers on GAN. Source cue: want the discriminator predict that these images are fake. The total discriminator loss the real loss plus the.
Retype any code from memory and diff.
This stretch centers on GAN. Source cue: grad generated images generator codings you display the generated images see Figure 18-16 you will see that the.
Retype any code from memory and diff.
This stretch centers on GAN. Source cue: road: driver would better off being the only one switch sides. course, there second possible Nash equilibrium: when.
Retype any code from memory and diff.
Ch18 panel 20
This stretch centers on GAN. Source cue: you perfect generator. Unfortunately, not that simple: nothing guarantees that the equilibrium will ever reached. The biggest difficulty.
Retype any code from memory and diff.
This stretch centers on GAN. Mode collapse while training GAN top row versus successful training without mode.
Retype any code from memory and diff.
This stretch centers on diffusion. Source cue: this buffer rather than just fake images produced the current generator This reduces the chances that the discriminator.
Retype any code from memory and diff.
Ch18 panel 21
This stretch centers on diffusion. Source cue: more diverse images, GANs were quickly sidelined. let now turn our attention diffusion models. Diffusion Models The ideas.
Retype any code from memory and diff.
This stretch centers on diffusion. Source cue: months later, 2021 paper OpenAI researchers Alex Nichol and Prafulla Dhariwal analyzed the DDPM architecture and proposed several.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: noted the original DDPM paper, the authors used 1,000, and they scheduled the variance such way that the.
Retype any code from memory and diff.
Ch18 panel 22
This stretch centers on diffusion. Source cue: the variance will gradually converge This because the standard deviation the pixel values also gets scaled the variance.
Retype any code from memory and diff.
This stretch centers on diffusion. Source cue: 18-5. Probability distribution the forward diffusion process t-1 t-1 Interestingly, there shortcut for the forward process: possible sample.
Retype any code from memory and diff.
This stretch centers on Ch18 ideas. Source cue: contrary, want create many new cats can training model that can perform the reverse process: going from can.
Retype any code from memory and diff.
Ch18 panel 23
This stretch centers on diffusion. Source cue: time step the pixel values get multiplied and noise with mean and variance gets added explained earlier So,.
Retype any code from memory and diff.
This stretch centers on diffusion.
Retype any code from memory and diff.
Closing
Generative stacks moved from GANs toward diffusion. Own AE/VAE maths; treat GAN mode collapse as history that still teaches.
19. Reinforcement Learning
Rewards, MDPs, value methods, policy gradients, and deep RL - with library notes toward Gymnasium.
MDP loop.
Ch19 panel 1
Study loop.
This stretch centers on DeepMind, AlphaGo, AlphaFold, AlphaCode. Source cue: learning one the most exciting fields machine learning today, and also one the oldest. has been around since.
Retype any code from memory and diff.
This stretch centers on DeepMind, AlphaFold, AlphaCode, PPO. Source cue: proteins with unprecedented accuracy. This game changer biology, chemistry, and medicine. fact, Demis Hassabis founder and CEO and.
Retype any code from memory and diff.
This stretch centers on PPO. Source cue: decision processes and lastly, actor-critic methods, including the popular PPO, which will use beat Atari game. let get.
Retype any code from memory and diff.
Ch19 panel 2
This stretch centers on Ch19 ideas. Source cue: set sensors, such cameras and touch sensors, and its actions consist sending signals activate motors. may programmed get.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: agent can observe stock market prices and decide how much buy sell every second. Rewards are obviously the.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Reinforcement learning using neural network policy The.
Retype any code from memory and diff.
Ch19 panel 3
Ops contrast.
This stretch centers on Ch19 ideas. Source cue: you can tweak: the probability and the angle range One possible learning algorithm could try out many different.
Retype any code from memory and diff.
This stretch centers on Gymnasium. Source cue: the policy space left and the agent corresponding behavior right Yet another approach use optimization techniques evaluating the.
Retype any code from memory and diff.
This stretch centers on PyBullet, MuJoCo, Gymnasium, CartPole. Source cue: You can speed time either adding more computing power won make the robot move any faster and generally.
Retype any code from memory and diff.
Ch19 panel 4
Practice hub.
This stretch centers on Gymnasium, CartPole, NumPy. Source cue: own machine instead Colab, and you followed the installation instructions https://homl.info/install-p, then you should good go. Let start.
Retype any code from memory and diff.
This stretch centers on NumPy, CartPole. Source cue: representing the cart horizontal position 0.0 center its velocity positive means right the angle the pole 0.0 vertical.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: integers and which represent accelerating left right. Other environments may have additional discrete actions, other kinds actions e.g.,.
Retype any code from memory and diff.
Ch19 panel 5
Fit discipline.
This stretch centers on Gymnasium, CartPole. This will happen.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: method free resources. Let hardcode simple policy that accelerates left when the pole leaning toward the left and.
Retype any code from memory and diff.
This stretch centers on CartPole. Source cue: that the cart oscillates left and right more and more strongly until the pole tilts too much. neural.
Retype any code from memory and diff.
Ch19 panel 6
Error parts.
This stretch centers on Ch19 ideas. Source cue: why are picking random action based the probabilities given the neural network, rather than just picking the action.
Retype any code from memory and diff.
This stretch centers on CartPole, PyTorch, PolicyNetwork, Sequential, ReLU. Source cue: example, the environment only revealed the position the cart but not its velocity, you would have consider not.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: units need for more this case Finally, want output single probability, have single output neuron. there were more.
Retype any code from memory and diff.
Ch19 panel 7
Train vs serve.
This stretch centers on Ch19 ideas. Source cue: actions, you would use Categorical distribution instead. Lastly, compute the log probability the sampled action i.e., either log.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: distribution. would just regular supervised learning. However, reinforcement learning the only guidance the agent gets through rewards, and.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: rewards called the action return. Consider the example Figure 19-6. agent decides right three times row and gets.
Retype any code from memory and diff.
Ch19 panel 8
Model promotion.
This stretch centers on CartPole. Source cue: rewards far into the future will count almost much immediate rewards. Typical discount factors vary from 0.9 0.99..
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: higher rewards. One popular algorithm, called REINFORCE Monte Carlo was introduced back 1992 Ronald Williams. has many variants,.
Retype any code from memory and diff.
This stretch centers on PyTorch, NumPy, CartPole. Source cue: given state where the time step and the observed return this action; represents the model parameters. Let use.
Retype any code from memory and diff.
Ch19 panel 9
Study loop.
This stretch centers on NumPy, CartPole. Source cue: array reward, two booleans indicating whether the game over truncated, and info dict which can safely ignore the.
Retype any code from memory and diff.
This stretch centers on PolicyNetwork, CartPole. Source cue: optional but common and recommended tweak the REINFORCE algorithm, stabilizes training. Next, the function computes the REINFORCE loss.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: discuss the end this chapter Tip Researchers try find algorithms that work well even when the agent initially.
Retype any code from memory and diff.
Ch19 panel 10
This stretch centers on Ch19 ideas. Source cue: the agent capable reaching right now. the agent progresses, explores different parts the environment, and can forget everything.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: learns estimate the value each state i.e., the expected return the value each action given state, then uses.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: bound leave that state and never come back, because other state points back s0. goes state s1, will.
Retype any code from memory and diff.
Ch19 panel 11
Ops contrast.
This stretch centers on Ch19 ideas. Source cue: example, the MDP represented Figure 19-8 has three states represented circles and three possible discrete actions each step.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: choice but take action a1, which will most likely lead back state s0, gaining reward the way. You.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: the current state equal the reward will get average after taking one optimal action, plus the expected optimal.
Retype any code from memory and diff.
Ch19 panel 12
Practice hub.
This stretch centers on Ch19 ideas. Source cue: Note the Bellman equation and the rest this chapter, optimal policy one that maximizes the expected sum discounted.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: state the kth iteration the algorithm. Note This algorithm example dynamic programming, which breaks down complex problem into.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: how works. Once again, you start initializing all the Q-value estimates zero, then you update them using the.
Retype any code from memory and diff.
Ch19 panel 13
Fit discipline.
This stretch centers on Ch19 ideas. Source cue: 0.2, 0.0 0.0, 1.0, 0.0 None, 0.0, 0.0, 1.0 None, 0.8, 0.1, 0.1 None rewards shape 10, -50.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: Now let run the Q-value iteration algorithm. applies Equation 19-4 repeatedly, all Q-values, for every state and every.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: increase the discount factor 0.95, the optimal policy changes: state the best action becomes through the fire This.
Retype any code from memory and diff.
Ch19 panel 14
Error parts.
This stretch centers on Ch19 ideas. Source cue: has only partial knowledge the MDP. general assume that the agent initially knows only the possible states and.
Retype any code from memory and diff.
This stretch centers on SGD. Source cue: error. more concise way writing the first form this equation use the notation which means bk. the first.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: the transition probabilities and the rewards are initially unknown see Equation 19-6 Q-learning works watching agent play e.g.,.
Retype any code from memory and diff.
Ch19 panel 15
Train vs serve.
This stretch centers on Ch19 ideas. Source cue: this, need step function that the agent can execute one action and get the resulting state and reward:.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: decay values state, action alpha values state, action alpha reward gamma next value state next state This algorithm.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: highest Q-value. Conversely, the REINFORCE algorithm on-policy: explores the world using the policy being trained. somewhat surprising that.
Retype any code from memory and diff.
Ch19 panel 16
Model promotion.
This stretch centers on Ch19 ideas. Source cue: still spending some time visiting unknown regions the MDP. quite common start with high value for e.g., 1.0.
Retype any code from memory and diff.
This stretch centers on DeepMind. Source cue: Deep Q-Learning The main problem with Q-learning that does not scale well large even medium MDPs with many.
Retype any code from memory and diff.
This stretch centers on DeepMind, DQN. Source cue: the state e.g., the distances the closest ghosts, their directions, and estimate Q-values, but 2013, DeepMind showed that.
Retype any code from memory and diff.
Ch19 panel 17
Study loop.
This stretch centers on CartPole. Source cue: the highest since assume will playing optimally and discount it, and this gives estimate the sum future discounted.
Retype any code from memory and diff.
This stretch centers on CartPole, DQN, Sequential, ReLU. Source cue: and outputs approximate Q-value. However, practice much more efficient use neural net that takes only state input, and.
Retype any code from memory and diff.
This stretch centers on DQN. Source cue: predicted Q-value argmax ensure that the agent explores the environment, use -greedy policy, meaning choose random action with.
Retype any code from memory and diff.
Ch19 panel 18
This stretch centers on NumPy.
Retype any code from memory and diff.
This stretch centers on NumPy, PyTorch. Source cue: all have shape batch size except for the observation tensors, which have shape batch size, The tensor function.
Retype any code from memory and diff.
This stretch centers on DeepMind, DQN. Source cue: and popping items the ends deque very fast, but random access can slow when the queue gets very.
Retype any code from memory and diff.
Ch19 panel 19
Ops contrast.
This stretch centers on DQN. Source cue: function that will sample batch experiences from the replay buffer and train the DQN performing single gradient descent.
Retype any code from memory and diff.
This stretch centers on MSE. Source cue: next state then keep only the max Q-value since assume that the agent will play optimally from now.
Retype any code from memory and diff.
This stretch centers on DQN. Source cue: can write the main training function and run it: from collections import deque def train dqn model, env,.
Retype any code from memory and diff.
Ch19 panel 20
Practice hub.
This stretch centers on DQN. Source cue: the agent behavior will gradually become less random, focusing more exploitation and less exploration. The function also records.
Retype any code from memory and diff.
This stretch centers on DQN, DeepMind. Source cue: than REINFORCE. had tweak the hyperparameters quite bit before stumbling upon this successful training run. you can see,.
Retype any code from memory and diff.
This stretch centers on DeepMind, DQN, CNN. Source cue: training using two DQNs instead one: the first the online model, which learns each step and used move.
Retype any code from memory and diff.
Ch19 panel 21
Fit discipline.
This stretch centers on DeepMind. Source cue: network prone overestimating Q-values. Indeed, suppose all actions are equally good: the Q-values estimated the target model should.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: progress. But how can estimate this One reasonable approach measure the magnitude the error large error indicates that.
Retype any code from memory and diff.
This stretch centers on DQN. Source cue: used 0.6, but the optimal value will depend the task. There one catch though: since the samples will.
Retype any code from memory and diff.
Ch19 panel 22
Error parts.
This stretch centers on DQN, DeepMind. Source cue: you will usually want increase the other well. One last noteworthy DQN variant the dueling DQN algorithm DDQN,.
Retype any code from memory and diff.
This stretch centers on DeepMind. Source cue: the best action should have advantage the model subtracts the maximum predicted advantage from all predicted advantages. The.
Retype any code from memory and diff.
This stretch centers on ActorCritic, Sequential, ReLU. Source cue: like athlete the actor learning with the help coach the critic Moreover, actor-critic methods support stochastic policies and.
Retype any code from memory and diff.
Ch19 panel 23
Train vs serve.
This stretch centers on Ch19 ideas. Source cue: for REINFORCE The critic network estimates the value each given state. The forward method takes batch states and.
Retype any code from memory and diff.
This stretch centers on MSE. Source cue: the state value The actor loss the same REINFORCE, except that multiply the log probability the error instead.
Retype any code from memory and diff.
This stretch centers on Ch19 ideas. Source cue: running discount factor next state value return target value This code first evaluates using the choose action and.
Retype any code from memory and diff.
Ch19 panel 24
This stretch centers on ActorCritic, CartPole, DeepMind. Source cue: which just calls the run epi sode and train function many times and returns the total rewards for.
Retype any code from memory and diff.
This stretch centers on DQN, A2C, GPU. Source cue: agent pushes some weight updates master network, then pulls the latest weights from that network. Each agent thus.
Retype any code from memory and diff.
This stretch centers on PPO, A2C, DQN. Source cue: speeds training and makes less likely repeatedly execute the same action when the critic produces imperfect estimates. This.
Retype any code from memory and diff.
Ch19 panel 25
This stretch centers on PPO, DQN, Gymnasium. Source cue: PPO great general-purpose algorithm good bet you not sure. SAC the most sample efficient for continuous action tasks,.
Retype any code from memory and diff.
This stretch centers on Gymnasium, BreakoutNoFrameskip, NumPy. Source cue: locally, then they are also preinstalled. Now that have SB3, the ALE interface, and the ROMs, are ready.
Retype any code from memory and diff.
This stretch centers on BreakoutNoFrameskip, VecFrameStack, NumPy, PPO, CnnPolicy. Breakout frame before left and after right preprocessing The ALE interface runs frames.
Retype any code from memory and diff.
Ch19 panel 26
This stretch centers on PPO, CnnPolicy, CNN, ActorCriticPolicy. Source cue: PPO ppo model PPO CnnPolicy envs stacked, device device, learning rate 2.5e-4, batch size 256, steps 256, epochs.
Retype any code from memory and diff.
This stretch centers on PPO. Source cue: limits the magnitude the policy updates avoid large changes that might cause catastrophic forgetting. coef the weight the.
Retype any code from memory and diff.
Closing
RL optimises behaviour under delayed reward. Start simple; simulate before you touch production controls.
Appendix A. Autodiff
AD modes.
Forward and reverse mode automatic differentiation pictured and tied to PyTorch autograd.
AppA panel 1
Study loop.
This stretch centers on PyTorch. Source cue: automatic differentiation autodiff feature works, and how compares other solutions. Suppose you define function x2y and you need.
Retype any code from memory and diff.
This stretch centers on AppA ideas. Source cue: sum these functions derivatives. The derivative times function times its derivative. From these rules, you can derive Equation.
Retype any code from memory and diff.
This stretch centers on AppA ideas. Source cue: point the function, gets infinitely close see Equation A-2 Equation A-2. Definition the derivative function point lim x-x.
Retype any code from memory and diff.
AppA panel 2
This stretch centers on AppA ideas. Source cue: and gets worse for more complicated functions The correct results are respectively and 10, but instead get:.
Retype any code from memory and diff.
This stretch centers on AppA ideas. Source cue: forward mode. Forward-Mode Autodiff Figure A-1 shows how forward-mode autodiff works even simpler function, xy. The graph for.
Retype any code from memory and diff.
This stretch centers on AppA ideas. Source cue: node function Calculus tells that the derivative the product two functions and can therefore construct large part the.
Retype any code from memory and diff.
AppA panel 3
Ops contrast.
This stretch centers on AppA ideas. Source cue: but for more complex function, forward-mode autodiff can produce huge graph that may tough simplify and lead suboptimal.
Retype any code from memory and diff.
This stretch centers on AppA ideas. Source cue: real numbers, and infinitesimal number such that but You can think the dual number something akin 42.0000 000024.
Retype any code from memory and diff.
This stretch centers on AppA ideas. Source cue: both and the derivative just one shot. Figure A-2 shows that the partial derivative with regard and which.
Retype any code from memory and diff.
AppA panel 4
Practice hub.
This stretch centers on PyTorch. Source cue: approximation, but suffers from the same major flaw, least when there are many inputs and few outputs the.
Retype any code from memory and diff.
This stretch centers on AppA ideas. Source cue: those values the bottom right each node e.g., The nodes are labeled for clarity. The output node n7:.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: that Now can proceed node n4: how much does vary when varies The answer n4. Since n2, find.
Retype any code from memory and diff.
Closing
Autodiff is why deep learning scales. Reverse mode fits scalar losses with many parameters.
Appendix B. Mixed Precision and Quantization
Quant path.
AMP, GradScaler, and quantization paths that cut memory and boost throughput without rewriting models.
AMP flow.
AppB panel 1
Study loop.
This stretch centers on PyTorch, Adam, GPU. Source cue: Quantization default, PyTorch uses 32-bit floats represent model parameters: that bytes per parameter. your model has billion parameters,.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: Great However, you try train the model using 16-bit floats, you may run into convergence issues, will see..
Retype any code from memory and diff.
This stretch centers on AppB ideas. Source cue: specifies how floating-point numbers are represented memory. flexible and efficient format which can represent tiny values and huge.
Retype any code from memory and diff.
AppB panel 2
This stretch centers on AppB ideas. Source cue: subnormal: useful represent the tiniest values. this case, 127 149. and then 255 and then NaN. 255 and.
Retype any code from memory and diff.
This stretch centers on AppB ideas. Source cue: bfloat8 formats were proposed Google Brain hence the and they offer wider range for the values, the cost.
Retype any code from memory and diff.
This stretch centers on PyTorch, Sequential, ReLU. Source cue: can written 20121 base since 178 and subtract from each digit, get these are the ternary weights stored.
Retype any code from memory and diff.
AppB panel 3
Ops contrast.
This stretch centers on PyTorch, Sequential, ReLU, GPU. Source cue: Precision Models you have 32-bit PyTorch model, you can convert all its parameters 16-bit floats which called half-precision.
Retype any code from memory and diff.
This stretch centers on Sequential, ReLU. Source cue: 16-bit input pred model 16-bit output But what you want build and train 16-bit model right from the.
Retype any code from memory and diff.
This stretch centers on AppB ideas. Source cue: infinity, causing training fail once some weights are infinite, the loss will infinite NaN avoid underflows, one solution.
Retype any code from memory and diff.
AppB panel 4
Practice hub.
This stretch centers on AppB ideas. Source cue: support although this improving and offers fewer bits for the fraction, which can cause some gradient updates ignored.
Retype any code from memory and diff.
This stretch centers on AppB ideas. Source cue: mixed-precision training instead. Mixed-Precision Training Mixed-precision training MPT was proposed Baidu and Nvidia researchers 2017, address the issues.
Retype any code from memory and diff.
This stretch centers on AppB ideas. Source cue: the benefits 16-bit training, without the instabilities. However, the model parameters take more space than 32-bit training because.
Retype any code from memory and diff.
AppB panel 5
Fit discipline.
This stretch centers on PyTorch, GradScaler. Source cue: should large enough avoid underflows, and small enough avoid overflows. Alternatively, your training script can adapt the factor.
Retype any code from memory and diff.
This stretch centers on GradScaler, TrainingArguments. Source cue: the loss: from torch.amp import GradScaler def train mpt model, optimizer, criterion, train loader, epochs, dtype torch.float16, init.
Retype any code from memory and diff.
This stretch centers on AppB ideas. Source cue: and energy consumption models have quantized entirely both weights and activations before you can use them the device..
Retype any code from memory and diff.
AppB panel 6
Error parts.
This stretch centers on AppB ideas. Source cue: 0.1 and 0.6, then the float 0.1 will mapped the byte the float 0.0 integer 36, 0.1 72,.
Retype any code from memory and diff.
This stretch centers on AppB ideas. Source cue: value The range known for weights, since their values not change after training. However, the range activation values.
Retype any code from memory and diff.
This stretch centers on ReLU, PyTorch. Source cue: 0.0988 and the original value 0.1 called the quantization noise: with 8-bit quantization, the quantization noise usually leads.
Retype any code from memory and diff.
AppB panel 7
Train vs serve.
This stretch centers on PyTorch. Source cue: will use the data type torch.quint8, which uses 8-bit unsigned integers:.
Retype any code from memory and diff.
This stretch centers on PyTorch. Source cue: 32-bit floats, bytes each Tip PyTorch also has torch.quantize per channel function which quantizes each channel separately: this.
Retype any code from memory and diff.
This stretch centers on ReLU, PyTorch. Source cue: scale 0.007401574868708849, zero point Figure B-3 shows some floats ranging between 0.94 and 0.93, quantized signed bytes i.e.,.
Retype any code from memory and diff.
AppB panel 8
Model promotion.
This stretch centers on PyTorch, CUDA. Source cue: fine-tune your model with some fake quantization get used the noise: this called quantization-aware training QAT Let start.
Retype any code from memory and diff.
This stretch centers on PyTorch, CUDA, RNN, Sequential, ReLU. Source cue: hardware accelerators, but other libraries do, such the bitsandbytes library will see shortly Once you have selected engine,.
Retype any code from memory and diff.
This stretch centers on QuantStub, DeQuantStub, Sequential, ReLU. Source cue: the next layer gets float inputs. Now let look static quantization. Static quantization This option best for CNNs,.
Retype any code from memory and diff.
AppB panel 9
Study loop.
This stretch centers on DeQuantStub, QuantStub. Source cue: are just passthrough for now. Next, the model can trained normally another option would take pretrained model and.
Retype any code from memory and diff.
This stretch centers on QuantStub, QuantizedLinear, DeQuantStub, DeQuantize, MinMaxObserver. Source cue: the outputs the QuantStub and Linear layers. The observer appended the QuantStub layer responsible for tracking the input.
Retype any code from memory and diff.
This stretch centers on HistogramObserver, ExecuTorch, PyTorch, ONNX. Source cue: outliers. Finally, they can even record histogram the observed values HistogramObserver making possible find optimal quantization range that.
Retype any code from memory and diff.
AppB panel 10
This stretch centers on AppB ideas. Source cue: such 4-bit weight support and per-block quantization, which each tensor split into small blocks and each block quantized.
Retype any code from memory and diff.
This stretch centers on PyTorch.
Retype any code from memory and diff.
This stretch centers on PyTorch, Sequential, ReLU, GPU. Source cue: the loss surface generally fairly smooth locally, gradients are likely similar within small region around the quantized value,.
Retype any code from memory and diff.
AppB panel 11
Ops contrast.
This stretch centers on PyTorch, GPU, Adam, AdamW, CUDA. Source cue: the pretrained weights. seen how implement PTQ and QAT using PyTorch torch.ao package. However, primarily designed for CPUs..
Retype any code from memory and diff.
This stretch centers on BitsAndBytesConfig, TinyLlama, GPU. Source cue: using Colab, you must first install the bitsandbytes library using pip install bitsandbytes, then run this code: from.
Retype any code from memory and diff.
This stretch centers on BitsAndBytesConfig, QLoRA, LoRA, GPU. Source cue: soon the dequantized weights are longer needed, they are dropped, memory usage remains low. this example, the BitsAndBytesConfig.
Retype any code from memory and diff.
AppB panel 12
Practice hub.
This stretch centers on GPU, QLoRA, CUDA, TheBloke, MSE. Source cue: accuracy drop. Although activation checkpointing reduces VRAM usage overall, can lead memory spikes when processing batches with long.
Retype any code from memory and diff.
Closing
Precision is a systems knob. Measure accuracy and latency together before you celebrate FLOPs.
Main Points
1. Split and metrics first. End-to-end discipline beats a clever model on a leaky split. 2. Thresholds are policy. Accuracy is an interview trap under imbalance. 3. Linear models remain shippable. Scorecards and calibrated logistics still run banks. 4. Trees win tabular. HGB/forests before deep nets on spreadsheets. 5. GD needs scale. Unscaled features make ravines. 6. Regularisation is a dial. Ridge, lasso, early stop, dropout - different inductive biases. 7. PyTorch loop is stable. to → optim → zero → forward → loss → backward → step → eval/no_grad. 8. Init and norm keep signals alive. He/Glorot, BN/LN, residual paths. 9. CNNs exploit locality. Know conv arithmetic before abandoning it. 10. Do not shuffle time. Sequence leakage is silent fraud. 11. Attention routes information. Transformers dominate NLP; own the block. 12. KV cache and GQA make decode feasible. Measure TTFT and p95. 13. LoRA/QLoRA are the fine-tune default. Full FT is the exception. 14. FlashAttention is IO. It does not fix bad data. 15. MoE is organisationally expensive. Earn it with eval and cost curves. 16. Diffusion leads many generative image paths. GANs taught the pain. 17. RL belongs in simulators first. Hard constraints before real capital. 18. AMP and quant are systems knobs. Track quality floors with latency. 19. Baselines forever. Seasonal naive, logistic, HGB - keep them in the registry. 20. Ch17 was a stub. Insight authored the speed stack; reconcile if homl.info ships later.
Frameworks and mental models
| Name | Says | Use when | |
|---|---|---|---|
| Bias-variance | Error decomposes | Learning curves hurt | PR vs ROC |
| Imbalance truth | Fraud/rare events | Train loop | Zero-forward-backward-step |
| Any PyTorch work | Prefill/decode | Different bottlenecks | LLM serve |
| PEFT first | Adapters before full FT | Domain tune LLMs | Tabular default |
A closer look (whole book)
The PyTorch edition is excellent pedagogy; production NLP/vision will still drag you into Hugging Face, vLLM, and vendor engines not fully spelled out in every chapter. Do not skip sklearn chapters because transformers are fashionable - bank tabular work still lives there.
Who's who (tools)
| Tool | Role | |
|---|---|---|
| scikit-learn | Tabular baselines, pipelines | PyTorch |
| Deep learning core | TorchVision | Images |
| Optuna | Hparam search | PEFT/LoRA |
| Efficient FT | Gymnasium | RL envs |
About this book
Hands-On Machine Learning with Scikit-Learn and PyTorch: Concepts, Tools, and Techniques to Build Intelligent Systems by Aurélien Géron, 2025. This is a chapter-by-chapter study of the book, with summaries, maps, tables and diagrams; short quotations keep their page references. It is shared free for personal study, is not for sale, and is not endorsed by the author or publisher. Please read and buy the original.
Insight Editions · rajeshrmahapatra.com/insight-editions