TLDR

• A score is not a decision. A model estimates a quantity under a dataset, objective and sampling scheme. Policy, evidence, human authority and downstream consequences determine the action.
• Start with the learning contract: target, observation unit, prediction time, eligible evidence, error costs and abstention route. Algorithm choice comes later.
• Validation must resemble use. Random splits can leak time, entities and future information into the past. Report uncertainty, subgroup behaviour, calibration and decision impact, not one headline metric.
• Complexity must earn its place. Linear models, trees and nearest neighbours remain valuable because they expose baselines, failure modes and operating trade-offs.
• The operating model includes monitoring and recovery. Data drift, score drift, policy changes and delayed outcomes need separate owners, thresholds and routes.

Reader and route

This edition is for engineers, data scientists, architects, product leaders and control practitioners who need to connect machine-learning mechanics to accountable decisions. Parts I and II establish learning, mathematics, model families and optimisation. Part III covers data preparation and evaluation. Part IV treats deep, specialised, unsupervised and ranking systems. Part V turns the material into design reviews and a governed lending-memo lab.

Evidence boundary

Merehaven Bank is wholly fictional. Every customer, dataset, metric, incident, committee and architecture attributed to it is synthetic. Library interfaces and code fragments are learning specimens; pin the selected runtime and verify current official documentation before use. The edition explains general methods and does not reproduce or imitate any named writer’s voice.

The delayed-outcome problem

Suppose a model estimates the probability that a small business will miss a repayment. The score arrives today; a reliable outcome may arrive months later. The credit team sees a neat decimal and a ranked list. That apparent precision hides at least six choices: who entered the training population, what counted as default, which evidence existed at prediction time, how missing data was encoded, what errors cost, and where a human may override or abstain.

The model can be statistically competent and operationally wrong. It can rank well but be badly calibrated. It can be calibrated overall while failing a small segment. It can pass an offline test and drift after policy changes. The useful object is therefore the whole decision system, not the fitted estimator.

The learning contract

Before training, write a compact contract with six fields: decision, unit, horizon, target, eligible evidence and error costs. Add a seventh field for abstention. This contract turns modelling into a falsifiable engineering task and provides the spine for evaluation, governance and monitoring.

Part I: Define the learning problem

A model can only be evaluated against a declared problem. The opening chapters turn observations, labels, probability and optimisation into an explicit learning contract.

Start with the learning contract

Why do we call it “learning” at all?

Machine learning fits parameters from examples so that a declared mapping can be tested on observations that were not used for fitting. The word learning names that parameter-estimation process; it does not imply understanding, intention or authority.

A useful specification names the observation unit, target, prediction time, eligible evidence, loss, validation population and abstention route. Without those fields, an apparently strong model may be answering a different question from the one the operating decision requires.

Notice what is missing from that definition. There is no mention of neurons. No mention of GPUs. No mention of large language models, attention, or any of the vocabulary that tends to dominate the conversation. Machine learning is older, broader, and stranger than any single technique. A decision tree trained on twelve features of a small business loan is machine learning. A logistic regression on credit bureau data is machine learning. A 400-billion-parameter transformer reading a management accounts PDF is also machine learning. The family is held together by one idea: the program’s behaviour comes from data rather than from rules that a person wrote down.

Think of it as the difference between two ways of building a fraud detection system.

Way one, rules-based. A senior fraud analyst sits with a business analyst for six weeks. They write things like: “If the transaction is over £500 and the merchant country is not the card country and the card has not been used abroad in the last ninety days, flag for review.” They end up with about two hundred rules. The rules run in a decision engine. When fraud patterns change, the analyst writes new rules. Everything the system does, a human somewhere chose.

Way two, learned. An engineer gathers five years of historical transactions, with a label on each one indicating whether it was later confirmed as fraudulent. She feeds the lot into a model. The model figures out, on its own, which combinations of features predict fraud. When she looks at what the model has learned, she sometimes sees things that surprise even the senior fraud analyst. Nobody wrote those rules. The model extracted them from the data.

Both ways can catch fraud. The second way scales in a particular direction: as you feed it more data, it can keep improving, because the model is not bottlenecked by the analyst’s time. That’s why, in practice, every large bank today runs both approaches in parallel, with rules doing the things that regulators insist on seeing explicitly and learned models doing the things that are too subtle or too fluid for rules to capture.

Read this as two parallel pipelines. The top one depends on a human to write rules, so its ceiling is the human’s attention. The bottom one depends on data and an algorithm, so its ceiling is the data’s quality and the algorithm’s capacity.

Formally, what we mean by “the machine learned” is this: there exists an algorithm A and a dataset D such that A(D) produces a function f, and f, when fed new inputs drawn from the same kind of world as D, produces useful outputs. That is literally all that “learning” means in this context. There is no consciousness, no understanding, no intent. Just a function built from examples.

Where this goes wrong. The most common failure mode of machine learning in banking is not model accuracy. It is model applicability. A team spends six months building a beautiful model that predicts default on SME term loans from 2015 to 2019, then deploys it in 2024 after two years of post-pandemic shocks have reshuffled the distribution of customers, and the model quietly stops working because the world it was trained on is no longer the world it operates in. This failure mode has a name: distribution shift. We will return to it repeatedly. For now, remember that a learned function is only as good as the match between the data it learned from and the data it will see.

If asked: “What’s the difference between machine learning and traditional software?”

What makes a learning problem “supervised”?

The word “supervised” is a bit misleading, and that’s worth ten minutes of clarity.

It comes from the following picture. Imagine a graduate student in machine learning in the late 1980s, sat in a basement at Carnegie Mellon, trying to build a program that can tell handwritten digits apart. She has a drawer full of index cards, each with a digit scrawled on it, and on the back of each card she has written what the digit actually is. She shows the front of a card to her program. The program guesses. She flips the card, checks the back, and if the program got it wrong, she tells it so, and her training procedure nudges the program’s internal state in a direction that would have produced the right answer. The backs of the cards are the “supervision.” A teacher is telling the student whether each answer is right or wrong.

That’s supervised learning. The formal setup is this. You have a dataset that looks like pairs:

{(x₁, y₁), (x₂, y₂), …, (x_N, y_N)}

Each x_i is a feature vector, which is just a list of numbers describing one example. Each y_i is a label, the correct answer for that example. The goal is to produce a function f such that, for a new feature vector x that was not in the training set, f(x) is a good guess for what the label should be.

Let’s make this concrete for credit risk. Imagine you are building a probability-of-default model for small business term loans In the synthetic Merehaven case. Each “example” is a loan that was originated some time in the last ten years. For each loan you have a feature vector that might look like this:

So x_i is the vector (250, 60, 1.45, 7, 3, 0.62, 820) for one loan i. Notice the convention: the subscript i indexes the loan, and the superscript (j) in parentheses indexes the feature. The loan in position 1 has a height, weight, sector and score, and those slots always mean the same thing for every loan. The first slot is always loan amount. The seventh slot is always the director’s score. If you scramble the slots, the model cannot learn anything, because the meaning of each position would be inconsistent across examples.

And the label y_i? In this case it is a simple binary: y_i = 1 if the loan went into default in the first 24 months after origination, and y_i = 0 otherwise. So our full dataset is a big table where each row is a historical loan, the first seven columns are features, and the last column is the outcome.

The goal of supervised learning is to find a function f such that given a new x (features of a loan application we have not yet approved), f(x) predicts the y (whether it is likely to default). Notice how everything is anchored on the fact that in the training data, we knew both the features and the outcome. The model never trains without knowing the answer on the training examples. That is the supervision.

Read this as two phases. During training you need both features and labels. During prediction you only need features, because the whole point is that the label is what you are trying to guess.

A subtle but critical point about labels. In the credit risk example, the label is “did this loan default within 24 months?” That sounds simple until you realise what it implies. To label a loan from January 2022, you must wait until at least January 2024 to know its true outcome. This means the freshest training data you can use for a 24-month default model is always at least two years stale. This is not a bug you can engineer away. It is a structural feature of the problem and it has huge consequences for how often you can retrain, how you monitor drift, and how you handle regime changes like the 2022 interest rate shock. Every time you look at a supervised model in finance, ask: “How was the label generated, and what is the lag?”

Formal statement. A supervised learning algorithm takes a dataset D = {(x_i, y_i)}_i = 1^N and produces a function f : ℝ^D → 𝒴, where 𝒴 is the output space. When 𝒴 is a finite set of categories, the problem is called classification. When 𝒴 is a real number, the problem is called regression. When 𝒴 is something stranger like a sequence or a tree, the problem is called structured prediction. Credit default (yes/no) is classification. Loss given default (a percentage) is regression. Generating a credit memo paragraph is structured prediction. Same framework, different output shapes.

Where this goes wrong. A very common failure in banking supervised learning is label leakage. Suppose you are predicting default and one of your features is “has a workout case open.” That feature is enormously predictive, but only because workout cases are opened after default has already been detected. Your model will look spectacular in backtest and catastrophic in production. Finding leakage is tedious, adversarial work, and it is one of the reasons senior ML engineers in regulated industries spend more time on data diligence than on model architecture.

If asked: “In a supervised classification problem, how do you spot label leakage?”

What if you have no labels at all?

“We’ve got six million active business customers. We want to design differentiated service propositions. Marketing wants five to eight segments. What do we have to work with?”

“We’ve got transaction history, product holdings, channel usage, industry code, turnover bands, tenure, digital logins.”

“What’s the label?”

“There isn’t one.”

“What do you mean there isn’t one?”

“There’s no pre-existing category. Marketing wants us to invent the segments.”

That’s an unsupervised learning problem. Unsupervised learning is what you do when your dataset is just {x_i}_i = 1^N, with no y_i to lean on. You are not trying to predict a known answer. You are trying to discover structure.

Unsupervised learning has three main flavours that you should be able to name on demand:

1. Clustering. Group the examples into piles such that examples within a pile are similar to each other and examples across piles are different. Customer segmentation is the canonical example.

2. Dimensionality reduction. Take a feature vector with hundreds or thousands of dimensions and compress it to a feature vector with just a handful, while preserving as much of the interesting structure as possible. If you have ever looked at a PCA plot of your customers and seen them spread out on a two-dimensional scatter you could actually read, that’s dimensionality reduction at work.

3. Outlier detection. Given the “typical” pattern of the data, output a score for each example indicating how weird it is. Anti-money-laundering (AML) transaction monitoring leans heavily on outlier detection. The system doesn’t have a labelled list of money launderers, because criminals don’t helpfully tag their transactions. Instead, the system builds a model of what “normal” looks like for each customer segment and flags transactions that deviate.

This diagram is the decision tree I actually draw on a whiteboard whenever someone says “we don’t have labels.” You ask what they are trying to achieve, and the answer usually lands cleanly in one of three branches.

The Merehaven Bank AML example, concretely. The bank processes tens of millions of transactions daily. Only a tiny fraction are suspicious. Labelling all transactions would be impossible. Instead, an outlier detection model builds, for each customer, a notion of their normal transaction profile: usual counterparties, usual amounts, usual times of day, usual seasonality. When a transaction comes in that is far from that profile, the model emits an anomaly score. Analysts then investigate the top-scoring anomalies. Over time, the analysts’ verdicts (was this actually suspicious?) can be fed back to create a supervised layer on top of the unsupervised one. This stacking of unsupervised and supervised is how most mature AML systems work in practice.

Where this goes wrong. Unsupervised learning has a much softer notion of “right answer” than supervised learning does. If a clustering algorithm gives you six customer segments, there is no ground truth to compare them against. You cannot simply compute an accuracy. This means evaluating unsupervised models is a judgement call, involving stability tests, silhouette scores, downstream business usefulness, and stakeholder buy-in. New data scientists sometimes present a clustering result with the confidence of a classifier’s accuracy number, and senior reviewers rightly push back. Know the difference.

If asked: “Why would you use unsupervised learning over supervised learning if you had a choice?”

Can half a label be better than none?

Here is a historical thread that most engineers don’t know.

In the late 1990s, a group of researchers at CMU and elsewhere became curious about a strange question. Imagine you are training a spam classifier and you have a hundred labelled emails but a million unlabelled ones. Intuitively, the unlabelled emails should be useless, because they don’t tell you anything about the label. But a series of papers in the early 2000s showed that this intuition was wrong. If you used the unlabelled data to learn something about the shape of the input distribution, and then used the labelled data to attach that shape to the correct class, you could sometimes outperform a model that had only the hundred labelled examples. The technique has various names, and the umbrella term is semi-supervised learning.

Let’s anchor this in a banking analogy.

Imagine you are training a model to predict whether a small business will take up a new FX hedging product. You only have labels on the 3,000 SMEs your analysts have actively pitched the product to. But you have 400,000 SMEs in your data warehouse with full transaction and product usage histories. A pure supervised model would only learn from the 3,000. A semi-supervised model finds ways to use the full 400,000 to shape its understanding of what “SME” means in your data, and then uses the 3,000 labels to calibrate a decision boundary within that shape.

Why does this work? Intuitively, because the unlabelled data tells you where the customers are in feature space. It reveals clusters, density, and structure. The labels tell you which side of a line you want to be on. With only the labels, you are drawing a line through a void. With the unlabelled data, you can at least see where the crowd stands before deciding where to cut.

The power of this idea, once you see it, is hard to overstate. It is the intuition behind many modern techniques including self-training, consistency regularisation, and in a looser form, the whole pre-training / fine-tuning paradigm that underpins large language models. The Merehaven Credit Workbench sitting on your analysts’s screen is, at its foundation, a semi-supervised descendant: a huge language model pre-trained on unlabelled text, then fine-tuned on a much smaller labelled dataset of credit memos, chat transcripts, and policy documents.

Where this goes wrong. Semi-supervised learning makes an assumption called the smoothness assumption: examples that are close in feature space should have similar labels. When that assumption fails, semi-supervised methods can make your model worse than using labels alone. A classic case in credit risk is where two clusters of SMEs look structurally similar on surface features (size, sector, turnover) but have wildly different default rates because of a hidden feature like management quality or customer concentration. A semi-supervised model will happily smooth the labels across the cluster and smuggle in errors the pure supervised model would never have made.

If asked: “When is semi-supervised learning a bad idea?”

How does a machine learn to play the long game?

Step into a completely different kind of problem. Forget predicting a static label. Imagine instead that you are teaching a computer to manage a portfolio of FX positions overnight. At every moment it can buy, sell, or hold. Every action it takes moves the portfolio into a new state. At the end of the week, it has made or lost money. You don’t want to tell it “this exact action was correct” because there is no single correct action. You want to tell it “at the end of the week, you were up three percent, well done” or “at the end of the week, you lost two percent, learn from that.”

This is reinforcement learning, and it has a vocabulary worth memorising.

• The environment is the world the agent acts in. For FX it’s the market. For a robot it’s the room. For a game it’s the game.
• The state is the feature vector describing the environment at a given moment.
• An action is something the agent can do in that state.
• A reward is a numerical signal the environment hands back, usually small at each step and occasionally large at key moments.
• A policy is a function that takes a state as input and outputs an action, or a distribution over actions.
• The goal of reinforcement learning is to learn a policy that maximises the expected cumulative reward over time.

Read this as a loop. The agent looks at the current state, picks an action according to its policy, the environment responds with a new state and a reward, and the agent updates its policy to do better next time. The interesting difficulty is that the reward today might be because of an action ten steps ago, not the most recent one. This is called the credit assignment problem and it’s the central technical challenge of the field.

Banking applications of reinforcement learning are real but narrower than hype suggests. The genuine wins are in sequential decision problems where you control something and get measurable feedback. Market-making engines on a trading desk. Order execution algorithms that decide how to break up a large parent order. Portfolio allocation strategies that rebalance over time. Dynamic pricing of short-term savings products. In all of these, the key ingredient is a reliable simulator or a safe testbed, because letting a raw RL agent learn on live capital is a career-ending decision. Most production RL systems in banks are trained in simulation for millions of episodes, then deployed with tight guardrails and constant monitoring.

Where RL is not a fit: anything where you don’t have a clear reward, anywhere the environment is hostile to experimentation, anywhere an individual wrong action has consequences you cannot undo. You don’t use RL to decide whether to approve a mortgage, because you can’t “experiment” on people’s home purchases. You might use RL inside the Merehaven Credit Workbench to decide which three pieces of information to show the analyst first, because that’s a cheap and reversible decision.

If asked: “Why is reinforcement learning harder than supervised learning?”

How does supervised learning actually work, step by step?

We have done definitions. Now let’s trace a complete supervised learning project end to end, with numbers, and make sure every link in the chain is concrete.

We’re building a simplified probability-of-default model for SME term loans In the synthetic Merehaven case. Let’s walk it in six steps.

Step 1: Define the problem. What exactly are we predicting? The candidate definitions include “will the loan default within the first twelve months after origination?” or “within 24 months?” or “will the loan enter forbearance at any point?” Each choice gives you a different label, a different time horizon, and a different downstream use case. For this walkthrough we pick 24-month default, because that matches Merehaven Bank’ internal rating horizon for commercial SME lending.

Step 2: Gather the data. You go to the data warehouse. You pull every SME term loan originated between January 2012 and December 2021. You get, say, 180,000 loans. For each one you collect: the features at origination (what we knew about the loan and borrower at the point of approval) and the label (did the loan default in the first 24 months?). You cut off 2022 and later because the 24-month window is not yet complete for those loans, so you cannot label them honestly.

Step 3: Clean and engineer features. This step takes at least half the total project time and is never glamorous. You deal with missing values (some loans have no director’s score because the director was international). You handle categorical features like sector code by converting them to one-hot encodings. You drop features that leak (workout flags, anything touched by collections). You normalise numerical features so they are on comparable scales. You handle outliers. You discover that a non-trivial percentage of your historical DSCR values are wrong because of a change in how Merehaven Bank calculated the ratio in 2017. You fix them. You document everything.

Step 4: Split the data. You do not train and evaluate on the same data, ever. You split your 180,000 loans into training (say, 70 percent), validation (15 percent) and test (15 percent). importantly, because this is time-series data, you split by origination date, not randomly. The training set is 2012 to 2018 loans. The validation set is 2019 loans. The test set is 2020 and 2021 loans. This mimics how the model will actually be used in production: trained on the past, evaluated on more recent data.

Step 5: Choose an algorithm and train. You pick a model family. For this walkthrough we will use SVM, which we study in the next section. You feed the training set to the SVM training procedure. The procedure finds the parameters w^* and b^* that separate defaulters from non-defaulters as well as possible, under the algorithm’s specific definition of “well.”

Step 6: Evaluate and iterate. You score the validation set with your trained model. You look at accuracy, precision, recall, AUC, and business-relevant measures like expected loss at a chosen approval cutoff. If the performance is acceptable, you re-run everything on the test set (once and only once, to get an unbiased estimate). If not, you go back and adjust: more features, a different algorithm, different hyperparameters. You iterate on the training and validation sets. You touch the test set at the very end.

This is a loop inside a loop. The inner loop is training, the middle loop is feature and hyperparameter iteration, and the test set sits outside both loops as the final judgement. If you ever show someone a model that was evaluated on data the iteration loop touched, you are lying with statistics and senior reviewers will catch you.

The same principle applies across banking. A credit memo PDF becomes a feature vector by extracting numerical fields (turnover, EBITDA, headcount) and one-hot encoding categorical fields (sector, region, entity type). A customer’s twelve-month transaction history becomes a feature vector by computing summary statistics (count, sum, mean, max, standard deviation) across several categories (direct debits, card spend, transfers in, transfers out). A management accounts spreadsheet becomes a feature vector by mapping line items to a canonical chart of accounts and taking the most recent values and ratios. The art is in choosing which summaries preserve the predictive signal and which destroy it.

Where this goes wrong. The single most common production failure in bank ML systems is training-serving skew in feature engineering. The training pipeline, written by a data scientist, computes DSCR one way. The serving pipeline, written three months later by an engineer who was not in the original room, computes DSCR a slightly different way. Both are “DSCR” in name. Neither team notices. In production the model performs worse than in backtest by a measurable margin. The fix is to have exactly one piece of code that computes features, and to call it from both training and serving. This pattern is sometimes called a feature store, and it is now the backbone of most serious bank ML platforms.

If asked: “What’s the biggest production risk in a supervised learning system you haven’t seen mentioned yet?”

Why does a line separate good borrowers from bad ones?

Now for the geometry.

Let’s say you have done the credit risk pipeline above, and you have 180,000 loans, each represented as a feature vector in seven dimensions. For simplicity of visualisation, pretend there are only two features: debt service coverage ratio (DSCR) on the x-axis, and loan-to-value (LTV) on the y-axis. Plot every historical loan as a point. Colour the defaulters red and the non-defaulters blue.

In 1963, Vladimir Vapnik and Alexey Chervonenkis, working in Moscow at the Institute of Control Sciences, were thinking about exactly this kind of picture. They asked a deceptively simple question. If the red points and blue points are separable by a straight line, which line should you draw?

There are infinitely many lines that separate them. Any of them would classify the training data perfectly. Vapnik’s insight, which took decades to mature into the Support Vector Machine in the 1990s, was that not all separating lines are equal. The best line is the one that is as far as possible from the nearest point of either class. The “as far as possible” matters, and we will see why in a moment.

First the analogy. Imagine you are driving a car down a winding country lane in Yorkshire. On one side is a stone wall. On the other is a ditch. You have a choice of where to aim the car. You could drive two inches from the wall and fourteen feet from the ditch. You could drive fourteen feet from the wall and two inches from the ditch. Or you could drive right down the middle, eight feet from each. Which do you choose? Obviously the middle. Why? Because you might flinch. The car might wobble. A fox might run out. The middle gives you the most margin for error. That is the core intuition of the Support Vector Machine.

Now the math, slowly.

The equation of a line in two dimensions can be written as w⁽¹⁾x⁽¹⁾ + w⁽²⁾x⁽²⁾ − b = 0, where w⁽¹⁾ and w⁽²⁾ are real numbers and b is another real number. The two w values control the line’s slope, and b controls where it sits. Any point (x⁽¹⁾, x⁽²⁾) that satisfies the equation is exactly on the line. Points with w⁽¹⁾x⁽¹⁾ + w⁽²⁾x⁽²⁾ − b > 0 are on one side. Points with w⁽¹⁾x⁽¹⁾ + w⁽²⁾x⁽²⁾ − b < 0 are on the other side.

In more than two dimensions the same equation works, we just have more terms:

w ⋅ x − b = 0

where w ⋅ x (read “w dot x”) is shorthand for w⁽¹⁾x⁽¹⁾ + w⁽²⁾x⁽²⁾ + … + w^(D)x^(D), summing the products of corresponding entries. In two dimensions this equation describes a line. In three dimensions it describes a plane. In D dimensions it describes a hyperplane, which is a (D − 1)-dimensional flat surface. For our seven-feature credit model, the decision boundary is a six-dimensional hyperplane sitting inside seven-dimensional space. You cannot picture this. Nobody can. But the algebra works exactly the same as the two-dimensional case, and the two-dimensional case is something you can draw.

The SVM then predicts the label of a new point x like this:

y = sign(w ⋅ x − b)

where sign returns +1 if the input is positive and −1 if the input is negative. So the prediction depends only on which side of the hyperplane the point falls on. One side is the “will not default” side. The other side is the “will default” side. The output is binary.

Let’s compute this by hand for one loan. Suppose our two features are DSCR and LTV, the trained parameters are w = (2.0, −3.0) and b = 0.5, and we have a new loan with DSCR  = 1.4 and LTV  = 0.6. Then:

w ⋅ x − b = (2.0)(1.4) + (−3.0)(0.6) − 0.5 = 2.8 − 1.8 − 0.5 = 0.5

The result is positive, so sign(0.5) = +1. Under our labelling convention this means “predicted non-defaulter.” If we flip the features to a riskier loan with DSCR  = 0.9 and LTV  = 0.85:

w ⋅ x − b = (2.0)(0.9) + (−3.0)(0.85) − 0.5 = 1.8 − 2.55 − 0.5 = −1.25

The result is negative, so sign(−1.25) = −1, predicted defaulter. Notice what the signs of the weights are telling us: the positive weight on DSCR means higher DSCR makes the score more positive (good), and the negative weight on LTV means higher LTV makes the score more negative (bad). The model has learned the intuitions a credit analyst would have.

That is the prediction step. We still have not explained how w and b get chosen. That’s the training step, and it’s where the margin comes in.

If asked: “What does it mean for two classes to be ‘linearly separable’ and why does it matter?”

What makes the best line the widest line?

Here’s the deeper question. There are infinitely many hyperplanes that separate the red and blue points in a linearly separable problem. Any of them achieves perfect training accuracy. Why should we prefer the one with the largest margin?

The answer is the most important single insight in classical machine learning, and it is worth sitting with.

Claim: A separating hyperplane that is far from the nearest training point is more likely to correctly classify new, unseen points than a separating hyperplane that is close to the nearest training point.

Argument: New points do not appear out of nowhere. They are drawn from the same underlying distribution as the training points. So if the new points come from the same world as the training points, they will tend to land near the training points, not far from them. A hyperplane that is far from all training points leaves a buffer around each point. New points that land slightly away from their training cousins will still be on the right side. A hyperplane that is tight against the training points has no buffer. The first new point that lands slightly differently from its training cousin will flip sides.

That is why we want the largest margin.

Formally, SVM defines the margin as the distance between two parallel hyperplanes that just touch the nearest points of each class. You can picture it as the width of the corridor between the two classes, centred on the decision boundary.

Read this as a corridor. The defaulters sit on one side. The non-defaulters sit on the other side. The decision boundary runs down the middle. The margin is the half-width of the corridor. SVM chooses the w and b that make the corridor as wide as possible while still correctly separating the training points.

The mathematical trick to make this work is elegant. You choose to scale w and b such that the parallel hyperplanes are at w ⋅ x − b = +1 and w ⋅ x − b = −1. With that convention, geometry gives you the width of the corridor as $\frac{2}{\|w\|}$, where ∥w∥ is the Euclidean norm of w, defined as $\sqrt{\sum_{j=1}^{D} (w^{(j)})^2}$. This is just the ordinary length of the vector w in standard geometric terms: square each component, add them up, take the square root.

So to maximise the corridor width $\frac{2}{\|w\|}$, you minimise ∥w∥. And you have to do it while keeping every training point correctly classified. The constraint for a correctly-classified positive example (y_i = +1) is that it lies on or beyond the +1 hyperplane: w ⋅ x_i − b ≥ +1. For a correctly-classified negative example (y_i = −1), the constraint is that it lies on or beyond the −1 hyperplane: w ⋅ x_i − b ≤ −1. Both can be written in a single compact form:

y_i(w ⋅ x_i − b) ≥ 1  for all i = 1, …, N

Check the algebra on a positive example: y_i = +1 gives (+1)(w ⋅ x_i − b) ≥ 1, which simplifies to w ⋅ x_i − b ≥ 1. Check on a negative example: y_i = −1 gives (−1)(w ⋅ x_i − b) ≥ 1, which rearranges to w ⋅ x_i − b ≤ −1. Both constraints captured in one line. That is the compact SVM optimisation problem:

minimise ∥w∥  subject to y_i(w ⋅ x_i − b) ≥ 1 for i = 1, …, N

Machines are very good at solving problems of this form. There is a whole field, convex optimisation, dedicated to them, and the specific structure of this problem (quadratic objective, linear constraints) means that industrial solvers can handle it reliably at scale. The solution, a specific pair w^* and b^*, is your trained SVM. Plug it into f(x) = sign(w^* ⋅ x − b^*) and you have a classifier.

Reading the optimisation in plain English. “Find the vector w that is as short as possible, subject to the constraint that all positive training examples score at least +1 and all negative training examples score at most -1.” Short w means wide corridor. Correct classification of all training points means the constraints are met. The solver finds the sweet spot where both conditions hold and the corridor is as wide as the data allows.

When real data is not perfectly separable, which is always the case in credit risk because a small fraction of very healthy-looking borrowers default for reasons that have nothing to do with the features you observe, the pure SVM has no solution. The fix is the soft-margin SVM, which introduces a penalty hyperparameter C that allows some points to sit inside the corridor or on the wrong side of the boundary, at a cost. Setting C is a trade-off between corridor width and training accuracy, and it is one of the things you tune during the validation loop.

Where this goes wrong. In high-dimensional problems with few training points, SVM (and every other classifier) suffers from a phenomenon called the curse of dimensionality. As the number of features grows, the volume of feature space grows exponentially, and any fixed number of training points becomes very sparse relative to the space they sit in. The “margin” intuition that worked beautifully in two dimensions becomes much weaker in two thousand. A concrete symptom: a spam classifier trained on a 20,000-word vocabulary with only 10,000 emails is already in a regime where fitting a good hyperplane is statistically fraught. The mitigations are feature selection, regularisation, and gathering much more data. In banking this means that throwing every column of the data warehouse at the model is a bad default, even though it feels like giving the model “more information.”

If asked: “Why does SVM try to maximise the margin rather than just find any separating hyperplane?”

Why does any of this work on borrowers we’ve never seen?

This is the deepest question in the chapter and it deserves a slow answer.

Here is the puzzle, stated plainly. We took 180,000 historical SME loans, we trained a model on them, and we now apply that model to a brand new SME loan that walks in the door tomorrow. The model has never seen this specific borrower. There is no principled reason, from pure logic alone, to believe that what worked on past loans will work on this one. So why does it?

The answer has two parts, and missing either one is what gets data scientists in trouble.

Part one: the independence and identical distribution assumption, usually written i.i.d. in textbooks. Supervised learning rests on the belief that the training data and the future test data are drawn independently from the same underlying probability distribution. “Same underlying distribution” means there is some fixed statistical process producing SME loans and their default outcomes, and every loan we see, past or future, is a fresh sample from that process. “Independently” means one loan’s outcome doesn’t directly depend on another’s. If the assumption holds, then patterns learned on past samples will hold on future samples, because they are samples from the same population.

Part two: the patterns the case shows must be well-tested. Even if the distribution is stable, our model might have memorised quirks of the training data rather than general patterns. A model that memorises quirks is said to be overfitting. The margin-maximisation principle of SVM is one specific defence against overfitting: by preferring the widest separating corridor, we are preferring simpler decision boundaries that don’t contort themselves around individual training points. Simpler models that fit the training data almost as well as complicated ones usually generalise better, which is the modern formalisation of a principle that goes back to William of Ockham in the fourteenth century: prefer simpler explanations. In machine learning this is called the bias-variance tradeoff, and we will meet it repeatedly.

Read this as the two pillars holding up generalisation. Kick either one and the building falls. This diagram is the single most important idea in the chapter; internalise it.

A concrete walkthrough of why the i.i.d. assumption matters in banking. Suppose you train a default model on 2015-2019 SME loans. You deploy it in January 2022. By mid-2022 the Bank of England has raised rates multiple times, energy prices have spiked, and the SMEs that were comfortably meeting their debt service on three percent loans are struggling on six. The borrowers in the test-time distribution (2022) are no longer behaving like the borrowers in the training distribution (2015-2019). The i.i.d. assumption has been violated by a regime change in the macro environment. Your model’s performance will quietly degrade. The fix is not “better features” or “more data from 2015-2019.” The fix is to retrain on more recent data, accept that the model will always lag reality, and monitor for drift so you know when retraining is overdue.

This is the single most important operational discipline in production ML. Models that were excellent at launch are mediocre a year later and dangerous two years later, not because the code has rotted but because the world has moved. The industry term for this is concept drift (when the relationship between features and labels changes) or covariate shift (when the distribution of features changes), and mature organisations have dashboards that track both on their live models.

A concrete walkthrough of why simplicity matters. Suppose you train a very complicated model, perhaps a deep neural network with millions of parameters, on those same 180,000 loans. With enough capacity, the model can memorise every training example perfectly and achieve 100 percent training accuracy. When you evaluate on the validation set, accuracy is much worse. What happened? The model learned the training set, not the underlying pattern. It treated the specific noise in the 2016 loans as signal. It fit a twisted, elaborate boundary that hugged every single training point. On new points that are slightly different, the twisted boundary misclassifies. This is overfitting. The SVM’s margin principle explicitly fights overfitting by pushing for the smoothest, flattest separating boundary consistent with the training data.

The take-home. Supervised learning works because of two big bets: a statistical bet that the world is stable enough that yesterday’s patterns apply tomorrow, and a philosophical bet that among patterns consistent with the training data, the simpler ones are more likely to be real. Neither bet is guaranteed to pay off. Both usually do, and the places where they don’t are where real engineering judgement matters.

If asked: “How would you detect that a production credit model has started to drift?”

Read the mathematics as operations

What are scalars, vectors, and matrices really?

Let’s start with the containers. Every single number that ever enters a machine learning system sits inside one of three containers, or something built on top of them.

A scalar is just a single number. Written in italic lowercase: x, a, b, α, θ. A customer’s age (42) is a scalar. A loan amount (£250,000) is a scalar. A probability of default (0.037) is a scalar. A hyperparameter like the SVM penalty C (1.0) is a scalar. If it’s one number, it’s a scalar.

A vector is an ordered list of scalars. Written in bold lowercase: x, w, b. The order matters: the first entry means the same thing across every vector in your dataset, and you can’t shuffle them without breaking the meaning. You can think of a vector in two equivalent ways: as an arrow in space pointing in some direction, or as a point in space. For machine learning the “point” intuition is more useful more often, but you’ll see the “arrow” intuition used whenever we talk about gradients or directions of change.

The individual entries of a vector are called attributes or features or components depending on the author’s taste. We write the j-th attribute of vector x as x^(j). The parenthesised superscript is just a position index. It is not a power. So x⁽²⁾ is the second attribute of x, whereas x² would mean x squared. When we need to talk about both the example index and the feature index, we stack them: x_i^(j) is the j-th feature of the i-th example. The i is which customer. The (j) is which of their attributes.

Here is one concrete vector from the Merehaven Bank data warehouse, representing a single SME customer:

x_i = [250, 60, 1.45, 7, 3, 0.62, 820]

This is a seven-dimensional vector. The first component (x_i⁽¹⁾) is the loan amount in thousands. The second (x_i⁽²⁾) is the term in months. The third (x_i⁽³⁾) is the debt service coverage ratio. And so on, exactly as we described in Chapter 1. The subscript i identifies which customer this is among the tens of thousands in the portfolio. The (j) positions are fixed: wherever you see a vector in this dataset, position 3 always means DSCR.

A vector is not just a tuple. It has algebraic structure. You can add two vectors to get another vector. You can multiply a vector by a scalar to get another vector. You can take the dot product of two vectors to get a scalar. These operations have geometric meanings (the sum of two arrows is their tip-to-tail composition, scalar multiplication stretches or shrinks an arrow, dot product measures how much one arrow projects onto another). We’ll use all of them.

A matrix is a rectangular grid of numbers: rows and columns. Written in bold uppercase: X, W, A. You can think of a matrix as a collection of vectors stacked together, which is how datasets are usually stored. If you have N customers and each one has D features, the whole dataset is a matrix X with N rows and D columns:

$$\mathbf{X} = \begin{bmatrix} x_1^{(1)} & x_1^{(2)} & \cdots & x_1^{(D)} \\ x_2^{(1)} & x_2^{(2)} & \cdots & x_2^{(D)} \\ \vdots & \vdots & \ddots & \vdots \\ x_N^{(1)} & x_N^{(2)} & \cdots & x_N^{(D)} \end{bmatrix}$$

Row i is customer i’s feature vector. Column j is the j-th feature across all customers. This is literally how pandas and SQL represent your data: rows are records, columns are fields. The matrix notation is just giving it a symbolic name so we can manipulate the whole dataset in one go.

Concrete walkthrough In the synthetic Merehaven case scale. The SME lending portfolio might have 180,000 active customers, each with 240 features after all the engineering is done. Then X is a 180, 000 × 240 matrix. That’s over 43 million numbers. Stored in 32-bit floats, it’s about 170 MB of data. You can load it into memory on a laptop. You couldn’t do this easily in 1995. The fact that you can today is why modern machine learning works.

Sets round out the basic containers. A set is an unordered collection of unique elements, written with calligraphic capitals like 𝒮 or 𝒟. Curly braces list the elements: {1, 3, 5, 8}. The training dataset in Chapter 1 was a set of pairs: 𝒟 = {(x₁, y₁), (x₂, y₂), …, (x_N, y_N)}. The key difference between a set and a vector is that sets have no order and no duplicates. {3, 1, 1, 5} is the same set as {1, 3, 5}. Vectors, by contrast, care about order and happily repeat values: [3, 1, 1, 5] is different from [1, 3, 5].

You’ll also see set operations occasionally. Intersection, written 𝒮₁ ∩ 𝒮₂, gives you the elements that appear in both sets. Union, written 𝒮₁ ∪ 𝒮₂, gives you the elements in either set. Cardinality, written |𝒮|, gives you the number of elements. Membership, written x ∈ 𝒮, asks whether x is in 𝒮. The special set ℝ denotes all real numbers, and the worked design uses it constantly: writing x ∈ ℝ means “x is some real number, could be anything.”

Read this as a hierarchy for ordered numerical data (scalar → vector → matrix) plus sets as a separate unordered container. Almost every data structure you meet in ML is one of these four or a generalisation of them (tensors, which are just matrices in more than two dimensions, are the obvious one).

Where this goes wrong. The most common beginner mistake in code is confusing row-major and column-major layouts, or forgetting whether a vector is a row or a column. In pure notation we are usually careful, but in NumPy a vector created as np.array([1,2,3]) is one-dimensional with no row/column distinction, while np.array(1,2,3) is a 2D row vector and np.array([[1],[2],[3]]) is a 2D column vector. Mixing them causes silent broadcasting bugs that change the semantics of your computation without raising an error. Every experienced engineer has been bitten. The defensive habit is to print the shape of every intermediate tensor during development, and to assert shapes explicitly in production code.

If asked: “What’s the difference between a vector and a set in machine learning notation?”

How do you sum thousands of things without losing the structure?

In machine learning you add things up a lot. Loss over the training set. Weighted sum over the features. Probabilities over outcomes. Gradients over examples. If you had to write these out longhand every time, the notation would collapse under its own weight. So mathematicians invented two shortcuts.

Capital sigma notation, ∑, is the summation shortcut. The letter is a capital Greek sigma, chosen because S suggests “sum.” Below the sigma you put the index variable and its starting value. Above you put the ending value. To the right you put the expression to sum. So:

$$\sum_{i=1}^{N} x_i = x_1 + x_2 + x_3 + \ldots + x_N$$

Read this out loud: “the sum from i equals 1 to N of x_i.” It means: let i take each integer value from 1 to N, compute x_i each time, and add them all up. The index i is a placeholder that exists only inside the sum. It’s called a dummy variable, and you can rename it freely: $\sum_{k=1}^{N} x_k$ means exactly the same thing.

Here’s a banking example that you compute every day somewhere in your bank. The total balance across all current accounts held by a single customer:

$$\text{TotalBalance}_i = \sum_{a=1}^{A_i} \text{balance}_{i,a}$$

where A_i is the number of accounts customer i holds and balance_i, a is the balance on customer i’s a-th account. In English: “sum the balances of all of customer i’s accounts to get their total.” One line of notation, one unambiguous meaning.

Sigma generalises to summing over the attributes of a vector:

$$\sum_{j=1}^{D} w^{(j)} x^{(j)} = w^{(1)} x^{(1)} + w^{(2)} x^{(2)} + \ldots + w^{(D)} x^{(D)}$$

You will see this exact pattern constantly, because this is the dot product of two vectors, and the dot product is the core ingredient of almost every linear model. When you read w ⋅ x or w^⊤x, you are reading this sigma in disguise.

Capital pi notation, ∏, is the same idea for products. The Greek letter is P for “product.” So:

$$\prod_{i=1}^{N} x_i = x_1 \cdot x_2 \cdot x_3 \cdot \ldots \cdot x_N$$

When do you multiply a long list of numbers in ML? Mainly when you compute likelihoods. If you have N independent observations and you want the probability of seeing all of them together, you multiply the individual probabilities. In parameter estimation (coming up later in this section) this is the workhorse.

Concrete walkthrough: a tiny weighted sum. Suppose we have a weight vector w = [0.3, −1.2, 0.05, 2.0] and a feature vector x = [1, 0.5, 200, 0.8]. The weighted sum is:

$$\sum_{j=1}^{4} w^{(j)} x^{(j)} = (0.3)(1) + (-1.2)(0.5) + (0.05)(200) + (2.0)(0.8)$$

 = 0.3 − 0.6 + 10 + 1.6 = 11.3

That single number 11.3 is the “score” produced by this linear model for this particular customer. In practice the weights come from training and the features come from the customer’s record, but the arithmetic is exactly this: multiply pairwise, add everything up.

A nasty thing that happens with products. If you multiply many small probabilities together, the result gets tiny very fast. Multiply a thousand probabilities of size 0.01 and you get 10⁻²⁰⁰⁰, which is smaller than any 64-bit float can represent. The computer returns zero and your algorithm breaks. The standard fix is the log trick: take the logarithm of the whole thing. Logarithms turn products into sums, because log (ab) = log (a) + log (b). So instead of computing ∏_ip_i directly, you compute ∑_ilog p_i, which is numerically well-behaved, and exponentiate at the end if you need to. Every mature ML library does this automatically under the hood, and you will see log-likelihoods instead of likelihoods written everywhere. Now you know why.

If asked: “Why do ML libraries work with log-probabilities instead of probabilities?”

What do you do to vectors?

You now have containers. You need verbs.

Vector addition is element-wise. Two vectors of the same length add component by component:

x + z = [x⁽¹⁾ + z⁽¹⁾, x⁽²⁾ + z⁽²⁾, …, x^(m) + z^(m)]

The result is another vector of the same length. Banking example: if x is the portfolio exposure across sectors this month and z is the change from last month, then x + z is where you are and where you are headed, sector by sector. You can subtract in exactly the same way.

Scalar multiplication stretches a vector. If c is a number and x is a vector, then:

cx = [cx⁽¹⁾, cx⁽²⁾, …, cx^(m)]

Multiply every component by c. If c > 1, the vector gets longer. If 0 < c < 1, it gets shorter. If c is negative, it flips direction. Banking example: you have a small sample of 1,000 SME customers with total annual fee income of £X. To estimate the fee income on a population of 10,000 similar customers, you multiply the sample total by 10. You have just scalar-multiplied a vector of fee amounts.

Dot product is the star of the show. Two vectors of the same length combine into a single scalar:

$$\mathbf{w} \cdot \mathbf{x} = \sum_{j=1}^{D} w^{(j)} x^{(j)}$$

This is exactly the weighted sum you already know. The dot product has a geometric meaning as well as an algebraic one: it equals ∥w∥∥x∥cos θ where θ is the angle between the two vectors. Two vectors pointing in the same direction have a large positive dot product. Two vectors pointing in opposite directions have a large negative dot product. Two perpendicular vectors have a dot product of zero. This matters enormously: the SVM from Chapter 1, the attention mechanism in transformers, the similarity score between two customer embeddings, and the cosine similarity used in recommendation engines are all dot products wearing different hats.

The dot product is also sometimes written w^⊤x, using the transpose notation ⊤. We’ll come back to the transpose in a moment. The two notations, w ⋅ x and w^⊤x, mean the same thing when both vectors are the same length.

Matrix-vector multiplication generalises the dot product. If W is a matrix with m rows and n columns, and x is a vector with n components, then Wx is a new vector with m components:

$$\mathbf{W}\mathbf{x} = \begin{bmatrix} w^{(1,1)} & w^{(1,2)} & w^{(1,3)} \\ w^{(2,1)} & w^{(2,2)} & w^{(2,3)} \end{bmatrix} \begin{bmatrix} x^{(1)} \\ x^{(2)} \\ x^{(3)} \end{bmatrix} = \begin{bmatrix} w^{(1,1)} x^{(1)} + w^{(1,2)} x^{(2)} + w^{(1,3)} x^{(3)} \\ w^{(2,1)} x^{(1)} + w^{(2,2)} x^{(2)} + w^{(2,3)} x^{(3)} \end{bmatrix}$$

The i-th entry of the result is the dot product of the i-th row of W with x. In other words, a matrix-vector multiplication is many dot products at once. A matrix with five rows, multiplied by a vector, gives you a five-dimensional result, which is five separate dot products bundled into one operation.

This pattern appears everywhere. The output of a single layer of a neural network is exactly Wx + b, followed by a nonlinearity. The transformation of a feature vector into a set of basis scores in PCA is a matrix-vector multiply. The scoring of a single credit application against a model with multiple output heads (probability of default, expected loss, probability of prepayment) is a matrix-vector multiply.

The transpose x^⊤ flips a vector from column shape to row shape, or a matrix from m × n to n × m. Concretely, if x is written as a column [x⁽¹⁾; x⁽²⁾; x⁽³⁾], then x^⊤ is the row [x⁽¹⁾, x⁽²⁾, x⁽³⁾]. This matters for matrix multiplication because you can only multiply a matrix by a vector when the inner dimensions match. You may need to transpose one of the operands to make that happen.

Concrete walkthrough: scoring three loans with one model. Suppose your credit model has weights w = [2.0, −3.0] and bias b = 0.5, where the features are DSCR and LTV as before. You have three new loans in your batch:

$$\mathbf{X} = \begin{bmatrix} 1.4 & 0.6 \\ 0.9 & 0.85 \\ 2.1 & 0.4 \end{bmatrix}$$

Row 1 is loan 1, row 2 is loan 2, row 3 is loan 3. The scores are:

$$\mathbf{X}\mathbf{w} = \begin{bmatrix} (1.4)(2.0) + (0.6)(-3.0) \\ (0.9)(2.0) + (0.85)(-3.0) \\ (2.1)(2.0) + (0.4)(-3.0) \end{bmatrix} = \begin{bmatrix} 2.8 - 1.8 \\ 1.8 - 2.55 \\ 4.2 - 1.2 \end{bmatrix} = \begin{bmatrix} 1.0 \\ -0.75 \\ 3.0 \end{bmatrix}$$

Subtract b = 0.5 from each: [0.5, −1.25, 2.5]. Apply the sign function: loan 1 is positive (non-default), loan 2 is negative (default), loan 3 is positive (non-default). Three predictions, one matrix multiply. This is batch scoring, and it’s how every production ML system actually processes inference: not one example at a time but many at once, because GPUs and modern CPUs are brutally efficient at matrix arithmetic.

If asked: “Why is almost all of modern ML built on matrix multiplications?”

What’s a function, and why should you care about its minimum?

Take a breath. You’ve done the containers and the operations. Now we’re going to talk about functions, because the entire training process of every ML model is: define a function that measures how bad your model is, then find the values of parameters that make that function as small as possible.

A function is a rule that takes an input and produces an output. If the function is called f, we write y = f(x), read as “y equals f of x.” The set of allowed inputs is called the domain and the set of possible outputs is called the codomain. You already know functions from school: f(x) = x² takes a real number and squares it. f(x) = 2x + 3 takes a real number and doubles it then adds three.

In machine learning, the functions we care about are usually loss functions or cost functions. Their input is a set of model parameters (weights, biases, thresholds). Their output is a single number measuring how poorly the model with those parameters performs on the training data. The smaller the output, the better the parameters. Our job is to find the parameters that produce the smallest possible output.

A very concrete loss function. Suppose you have a simple one-feature linear regression model predicting a customer’s lifetime value from their current month’s fee income: ŷ = wx + b. For one customer with actual lifetime value y, the squared error is (ŷ − y)² = (wx + b − y)². For a whole training set of N customers, the total squared error is:

$$L(w, b) = \frac{1}{N} \sum_{i=1}^{N} (w x_i + b - y_i)^2$$

L is a function of two parameters, w and b. Given any specific pair, L returns a single number: the average squared error on the training set. Our goal is to find the specific w^* and b^* that minimise L.

Local versus global minimum. A function can have many low points. A local minimum at x = c means that f(x) ≥ f(c) for every x in some small neighbourhood around c: it’s the lowest point you can see if you only look a short distance to either side. A global minimum is the absolute lowest point anywhere in the domain. Every global minimum is also a local minimum, but not vice versa.

This distinction is critical in practice. The SVM from Chapter 1 has a convex loss function, which is a special kind of function whose only local minimum is also its global minimum. This means you can’t get stuck in a mediocre solution while optimising it. Linear regression is also convex. The loss functions of deep neural networks, on the other hand, are non-convex. They have countless local minima. Training a deep network well means not just finding a low point but finding a good one, and much of the art of modern deep learning is about techniques for escaping bad local minima and settling into basins that generalise well.

Read this as a fork. Convex losses are well-behaved and any sensible downhill algorithm converges to the best answer. Non-convex losses require more sophisticated training strategies and there is a whole research literature on them.

Banking example of the local/global distinction. In credit model development, you might define a loss that combines predictive accuracy with fairness constraints and capital efficiency. The combined loss is often non-convex. Two different training runs from different initial weights can end up at two different “solutions” with similar overall loss but different behaviours on slices of the portfolio. Model risk management In the synthetic Merehaven case and every serious bank insists on training stability checks precisely because of this: run the same training procedure several times and verify you get materially similar models, not wildly different ones.

One more bit of notation before we move on. The argmax operator, written arg max_x ∈ Af(x), returns the element of A that makes f as big as possible. The argmin does the same for the minimum. Contrast with max_x ∈ Af(x), which returns the maximum value itself. So max_xf(x) = 10 says “the biggest value f achieves is 10,” while arg max_xf(x) = 3 says “the input that produces that maximum is x = 3.” These are separate things and ML papers use both. In classification, you will see ŷ = arg max_cP(y = c|x) meaning “predict the class whose posterior probability is highest.” The argmax picks the winning class, not its probability.

The assignment operator ← is the last small piece of notation. It means “set the thing on the left to the value of the thing on the right.” So w ← w − 0.01∇L means “update w by subtracting 0.01 times the gradient.” It’s the pseudocode equivalent of w = w - 0.01 * grad_L in actual code. The arrow makes clear this is an action, not an equation.

If asked: “Why do we frame training as minimising a loss function?”

How do you find the direction that goes down fastest?

Finding the minimum of a loss function is the core training problem. For simple convex losses you can often solve it analytically with a closed-form formula (we’ll see this in Chapter 3 for linear regression). For most interesting models you can’t, and you have to find the minimum numerically, by taking many small downhill steps. To take a downhill step you need to know which direction is downhill. That’s what derivatives and gradients are for.

The derivative of a function f of a single variable, written f^′(x) or $\frac{df}{dx}$, is another function that tells you how fast f is changing at each point. If f^′(x) > 0 at some x, the function is going up as you move right. If f^′(x) < 0, it’s going down. If f^′(x) = 0, the function is locally flat: you are at a peak, a valley, or a plateau.

The everyday intuition: imagine you are standing on a hill and you want to know which way is down. You take a tiny step in some direction and see how much your elevation changed. That’s the derivative of elevation with respect to position in that direction. Take the step in the direction where your elevation drops fastest and you are following the negative derivative. Do this repeatedly and you will eventually reach a low point. That is, in one sentence, how every neural network in the world is trained.

Derivatives of simple functions are memorised facts. Three you should know by muscle memory:

• If f(x) = x², then f^′(x) = 2x.
• If f(x) = cx for any constant c, then f^′(x) = c.
• If f(x) = c for any constant, then f^′(x) = 0.

The first one makes sense: at x = 3, the function x² is changing at rate 6, so a small step right of size 0.01 changes the function by approximately 0.06. The second one is obvious: a straight line with slope c is always changing at rate c. The third one is obvious too: a constant function isn’t changing at all.

The chain rule is how you handle derivatives of functions built from other functions. If F(x) = f(g(x)), meaning you first apply g to x and then apply f to the result, then:

F^′(x) = f^′(g(x)) ⋅ g^′(x)

In words: the rate of change of the whole thing is the rate of change of the outer function (evaluated at where the inner function landed) times the rate of change of the inner function. If F(x) = (5x + 1)², let g(x) = 5x + 1 and f(u) = u². Then g^′(x) = 5 and f^′(u) = 2u, so f^′(g(x)) = 2(5x + 1). Multiply: F^′(x) = 2(5x + 1) ⋅ 5 = 50x + 10.

The chain rule matters because modern neural networks are functions composed of many functions composed of many functions, and the way we compute their gradients is by chaining together the derivatives of each layer. The backpropagation algorithm, which we’ll meet properly in Chapter 6, is nothing but the chain rule applied systematically to a deeply nested function.

The gradient generalises the derivative to functions of more than one input. If your function takes a vector x = (x⁽¹⁾, x⁽²⁾, …, x^(D)) and returns a single number, the gradient is a vector whose entries are the partial derivatives, each one computed by pretending all the other inputs are fixed constants:

$$\nabla f = \left[ \frac{\partial f}{\partial x^{(1)}}, \frac{\partial f}{\partial x^{(2)}}, \ldots, \frac{\partial f}{\partial x^{(D)}} \right]$$

The upside-down triangle ∇ is called “nabla” and it’s pronounced “del.” Read ∇f as “the gradient of f” or “del f.” The curly ∂ is the partial derivative sign, distinguishing it from the plain d used for one-variable derivatives.

Concrete walkthrough. Suppose f(x⁽¹⁾, x⁽²⁾) = ax⁽¹⁾ + bx⁽²⁾ + c for constants a, b, c. The partial derivative with respect to x⁽¹⁾ is obtained by treating x⁽²⁾ and c as constants and differentiating as usual: $\frac{\partial f}{\partial x^{(1)}} = a$, because ax⁽¹⁾ differentiates to a and the other terms (now constants) differentiate to zero. Similarly, $\frac{\partial f}{\partial x^{(2)}} = b$. So the gradient is ∇f = [a, b].

The geometric meaning of the gradient is critical. At any point in input space, the gradient vector points in the direction of steepest ascent of the function, and its length tells you how steep the ascent is. The negative gradient, −∇f, points in the direction of steepest descent. If you stand at any point and take a small step in the direction of −∇f, your function value drops as much as any step of that size could possibly make it drop. This is the single most important geometric fact in machine learning and it’s worth staring at until it sinks in.

Read this as the iteration at the heart of essentially every training algorithm in ML. Compute where you are, compute which way is down, step that way by a small amount, repeat. The α is the learning rate, a hyperparameter that controls how big each step is. Too small and training is slow. Too big and you overshoot minima and bounce around. Finding a good learning rate is an art that has generated a whole family of techniques (Adam, RMSProp, learning rate schedules, one-cycle policies) all of which are variations on this basic loop.

Banking example: why the gradient matters for mortgage pricing. Suppose Merehaven Bank wants to set the interest rate on a new fixed-rate mortgage product to maximise risk-adjusted return. The return is a function of the rate offered, the take-up probability at that rate, the default probability, and the funding cost. Each of these depends on the rate in a different way. You write the total expected return as a function R(r) of the rate r, then compute $\frac{dR}{dr}$ and look for the rate where it is zero. That zero-derivative point is a local maximum or minimum of return. Combined with a second-derivative check (or just computing R on a grid), you find the rate that maximises return. No magic. Just calculus applied to a business problem.

Where this goes wrong. In high dimensions, the gradient-descent loop has famous failure modes. The loss surface can have saddle points where the gradient is zero but you are not at a minimum. It can have plateaus where the gradient is tiny and progress stalls. It can have pathological curvature where you oscillate along some directions and crawl along others. Modern optimisers like Adam use per-parameter adaptive learning rates and momentum to mitigate all of these. For the linear and convex models in the next chapter, plain gradient descent is fine. For neural networks in Chapter 6 and beyond, you will care about optimiser choice.

If asked: “Explain the relationship between a derivative, a gradient, and backpropagation in one minute.”

What is randomness, and how do we talk about it with math?

We pivot from the deterministic world of algebra and calculus into the uncertain world of probability. You need this pivot because machine learning is fundamentally about uncertainty. No credit model predicts default with certainty. No fraud detector catches every fraud. Everything we do is a statement about likelihoods and distributions, and probability is the language for those.

A random variable is a variable whose value is the numerical outcome of some random process. It’s written with a capital italic letter, usually X or Y. Examples of random variables that matter In the synthetic Merehaven case:

• X = the number of direct debits on a random customer’s account next month
• Y = the amount of a random ATM withdrawal in pounds
• Z = the indicator of whether a loan defaults in the next 24 months (1 or 0)
• T = the time between a customer’s first and second product purchases, in days

Random variables come in two flavours. A discrete random variable can only take a countable set of values: 0, 1, 2, 3, and so on, or {red, yellow, blue}. The number of direct debits and the default indicator are discrete. A continuous random variable can take any value in some interval of real numbers. The ATM withdrawal amount and the time between purchases are continuous (at least, we model them that way, even though in practice amounts are in pennies).

Probability mass function (PMF). For a discrete random variable, the distribution is given by a probability mass function, which is just a list of probabilities: one for each possible value. We write Pr(X = x_i) for the probability that X takes the specific value x_i. All these probabilities are non-negative, and they sum to 1 because the variable has to take some value. A fair six-sided die has a PMF with six entries of $\frac{1}{6}$ each, summing to 1.

Probability density function (PDF). For a continuous random variable, the probability that X takes any specific exact value is zero, because there are infinitely many possible values. Instead, we describe the distribution by a density function f_X(x) that says how concentrated the probability is near each value. The probability that X falls in an interval [a, b] is the area under the density curve between a and b. The total area under the curve is 1. Densities can be larger than 1 at individual points, as long as the total area is 1: a very concentrated distribution has a very tall narrow peak.

Read this as two parallel worlds. Discrete and continuous random variables have analogous machinery: PMFs instead of PDFs, sums instead of integrals, but the core ideas are the same. Most banking applications mix both: loan amounts are continuous, default indicators are discrete, and a full model has to handle both.

The expectation. For any random variable, the most important single summary is its expected value or expectation, written 𝔼[X]. For a discrete variable, it’s the weighted average of possible values, weighted by their probabilities:

𝔼[X] = ∑_ix_i ⋅ Pr(X = x_i)

For a continuous variable, the sum becomes an integral:

𝔼[X] = ∫_ℝx ⋅ f_X(x) dx

The integral is just the continuous version of the sum. You can read ∫ as “sum over all tiny slices” of the quantity to its right. The expected value is also called the mean or average, and is often denoted by the Greek letter μ (mu).

Concrete walkthrough: expected value of a default outcome. Let Z be the default indicator for a particular loan, with Pr(Z = 1) = 0.03 (a 3% default probability) and Pr(Z = 0) = 0.97. Then:

𝔼[Z] = (1)(0.03) + (0)(0.97) = 0.03

The expected value of a 0/1 indicator is exactly the probability that it is 1. This simple identity is the reason probability of default estimates can be directly plugged into expected loss calculations. Useful fact.

Concrete walkthrough: expected loss. Now imagine the loan has exposure at default of £500,000 and loss given default of 40%. The loss (if default occurs) is £500,000 × 0.4 = £200,000. The loss random variable L takes value £200,000 with probability 0.03 and £0 with probability 0.97. So:

𝔼[L] = (200, 000)(0.03) + (0)(0.97) = 6, 000

The expected loss on this loan is £6,000. If you aggregate this across all loans in the portfolio, you get the portfolio expected loss, which drives provisioning under IFRS 9 and capital under IRB. The expectation operator is doing real work here.

The variance and standard deviation. The expectation tells you the centre of a distribution, but not how spread out it is. For that you want the variance σ², defined as the expected squared deviation from the mean:

σ² = 𝔼[(X − μ)²]

And the standard deviation σ is the square root of the variance. It has the same units as the original variable, which makes it easier to interpret. A portfolio with expected loss £6 million per quarter and standard deviation £500k is tightly concentrated. The same expected loss with standard deviation £5 million is terrifyingly spread out.

For a discrete random variable with a finite set of outcomes, you can compute the variance directly:

σ² = ∑_iPr(X = x_i)(x_i − μ)²

The variance is always non-negative. A variance of zero means the variable is deterministic: it always takes the same value. A large variance means the variable bounces around a lot. In banking, variance of return is the classical measure of risk in portfolio theory, and the Sharpe ratio, the VaR, and the RWA calculations all depend on variance-like quantities somewhere in their derivation.

If asked: “Why are expectation and variance such central concepts in probability-based machine learning?”

How can you trust an estimate from a tiny sample?

Here’s a problem that keeps data scientists honest. You almost never know the true probability distribution of the thing you care about. You have a sample, maybe a few thousand or a few hundred thousand data points, drawn from that distribution. You want to make statements about the distribution itself, not just the sample. How do you do that responsibly?

This is the territory of statistical estimation. The core setup: there is some unknown quantity about the population (call it θ, maybe it’s the true mean, or the true default rate, or a model parameter). You compute some quantity from your sample (call it θ̂, the little hat indicates “estimate”) and you use it as a stand-in for θ. The question is: under what conditions is θ̂ a good stand-in?

An estimator is called unbiased if, on average across all possible samples you could draw, the estimator equals the true value:

𝔼[θ̂(S_X)] = θ

In words: “the expected value of the estimator, where the expectation is taken over the randomness in drawing samples, equals the thing it’s estimating.” Unbiasedness is a modest but important property. It doesn’t say your estimate from this particular sample will be correct. It says that if you drew many samples and averaged the estimates, you’d converge on the truth.

The most important unbiased estimator. If you have a sample S_X = {x₁, x₂, …, x_N} from a distribution with unknown mean μ, the sample mean is:

$$\hat{\mu} = \frac{1}{N} \sum_{i=1}^{N} x_i$$

This is literally just the arithmetic average of your sample. It’s an unbiased estimator of the true mean. If you compute it on a sample of 1,000 customers and then on a sample of 1,000,000, both are unbiased. The difference is that the estimate from the larger sample is much more likely to be close to the true value in any single draw, because its variance is lower. “Unbiased in expectation” doesn’t mean “accurate in any single sample.”

Concrete walkthrough: estimating a default rate. Suppose you sample 10,000 SME loans from the bank’s historical portfolio and find that 372 of them defaulted within 24 months. Your sample mean is 372/10, 000 = 0.0372, or 3.72%. This is your unbiased estimate of the true portfolio-wide 24-month default rate. But you want to know how reliable it is. A quick confidence interval calculation (using the fact that a binomial proportion has approximate standard error $\sqrt{p(1-p)/n}$) gives a standard error of about 0.19%, so a 95% confidence interval is roughly [3.34%, 4.10%]. You can report “the default rate is 3.72%, with 95% confidence it’s between 3.34% and 4.10%.” You cannot report “the default rate is 3.72%” without qualification, and you cannot make decisions that assume the estimate is exact.

Where this goes wrong. Unbiased estimators can still be misleading when the sample is not representative. If you sample 10,000 loans but your sampling procedure over-represents one sector, your sample mean of default rate is an unbiased estimate of the sector-weighted default rate, not the true portfolio-wide rate. Sampling design matters as much as estimator choice. In credit risk specifically, historical data is often biased by the fact that declined applications weren’t funded and so you have no outcome for them, a phenomenon called reject inference. Handling it properly is a whole subfield, and model validation teams at UK banks spend significant time scrutinising how their modelling teams have approached it.

If asked: “What’s the difference between saying an estimator is unbiased and saying it’s accurate?”

How do you update a belief in the light of new evidence?

The rule says:

$$\text{Pr}(X = x \mid Y = y) = \frac{\text{Pr}(Y = y \mid X = x) \, \text{Pr}(X = x)}{\text{Pr}(Y = y)}$$

Read this carefully. The bar | means “given that.” So Pr(X = x ∣ Y = y) is “the probability that X takes value x, given that we have observed that Y takes value y.” This is called a conditional probability. The left-hand side is what we want to know. The right-hand side is how we compute it from other things we can more easily get at.

The rule has a poetic structure. The numerator multiplies two things: the probability of observing the evidence if our hypothesis is true, and the prior probability of the hypothesis. The denominator is just the total probability of the evidence regardless of hypothesis, which serves as a normalising constant. In words:

$$\text{posterior} = \frac{\text{likelihood} \times \text{prior}}{\text{evidence}}$$

The posterior is the updated belief about the hypothesis after seeing the evidence. The prior is what you believed before. The likelihood is how compatible the evidence is with the hypothesis. The evidence is the total probability of the observation under all possible hypotheses, and it’s often the hardest part to compute.

A banking example that sticks. Suppose 0.5% of all wire transfers over £100k In the synthetic Merehaven case are actually fraudulent. So the prior is Pr(fraud) = 0.005. Suppose a particular anomaly detection model flags a transaction as suspicious. The model has the following performance:

• If a transaction is genuinely fraudulent, the model flags it 95% of the time. So Pr(flag ∣ fraud) = 0.95. This is the likelihood.
• If a transaction is genuine, the model still flags it 2% of the time (false positives). So Pr(flag ∣ genuine) = 0.02.

A new transaction comes in, and the model flags it. What’s the probability it’s actually fraudulent?

Let’s compute carefully. We want Pr(fraud ∣ flag). Bayes:

$$\text{Pr}(\text{fraud} \mid \text{flag}) = \frac{\text{Pr}(\text{flag} \mid \text{fraud}) \cdot \text{Pr}(\text{fraud})}{\text{Pr}(\text{flag})}$$

The denominator Pr(flag) is the total probability of getting a flag, combining both sources:

Pr(flag) = Pr(flag ∣ fraud) ⋅ Pr(fraud) + Pr(flag ∣ genuine) ⋅ Pr(genuine)

 = (0.95)(0.005) + (0.02)(0.995) = 0.00475 + 0.01990 = 0.02465

Now plug in:

$$\text{Pr}(\text{fraud} \mid \text{flag}) = \frac{(0.95)(0.005)}{0.02465} = \frac{0.00475}{0.02465} \approx 0.193$$

So if the model flags a transaction, there’s only about a 19% chance it’s actually fraudulent. That’s counter-intuitive. The model is “95% accurate at detecting fraud,” isn’t it? Yes, but fraud is rare, and even a small 2% false positive rate applied to the vast majority of genuine transactions produces many more false alarms than true alarms. This is why every serious fraud and AML operation needs human review queues sized to handle the false positive load, and why model operators obsess over the trade-off between recall and precision.

This is also why any headline like “AI detects cancer with 95% accuracy” should be read with suspicion until you know the base rate. The famous mathematical trap, called base rate neglect, gets people every time.

Read this as a pipeline for updating belief. You start with a prior, you observe evidence, you weight by likelihood, and you end with a posterior. The posterior is the new quantity you should base decisions on. The diagram is the calculation we just did, arrow by arrow. Memorise the shape.

If asked: “A diagnostic model is 99% accurate. It says someone is a fraudster. How worried should the bank be?”

How do you find the best parameters for a model of the world?

We’ve seen that a model is a function with parameters, and training is finding good values for those parameters. Bayes’ rule gives us a principled way to do the finding.

Suppose you assume that the data you’ve observed was generated by some distribution f_θ whose shape you know (say, a Gaussian) but whose parameters θ you don’t know. You want to estimate θ from the data. One approach is maximum a posteriori (MAP) estimation, which says: pick the value of θ that has the highest posterior probability given the data.

Applying Bayes, the posterior over parameters is:

$$\text{Pr}(\theta = \hat{\theta} \mid X = x) = \frac{\text{Pr}(X = x \mid \theta = \hat{\theta}) \cdot \text{Pr}(\theta = \hat{\theta})}{\text{Pr}(X = x)}$$

The denominator doesn’t depend on θ, so for finding the maximum you can ignore it. This leaves:

θ̂_MAP = arg max_θ Pr(X = x ∣ θ) ⋅ Pr(θ)

For a dataset of multiple independent observations, the likelihood of the whole dataset is the product of individual likelihoods:

$$\hat{\theta}_{\text{MAP}} = \arg\max_\theta \, \prod_{i=1}^{N} \text{Pr}(x_i \mid \theta) \cdot \text{Pr}(\theta)$$

This is beautiful but numerically nasty. Products of many small numbers underflow. Remember the log trick. Take logs of everything:

$$\hat{\theta}_{\text{MAP}} = \arg\max_\theta \, \sum_{i=1}^{N} \log \text{Pr}(x_i \mid \theta) + \log \text{Pr}(\theta)$$

Now we’re maximising a sum, not a product, and we’re safe from underflow. If you drop the prior term log Pr(θ), you get maximum likelihood estimation (MLE), which is the same thing with a “flat” prior that treats all parameter values as equally plausible before seeing data. MLE is the workhorse of classical statistical fitting. MAP is its Bayesian cousin, regularised by a prior.

Concrete walkthrough: fitting a Gaussian. Suppose we want to model the distribution of debt service coverage ratios across Merehaven Bank SME customers and we assume it’s approximately Gaussian. The Gaussian density is:

$$f_\theta(x) = \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)$$

with parameters θ = (μ, σ). Given a sample of N observed DSCRs, the maximum likelihood estimate of μ is the sample mean (which we met above), and the MLE of σ² is the sample variance. You get clean closed-form answers because the Gaussian has nice mathematical structure. For most other distributions, you don’t get closed-form answers, and you have to optimise the log-likelihood numerically. Numerically means gradient descent, which we spent the middle of this section building up.

So the full story of parameter estimation is: write down the log-likelihood of your data under your model, take its gradient with respect to the parameters, and step downhill (or uphill, depending on whether you’re minimising the negative log-likelihood or maximising the log-likelihood; same thing). This is exactly how logistic regression, probabilistic graphical models, and even many deep learning models with probabilistic output layers are trained. You’ve now seen the whole recipe.

Where this goes wrong. MLE overfits. With enough parameters relative to data, you can drive the likelihood arbitrarily high by memorising the training set. MAP with a sensible prior acts as a regulariser, pulling the estimate toward plausible values and preventing the worst of the overfitting. When you see terms like L2 regularisation in deep learning, they can often be re-derived as MAP estimation with a Gaussian prior over weights. Modern practice blurs the line between pure optimisation and Bayesian estimation, and good engineers keep both perspectives in mind.

If asked: “What’s the difference between maximum likelihood estimation and maximum a posteriori estimation?”

Parameters versus hyperparameters, and the other vocabulary distinctions

The last block of this section is a cluster of terminological distinctions that every practitioner uses reflexively and that beginners constantly mix up. You should be able to recite these in your sleep.

Parameters versus hyperparameters. A parameter is a value that the learning algorithm modifies during training, on the basis of the data. The weights w and bias b of a linear model are parameters. The weights of every neuron in a neural network are parameters. Parameters are what “learning” means: they change during training.

A hyperparameter is a value that the data scientist sets before training begins and that does not change during training. The SVM penalty C is a hyperparameter. The learning rate α in gradient descent is a hyperparameter. The number of layers in a neural network, the number of neighbours in k-NN, the depth of a decision tree, the regularisation coefficient in ridge regression, all hyperparameters.

Read this as the two kinds of knobs on a model. The hyperparameter knobs are set by you. The parameter knobs are set by the training algorithm. Both affect model behaviour. Neither can be skipped.

Classification versus regression. We met these briefly in Chapter 1. Classification assigns a label from a finite set of classes: spam or not spam, default or no default, high risk or low risk. Regression predicts a real-valued number: loss given default as a percentage, house valuation in pounds, expected time to next purchase in days. Same training data shape, different output shape, different loss functions, different evaluation metrics.

Model-based versus instance-based learning. This is a subtle but important distinction. A model-based algorithm uses the training data to compute a model with a relatively small set of parameters, and then throws the training data away. The model is self-contained and can predict on new inputs by a quick computation using just its parameters. Linear regression, logistic regression, SVM, and neural networks are all model-based.

An instance-based algorithm doesn’t compute a compact model. Instead it stores the entire training set and, at prediction time, compares the new input against the stored examples to make its decision. k-Nearest Neighbours (k-NN) is the canonical example. To predict the class of a new input, you find the k closest training examples in feature space and return the majority vote of their labels. There’s no “model” to speak of. The training data is the model.

Shallow versus deep learning. A shallow learning algorithm learns parameters directly from input features. Linear regression, SVM, logistic regression, and classical tree ensembles are shallow: the parameters they learn relate directly to the raw features you fed them. Deep learning, by contrast, stacks multiple learned layers, where each layer’s parameters operate not on raw features but on the outputs of the previous layer. In a deep neural network with six layers, only the first layer’s weights act directly on the input features; every subsequent layer’s weights act on transformed representations that the earlier layers constructed.

The distinction matters because deep learning’s extra layers enable the model to learn useful representations automatically, which is powerful and dangerous. Powerful because you don’t have to hand-engineer features as painstakingly. Dangerous because the learned representations are hard to interpret, which matters enormously in regulated banking contexts where you must be able to explain every decision. We’ll come back to deep learning in Chapter 6, and to interpretability of deep models in later chapters.

If asked: “Explain the difference between model-based and instance-based learning with a banking example.”

Part II: Fit models before adding machinery

Simple models establish the baseline and expose the objective. Optimisation then becomes an observable route from initial parameters to a tested operating point.

Choose transparent baselines before complexity

How do you draw the best straight line through a cloud of points?

In 1805, a French mathematician called Adrien-Marie Legendre, working on a problem in astronomy, published a technique for fitting a line to a collection of noisy observations. He called it the “method of least squares,” and the paper was a modest six-page appendix to a larger astronomy monograph. Legendre argued, with typical French clarity, that if you wanted to choose the line that best fit the data, you should choose the one that minimised the sum of squared differences between the line and the data points. He did not bother to prove it was optimal. He simply said it was “of all the principles which can be proposed, the most general, the most exact, and the easiest to apply.”

Three years later, Carl Friedrich Gauss published a more rigorous derivation and claimed, annoyingly, that he had been using the method since 1795 but had not bothered to write it down. Whether Gauss was first or second, the least-squares method has been the workhorse of every field that fits a line through points for more than two centuries, and it is exactly what modern linear regression computes.

Here’s what we’re trying to do. Imagine you are on the retail banking strategy team and you want a quick model that predicts a customer’s annual fee income from a handful of their observable attributes: years tenure, number of products held, total deposits, and average monthly card spend. You have 50,000 historical customers with all those attributes plus their actual annual fee income. You want a function that takes the four features and returns a fee estimate for a new customer.

The linear regression assumption is that fee income is, approximately, a weighted sum of the features plus a constant:

f_w, b(x) = w ⋅ x + b = w⁽¹⁾x⁽¹⁾ + w⁽²⁾x⁽²⁾ + w⁽³⁾x⁽³⁾ + w⁽⁴⁾x⁽⁴⁾ + b

The vector w holds four weights, one per feature. The scalar b is the bias, a baseline fee income when all features are at zero. Our job is to find the specific w and b that best match the historical data.

The “best” needs a precise definition. In linear regression, “best” means: the values of w and b that make the model’s predictions as close to the true fee incomes as possible, on average, where “close” is measured by the squared difference. For one historical customer i the squared error is (f_w, b(x_i) − y_i)². For the whole training set the average squared error is:

$$L(\mathbf{w}, b) = \frac{1}{N} \sum_{i=1}^{N} (f_{\mathbf{w}, b}(\mathbf{x}_i) - y_i)^2$$

This is the mean squared error or MSE, and it is the most famous loss function in classical statistics. L is a function of the parameters. Plug in any specific w and b, get a number. The number tells you how bad those parameters are. Find the w^* and b^* that make L as small as possible and you have your linear regression.

Contrast with the SVM from Chapter 1. In SVM, the hyperplane sits as far as possible from all training points, because it’s a decision boundary separating two classes and the distance is your safety margin. In linear regression, the hyperplane sits as close as possible to all training points, because it’s a prediction surface that has to pass through the middle of the data cloud, not skirt around it. Same algebraic object (a hyperplane), opposite geometric role. The picture below shows the difference.

Read this as “same shape, opposite job.” A hyperplane is just a flat surface, and what makes it SVM or linear regression is what you tell it to do.

Why squared error specifically, and not absolute error or cubic error? This is the question every thoughtful beginner asks, and it has three answers.

First, mathematical convenience. The squared function is smooth and differentiable everywhere, while absolute value has a kink at zero that makes the derivative undefined. Smooth functions are easier to optimise, and in 1805 Legendre needed an optimisation technique that could be done by hand. Squared error gave him one.

Second, closed-form solutions. If you write out the gradient of the MSE with respect to w and b, set it equal to zero, and solve, you get a direct algebraic formula for w^* and b^*. No iterative optimisation, no learning rate, no stopping criterion. Just a matrix inversion. The closed-form solution is:

w^* = (X^⊤X)⁻¹X^⊤y

assuming we’ve folded b into w by adding a column of ones to X. Don’t panic about the notation if it’s new. What matters is that this is a formula you can evaluate in one pass over your data. No iterative training. No gradient descent. Plug in the data, get the parameters. For decades this was the definition of a well-behaved model: it had a closed form.

Third, the statistical interpretation. If you assume the true relationship between features and target is linear plus Gaussian noise, then the least-squares estimate is the maximum likelihood estimate of the parameters. We derived maximum likelihood in Chapter 2, and now we see it popping up again as a justification for the squared loss. The same mathematical principle keeps returning, wearing different hats.

A concrete walkthrough with tiny numbers. Forget the four-feature fee model for a moment. Consider the simplest possible linear regression: one feature, three training points. Our feature is “years tenure” and our target is “annual fee income in pounds.” Training data:

Assume a linear model ŷ = wx + b with scalar parameters. The closed-form least-squares solution for a single feature is:

$$w^* = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2}, \quad b^* = \bar{y} - w^* \bar{x}$$

Compute the means: x̄ = (2 + 5 + 8)/3 = 5, ȳ = (120 + 210 + 330)/3 = 220. Compute the deviations: (x_i − x̄) = (−3, 0, 3), (y_i − ȳ) = (−100, −10, 110). Products: (−3)(−100) + (0)(−10) + (3)(110) = 300 + 0 + 330 = 630. Sum of squared deviations: 9 + 0 + 9 = 18. So w^* = 630/18 = 35, and b^* = 220 − 35 ⋅ 5 = 45.

The fitted model is ŷ = 35x + 45. For a new customer with 4 years tenure, predicted fee is 35 ⋅ 4 + 45 = 185. That’s it. One feature, three points, pencil and paper arithmetic, real working model. Scale this up to 240 features and 180,000 points and the same principle holds: matrix algebra does the work. No gradient descent required.

Formal statement of the training problem. Given N labelled examples {(x_i, y_i)}_i = 1^N with y_i ∈ ℝ, find parameters (w^*, b^*) that minimise $L(\mathbf{w}, b) = \frac{1}{N} \sum_{i=1}^{N} (\mathbf{w}^\top \mathbf{x}_i + b - y_i)^2$. When this minimisation is done, the prediction for any new input x is ŷ = w^*⊤x + b^*. Reading the loss left to right: for each training example, compute the model’s prediction, subtract the true value, square the difference, add them all up, divide by the count. Any choice of parameters that makes this number smaller is a better choice.

Where this goes wrong. Linear regression assumes two things that are frequently violated. First, that the true relationship is actually linear in the features, which means that doubling a feature exactly doubles its contribution. Fee income does not grow linearly with tenure forever. The tenth year of a relationship does not add twice as much fee income as the fifth year; there are diminishing returns. If you blindly fit a linear model to a curved relationship, you end up with a line that is systematically wrong everywhere. The fix is feature engineering: add a tenure² term, or use a log transform, or switch to a model that handles non-linearity natively.

Second, linear regression assumes that all training points are equally important and equally reliable. A single extreme outlier can drag the line far away from the rest of the data, because the squared error for that one point dominates the sum. If your training set contains a wealth management customer whose annual fee is £30,000 while everyone else is under £1,000, the linear regression will skew heavily toward accommodating that one customer, and the predictions on ordinary customers will suffer. The fixes are well-tested regression (which uses a less punishing loss function, like Huber loss or absolute error) or outlier removal before fitting.

There is also a third failure mode you should know about by name: multicollinearity. If two of your features are nearly linear combinations of each other (for example, “total deposits” and “total current account balance” in a dataset where most customers hold only current accounts), the matrix X^⊤X becomes nearly singular and the inversion becomes numerically unstable. The closed-form solution gives you wild weights that cancel each other out and react catastrophically to small changes in the training data. The fixes are feature selection (drop one of the correlated features), regularisation (ridge regression, which we’ll meet in Chapter 5), or using pseudo-inverse variants.

If asked: “Why do the worked design uses squared error instead of absolute error in linear regression?”

How do you turn a straight line into a probability?

Now we face a slightly different problem. Linear regression predicts a real number. What if you want to predict a binary outcome, like whether a customer will default on a credit card payment in the next ninety days? You could just fit a linear regression with y_i ∈ {0, 1}, and some people do, but it has an obvious defect: the model can predict values less than zero or greater than one, which are meaningless as probabilities. The fix, invented in the 1940s by the Belgian mathematician Pierre-François Verhulst and refined into its modern form by Sir David Cox in 1958, is to take the linear score and squash it into the range (0, 1) using the logistic function:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

The σ is a lowercase Greek sigma, and the function is called the sigmoid because its graph is S-shaped. Some key values: σ(0) = 0.5, σ(2) ≈ 0.88, σ(−2) ≈ 0.12, σ(5) ≈ 0.993, σ(−5) ≈ 0.007. As z goes to positive infinity, σ(z) approaches 1. As z goes to negative infinity, σ(z) approaches 0. For any real input z, the output is a valid probability.

The logistic regression model is then:

$$f_{\mathbf{w}, b}(\mathbf{x}) = \sigma(\mathbf{w}^\top \mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^\top \mathbf{x} + b)}}$$

You can see the familiar w^⊤x + b term from linear regression, wrapped inside the sigmoid. That’s all logistic regression is: a linear score fed through a squashing function. The output is interpreted as Pr(y = 1 ∣ x): the probability that the true label is 1 given the observed features. For a credit card default model, it’s the probability of default. For a fraud model, it’s the probability of fraud. For a cross-sell model, it’s the probability of take-up.

Why not just use linear regression with binary labels? Three reasons. First, the predictions can fall outside [0, 1], which you then have to clip, and clipped predictions are not differentiable at the clipping boundary, so you can’t train them with gradient methods. Second, the variance of the errors depends on the predicted probability, violating one of the assumptions of linear regression (homoscedasticity). Third, the squared error loss is the wrong loss for probabilities: it treats a prediction of 0.99 when the truth is 1 as about the same as a prediction of 0.51 when the truth is 1, whereas any sensible probability scoring rule would reward the confident correct prediction far more. Logistic regression fixes all three issues by choice of model (sigmoid) and choice of loss (log-likelihood, below).

How do we train it? Not by squared error. Instead, the worked design uses maximum likelihood estimation, which you met in Chapter 2. Here’s how it unfolds.

For a single training example (x_i, y_i), the model says the probability of observing the label y_i is f_w, b(x_i) if y_i = 1, and 1 − f_w, b(x_i) if y_i = 0. These two cases can be combined into a single compact expression:

Pr(y_i ∣ x_i, w, b) = f_w, b(x_i)^y_i ⋅ (1 − f_w, b(x_i))^{1 − y_i}

Check the math. When y_i = 1, the exponent on the first term is 1 and the exponent on the second is 0, so the whole thing simplifies to f_w, b(x_i). When y_i = 0, the exponent on the first is 0 and the second is 1, so it simplifies to 1 − f_w, b(x_i). One formula, two cases, no branching. The trick of using y_i and (1 − y_i) as exponents to pick between two expressions is a common idiom and you’ll see it constantly.

For the whole training set, assuming examples are independent, the likelihood is the product of individual likelihoods:

$$L(\mathbf{w}, b) = \prod_{i=1}^{N} f_{\mathbf{w}, b}(\mathbf{x}_i)^{y_i} \cdot (1 - f_{\mathbf{w}, b}(\mathbf{x}_i))^{1 - y_i}$$

We want to find w and b that maximise this. But we know from Chapter 2 that products of many small numbers are numerically nasty. Take the logarithm to turn products into sums, and we get the log-likelihood:

$$\log L(\mathbf{w}, b) = \sum_{i=1}^{N} \left[ y_i \log f_{\mathbf{w}, b}(\mathbf{x}_i) + (1 - y_i) \log(1 - f_{\mathbf{w}, b}(\mathbf{x}_i)) \right]$$

Maximising the log-likelihood is equivalent to maximising the likelihood, because logarithm is a strictly increasing function, so the location of the maximum is the same. Flip the sign and you get the binary cross-entropy loss, which is what you’ll see in almost every code example:

$$\text{BCE}(\mathbf{w}, b) = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \log f_{\mathbf{w}, b}(\mathbf{x}_i) + (1 - y_i) \log(1 - f_{\mathbf{w}, b}(\mathbf{x}_i)) \right]$$

The minus sign in front and the division by N are conventions so that the number is positive, normalised per example, and gets smaller as the model improves. When you see “cross-entropy loss” in a deep learning framework, this is exactly what it means.

importantly, logistic regression has no closed-form solution. Unlike linear regression, there is no matrix formula that gives you w^* and b^* directly. The log-likelihood is a concave function of the parameters (its negative is convex), so it has a unique global maximum, but finding that maximum requires iterative optimisation: gradient descent, Newton’s method, or quasi-Newton methods like L-BFGS. This is the first place in our journey where we meet an algorithm whose training is not an algebraic formula but an iterative procedure. Chapter 4 covers the mechanics of gradient descent in detail; for now just know that training a logistic regression in practice means “start with random weights, compute the gradient of the BCE loss, step downhill, repeat until convergence.”

Read this as the full loop. At inference time, you run the top path: linear score, sigmoid, probability, threshold. At training time, you also run the bottom path: compute the loss, take its gradient, update the parameters, iterate.

Concrete walkthrough: credit card default In the synthetic Merehaven case. Suppose you are building a simple credit card default predictor with three features: utilisation ratio (current balance divided by credit limit), months since last late payment (high means clean history), and number of credit products held. Say after training you get weights w = (3.5, −0.2, 0.4) and bias b = −2.1. These weights have interpretable meanings: higher utilisation pushes toward default (positive weight on utilisation), more months since last late pushes away from default (negative weight), and more credit products modestly pushes toward default (mildly positive weight).

For a customer with utilisation 0.8, last late payment 24 months ago, holding 3 credit products:

z = 3.5 ⋅ 0.8 + (−0.2) ⋅ 24 + 0.4 ⋅ 3 + (−2.1) = 2.8 − 4.8 + 1.2 − 2.1 = −2.9

$\sigma(-2.9) = \frac{1}{1 + e^{2.9}} = \frac{1}{1 + 18.17} \approx 0.052$

The model says there’s a 5.2% chance this customer defaults in the window of interest. Low risk. If the utilisation were instead 0.95 and the last late payment were 3 months ago:

z = 3.5 ⋅ 0.95 + (−0.2) ⋅ 3 + 0.4 ⋅ 3 + (−2.1) = 3.325 − 0.6 + 1.2 − 2.1 = 1.825

$\sigma(1.825) = \frac{1}{1 + e^{-1.825}} = \frac{1}{1 + 0.161} \approx 0.861$

86.1% chance of default. High risk. The same model, the same weights, giving materially different probabilities for different customers. This is why logistic regression is the absolute workhorse of retail credit risk in banking: every scorecard at every high-street bank, from Merehaven Bank to Barclays to HSBC, is a logistic regression at its heart, often with additional discretisation or binning on top for interpretability. The weights are reviewed by credit experts and challenge teams; the scores are mapped to rating grades; and the whole apparatus has been stable for thirty-plus years because it works.

Where this goes wrong. Three common failure modes.

First, probability calibration drift. A logistic regression trained on one population may give probabilities that are systematically biased on a different population. If your training data had 5% defaults and your production population has 3% defaults, the raw probabilities will be too high. Calibration techniques like Platt scaling or isotonic regression can correct this, but the drift has to be monitored. In credit risk this matters because probability of default is used directly for regulatory capital calculation and any miscalibration flows through to capital numbers.

Second, nonlinear interactions. Logistic regression is linear in its features. If the true relationship between features and default involves interactions (e.g., high utilisation is only a problem when tenure is short), a plain logistic regression will miss them unless you explicitly engineer interaction terms. This is why logistic regression in regulated credit often sits alongside or is replaced by gradient boosted trees, which find interactions automatically.

Third, separation. If your features are so predictive that they perfectly separate defaulters from non-defaulters in the training set, the maximum likelihood estimate of w is infinite. This is mathematically true and computationally disastrous. The training procedure fails to converge, and the weights you get are extreme and meaningless. The fix is regularisation, which we come back to in Chapter 5, and which in essence adds a penalty term to the loss that keeps weights from growing too large.

If asked: “Why is logistic regression called ‘regression’ if it’s actually a classification algorithm?”

How do you build a model that a credit officer can read?

Decision trees are the most interpretable machine learning algorithm in common use, and interpretability is the reason they refuse to die even in the age of massive neural networks. A decision tree is just a sequence of yes/no questions about the features, arranged in a branching structure, with a prediction at each leaf. You can print one on a sheet of A4 paper, hand it to a credit officer, and they can trace through it exactly as they would trace through a policy document.

Here’s what one looks like for a simplified SME default model:

Read this top to bottom. Every new loan enters at the top. The first question asks whether DSCR is below 1.1. If yes, we check whether LTV is also above 0.75. If both answers put us in the riskiest category, we predict a 42% probability of default. If not, we land in one of four leaves, each with a specific predicted probability. The whole model is summarised by this picture and the four numbers at its leaves. Any credit officer can read it. Any compliance reviewer can challenge it. Any model risk team can reproduce its output by hand.

This interpretability is a profound advantage. When a customer disputes a credit decision under applicable data-protection and consent rules article 22, a logistic regression answer would be something like “your score was -1.8 because the model weights your utilisation at 3.5 and your utilisation was high.” A decision tree answer is “your application had DSCR 0.95 and LTV 0.82, which placed it in the branch where we predict default with probability 0.42, above the approval threshold.” The second answer is comprehensible to a human whose job title is not “statistician.”

How does the algorithm figure out which questions to ask, and in what order? This is where the cleverness lies. The basic recipe, known as ID3 (Iterative Dichotomiser 3, invented by Ross Quinlan in 1986 at the University of Sydney), is deceptively simple and deeply elegant. You start with all your training data in one bucket. You try every possible question you could ask at the root, score each one by how much “mess” it eliminates, and pick the best. You then recursively repeat this procedure on each of the two resulting buckets. You stop when the bucket is pure enough, or when it’s too small to split further, or when you hit a pre-set depth limit.

The bit we need to define is “mess.” In ID3, mess is measured by entropy, a concept borrowed from information theory. For a set S of labelled examples, define p as the proportion of examples in S that are class 1 (say, defaulters). The entropy of S is:

H(S) = −plog p − (1 − p)log (1 − p)

What does this number mean? When p = 0 or p = 1, everything in S is one class, no mess, and H(S) = 0 (the conventions are that 0log 0 = 0). When p = 0.5, half the set is each class, maximum mess, and H(S) = 1 if you use log₂ or ln 2 if you use natural log. Between these extremes, the entropy rises and falls smoothly. It’s a measure of “how uncertain you’d be if you had to guess the label of a random example from S.” When uncertainty is highest (50/50), entropy is highest. When uncertainty is zero (everyone is one class), entropy is zero.

When you split S on a feature and threshold, you get two subsets S₋ and S₊. The combined entropy of the split, weighted by how many examples ended up on each side, is:

$$H(S_-, S_+) = \frac{|S_-|}{|S|} H(S_-) + \frac{|S_+|}{|S|} H(S_+)$$

Read this as “the entropy of the left bucket weighted by its size, plus the entropy of the right bucket weighted by its size, divided by the total size.” A good split is one where both S₋ and S₊ are more homogeneous than S was. In that case the weighted combined entropy is lower than H(S), and the difference is called the information gain. The algorithm searches over every feature j and every threshold t, computes the information gain, and picks the pair with the highest gain.

Concrete walkthrough with numbers. Suppose you have 12 historical loans, 4 defaulters and 8 non-defaulters. The entropy of the whole set is $H(S) = -\frac{4}{12} \log_2 \frac{4}{12} - \frac{8}{12} \log_2 \frac{8}{12} \approx 0.918$.

You try splitting on “DSCR < 1.1”. Suppose this gives you $S_- = $ 7 loans (5 defaulters, 2 non-defaulters) and $S_+ = $ 5 loans (no defaulters, 5 non-defaulters). Wait, that doesn’t quite work with our totals; let me recount. Try: $S_- = $ 7 loans (4 defaulters, 3 non-defaulters) and $S_+ = $ 5 loans (0 defaulters, 5 non-defaulters). Total checks: 4 + 0 = 4 defaulters, 3 + 5 = 8 non-defaulters. OK.

$H(S_-) = -\frac{4}{7} \log_2 \frac{4}{7} - \frac{3}{7} \log_2 \frac{3}{7} \approx 0.985$. That’s high. The left bucket is still messy.

$H(S_+) = -\frac{0}{5} \log_2 \frac{0}{5} - \frac{5}{5} \log_2 \frac{5}{5} = 0$. The right bucket is pure. All five are non-defaulters. Clean.

Weighted combined entropy: $\frac{7}{12} \cdot 0.985 + \frac{5}{12} \cdot 0 \approx 0.575$.

Information gain: 0.918 − 0.575 ≈ 0.343.

That’s a meaningful reduction in entropy. The split on DSCR has cleanly carved off a chunk of non-defaulters. The algorithm tries every other feature and threshold, computes the information gain for each, and picks the highest. Then it recurses into S₋ and continues splitting. S₊ is pure, so it becomes a leaf with prediction “non-default” and no further splitting is needed.

Formal statement of the algorithm. ID3 builds a decision tree by recursively selecting the feature and threshold that maximise information gain on the current subset of training data, stopping when the subset is pure, too small, or when the gain is below a threshold. At a leaf, the prediction is the majority class (for classification) or the mean target (for regression). The resulting model is a tree structure where each internal node asks a question about one feature and each leaf emits a prediction.

1. All examples in the leaf are the same class, so the leaf is perfectly pure.
2. No remaining feature can split the examples (the features don’t distinguish them).
3. The best available split reduces entropy by less than a pre-set threshold ϵ.
4. The tree has reached a maximum depth d set in advance.

Both ϵ and d are hyperparameters. They have to be chosen before training. Too lenient (small ϵ, large d) and the tree grows enormous, memorising noise in the training data and overfitting spectacularly. Too strict and the tree is too shallow to capture real patterns. The right values come from validation-set tuning, which we cover in Chapter 5.

The extension to C4.5. ID3 is the academic version. In practice, the most widely used single-tree algorithm is C4.5, also from Quinlan (1993), which adds three important refinements:

• Handles continuous and discrete features. ID3 really wants discrete features; C4.5 handles continuous features by finding the best threshold as part of the split selection.
• Handles missing values. Real banking data has missing values everywhere (a customer’s previous address may not be recorded; an SME’s last year of management accounts may be overdue). C4.5 has explicit machinery for propagating examples with missing values down both branches of a split.
• Pruning. After the tree is fully grown, C4.5 goes back through and prunes branches that don’t earn their keep, replacing them with leaves. Pruning is a regularisation technique that reduces overfitting without needing to stop growth early. It’s sometimes more effective than early stopping.

Where this goes wrong. Decision trees have several famous failure modes.

First, they are greedy. ID3 and C4.5 make each split decision locally, without looking ahead to future splits. The best first split is not always the first split of the globally best tree. As a result, decision trees often find suboptimal solutions. The cure is ensembles: random forests and gradient boosted trees, which combine many trees to cancel out the errors of individual greedy choices. We’ll meet these in Chapter 7.

Second, they are unstable. A small change in the training data can cause the entire tree structure to change, because a change in which feature is chosen at the root cascades through everything below. This is why individual decision trees are rarely used in production credit risk today; ensembles smooth out the instability.

Third, they struggle with smooth relationships. If the true relationship between a feature and the target is genuinely smooth (like a linear trend), a decision tree has to approximate it with a staircase of splits, which is clunky and needs many splits to look smooth. Linear and logistic regression handle smooth relationships natively and elegantly.

And fourth, they overfit if allowed to grow without limit. A deep tree can memorise individual training points with unique paths, achieving zero training error while being useless on new data. Pruning and depth limits are the standard defences.

If asked: “Why do decision trees tend to overfit, and what do we do about it?”

How does SVM deal with data that isn’t a clean straight line?

In Chapter 1 we met SVM as the classifier that draws the widest possible corridor between two classes. That was the happy case: the classes are linearly separable and you just find the widest separating hyperplane. Real data is not that tidy. Two failure modes are nearly universal.

Failure mode one: noise. Even when the classes are mostly separable, there are often a few points in the wrong place. A healthy-looking SME that defaulted because of an unexpected fraud at its largest customer. A fraudulent transaction that looks utterly ordinary because the criminal was patient. A single mislabelled record in the training data. These noise points mean no straight line can separate the classes perfectly, and the hard-margin SVM from Chapter 1 has no solution: its constraints cannot all be satisfied simultaneously.

Failure mode two: inherent non-linearity. Even with clean labels, sometimes the geometry of the problem is genuinely not linear. Picture a situation where the “default” customers are clustered in a ring around the “non-default” customers in feature space. No straight line can carve out the ring. You need a curved boundary, and a hyperplane is by definition flat.

SVM has elegant answers to both. Let’s take them one at a time.

Noise, and the soft-margin SVM. The original (hard-margin) SVM requires every training point to sit on the correct side of the margin. For noisy data, this constraint is too strict. The soft-margin variant, invented by Corinna Cortes and Vladimir Vapnik in 1995, relaxes the constraint by allowing training points to violate the margin, at a cost.

The trick is the hinge loss:

L_hinge(y_i, w, b) = max (0, 1 − y_i(w^⊤x_i − b))

Read this carefully. If the point x_i is on the correct side of the margin (that is, y_i(w^⊤x_i − b) ≥ 1), the expression inside the max is negative or zero, so the max gives zero: no penalty. If the point is on the wrong side of the margin (either inside the corridor or entirely misclassified), the expression inside the max is positive, and the loss grows linearly with how far wrong the point is.

The combined objective for the soft-margin SVM is:

$$\min_{\mathbf{w}, b} \left[ C \cdot \|\mathbf{w}\|^2 + \frac{1}{N} \sum_{i=1}^{N} \max(0, \, 1 - y_i(\mathbf{w}^\top \mathbf{x}_i - b)) \right]$$

Two terms. The first, C∥w∥², is the same old margin-maximisation term: keep w small so the corridor is wide. The second is the average hinge loss: pay a penalty for any point that violates the margin. The hyperparameter C controls the trade-off: large C means “heavily penalise margin violations, which gives a narrower margin that fits the training data tightly”; small C means “be tolerant of violations in exchange for a wider margin.”

Setting C is one of the most important choices when using SVM. Too large and the model overfits by twisting the margin to accommodate noisy points. Too small and the model underfits by ignoring real structure. The right value comes from validation-set tuning. In banking use cases where training sets are modest and noise is real, a moderate C is usually appropriate.

Non-linearity, and the kernel trick. This is one of the most beautiful ideas in classical machine learning, and it deserves a proper story.

In the late 1990s, Vapnik and peers at AT&T Bell Labs were trying to extend SVM to problems where the classes were not linearly separable in the original feature space. The obvious idea was to transform the features into a higher-dimensional space where they would be separable. Take a two-dimensional problem where the classes form concentric rings, for example, and map each point (q, p) to a new point $(q^2, \sqrt{2} q p, p^2)$ in three dimensions. In this new space, the ring-shaped boundary becomes a plane, and SVM can find it directly.

The problem is that this explicit mapping can be expensive. If you want to project into a 100-dimensional space, you compute 100 new features per point. If you want a 1,000-dimensional space, 1,000 features. For some useful mappings, the target space is formally infinite-dimensional and you can’t compute the features explicitly at all.

Here’s the insight. The SVM optimisation, when written in its “dual form” using Lagrange multipliers, depends on the training data only through the dot products x_i^⊤x_k between pairs of examples. Not the individual features. Just the dot products. If you had a way to compute the dot product in the transformed space without explicitly computing the transformed features, you could run the SVM optimisation as if you were working in the high-dimensional space, while never actually going there.

That’s the kernel trick. A kernel function k(x_i, x_k) is a function that takes two original feature vectors and returns the value that the dot product of their transformed versions would have, without actually computing the transformations. Some famous kernels:

Linear kernel: k(x, x^′) = x^⊤x^′. The identity: no transformation. SVM with a linear kernel is just the plain linear SVM.

Polynomial kernel: k(x, x^′) = (x^⊤x^′ + c)^d for some degree d. Corresponds to projecting into a space where the features include all products of up to d original features. A degree-2 polynomial kernel implicitly captures all pairwise interactions.

Radial basis function (RBF) kernel: $k(\mathbf{x}, \mathbf{x}') = \exp\left( -\frac{\|\mathbf{x} - \mathbf{x}'\|^2}{2\sigma^2} \right)$. This corresponds to projecting into an infinite-dimensional space, which sounds wild but is mathematically clean. The RBF kernel is the most popular SVM kernel in practice because it handles most non-linearities well with just one hyperparameter (σ, the bandwidth) to tune.

Read this as two paths to the same destination. The top path explicitly transforms the features, which is expensive or impossible. The bottom path uses a kernel function to compute the final dot product directly, which is cheap. Both lead to the same SVM solution. The kernel trick is how you get the benefits of the high-dimensional space without paying the cost.

Concrete walkthrough of the quadratic kernel. Take a 2D input x₁ = (q₁, p₁) and x₂ = (q₂, p₂). Define the transformation $\phi(\mathbf{x}) = (q^2, \sqrt{2} q p, p^2)$ into 3D. The dot product in the transformed space is:

ϕ(x₁)^⊤ϕ(x₂) = q₁²q₂² + 2q₁p₁q₂p₂ + p₁²p₂²

Now compute the kernel k(x₁, x₂) = (x₁^⊤x₂)² = (q₁q₂ + p₁p₂)². Expand:

(q₁q₂ + p₁p₂)² = q₁²q₂² + 2q₁q₂p₁p₂ + p₁²p₂²

Identical. The kernel gives you the result of working in 3D by doing arithmetic in 2D. If you had instead projected into 100 dimensions, you’d still just compute (q₁q₂ + p₁p₂)²; the 100-dimensional explicit projection would take 100 multiplications per point, but the kernel takes 3 (two multiplies plus a square). The saving scales massively as the projected space grows.

Banking applications of non-linear SVM are narrow but real. In regulated credit risk, SVM with non-linear kernels is rarely used because the decision boundary is hard to explain to regulators and the model does not give calibrated probabilities out of the box. In fraud detection and AML, where interpretability requirements are less stringent and subtle non-linear patterns matter, RBF-kernel SVM was a staple technique through the late 2000s before being largely displaced by gradient boosting and later deep learning. You will still find it in legacy systems and as a baseline in model evaluations.

If asked: “Explain the kernel trick in one paragraph without jargon.”

What if the simplest possible algorithm is also one of the best?

k-Nearest Neighbours is the anti-algorithm. It trains nothing. It builds no model. It has no weights to fit, no parameters to optimise, no loss function to minimise. All it does is store the entire training set and, when asked to predict on a new input, find the k training examples closest to the new input and return the majority vote (for classification) or the average (for regression). That’s it. The whole algorithm fits in three sentences.

Despite its simplicity, or because of it, k-NN is remarkably useful, and it is the first technique every competent data scientist reaches for when they need a sanity check baseline. If your fancy new model can’t beat k-NN on a given problem, either the problem is extremely hard or your fancy model is broken. Either way, k-NN gives you an honest floor.

The formal algorithm. At prediction time:

1. For the new input x, compute the distance from x to every training example.
2. Identify the k training examples with the smallest distances.
3. For classification, return the majority class among those k. For regression, return the average target value.

At training time: do nothing. Just remember the data. This is what “instance-based learning” means, and k-NN is the canonical example. The training set is the model.

Choosing the distance metric. Everything depends on how you measure closeness. The most common choice is Euclidean distance:

$$d(\mathbf{x}_i, \mathbf{x}_k) = \sqrt{\sum_{j=1}^{D} (x_i^{(j)} - x_k^{(j)})^2}$$

In words: subtract corresponding entries of the two vectors, square the differences, sum them, take the square root. In two dimensions this is the familiar straight-line distance from school geometry. In higher dimensions it generalises directly.

Cosine similarity is another common choice, and especially popular when the vectors represent directions rather than positions. Defined as:

$$s(\mathbf{x}_i, \mathbf{x}_k) = \cos(\angle(\mathbf{x}_i, \mathbf{x}_k)) = \frac{\sum_{j=1}^{D} x_i^{(j)} x_k^{(j)}}{\sqrt{\sum_{j=1}^{D} (x_i^{(j)})^2} \cdot \sqrt{\sum_{j=1}^{D} (x_k^{(j)})^2}}$$

This is the dot product of the two vectors divided by the product of their lengths. If two vectors point in the same direction, cosine similarity is 1. If they are perpendicular, it’s 0. If they point in opposite directions, it’s −1. To use it as a distance (where smaller means closer), you negate it or subtract from 1. Cosine similarity is the standard metric for comparing text embeddings, document embeddings, and customer behavioural vectors where the magnitude is less informative than the direction.

Other distance metrics include:

• Manhattan (L1) distance: ∑_j|x_i^(j) − x_k^(j)|. Summed absolute differences, like city-block distance in a grid layout.
• Chebyshev distance: max_j|x_i^(j) − x_k^(j)|. The largest single-dimension gap.
• Mahalanobis distance: Euclidean distance but scaled by the inverse covariance matrix of the features. It accounts for feature correlations and different scales.
• Hamming distance: for binary or categorical vectors, the number of positions where they differ.

Choosing the right distance is a problem-specific decision. For numerical features on comparable scales, Euclidean is the default. For text, cosine. For high-dimensional sparse data, cosine or L1. For features on vastly different scales (e.g., age in years versus income in pounds), you should standardise first or use Mahalanobis.

Read this as the full flowchart of k-NN inference. No training step, no parameters to store; the algorithm is pure lookup and aggregation.

Choosing k. The single hyperparameter k controls the bias-variance trade-off. Small k (like 1) means each prediction depends on a single nearest neighbour, so the model is very flexible but sensitive to noise: one bad training example can flip a prediction. Large k (like 50) averages over many neighbours, smoothing out noise but also smoothing out real structure. The sweet spot depends on the problem and is typically found by validation.

A useful rule of thumb: start with $k = \sqrt{N}$ where N is the size of your training set. For N = 10, 000, try k = 100. Adjust from there based on validation performance. An even better practice is to use odd values of k for binary classification to avoid tied votes.

Concrete walkthrough: customer similarity for cross-sell. In the synthetic Merehaven case, suppose the commercial banking marketing team wants to identify which SMEs are most likely to take up a new foreign exchange hedging product. They have 500 SMEs that already hold it (the “positive” examples) and 50,000 SMEs that don’t (the potential market). For each SME, they have a 40-dimensional feature vector capturing industry, size, payment patterns, product usage, and geography. They use k-NN with k = 10 and cosine similarity.

For a candidate SME, they compute cosine similarity to all 50,500 training SMEs, pick the 10 closest, count how many are FX product holders, and emit that count (or its fraction) as a score. A candidate whose 10 nearest neighbours include 6 existing FX users gets a score of 0.6. A candidate whose 10 nearest include no existing users gets 0.0. The team ranks all candidates by this score and the top 2,000 get contacted by their analysts.

Formal statement. Given a training set {(x_i, y_i)}_i = 1^N and a distance function d, the k-NN prediction for new input x is:

ŷ(x) = aggregate({y_i : x_i ∈ NN_k(x)})

where NN_k(x) denotes the set of k training points closest to x under d, and “aggregate” is majority vote for classification or mean for regression. There are no parameters w or b; the hyperparameters are k and the choice of d.

Where this goes wrong. k-NN has several serious limitations.

First, prediction is slow and memory-heavy. Every prediction requires computing the distance to every training point. For a training set of a million examples this is expensive, and for billions it is infeasible with naive implementations. The fix is approximate nearest neighbour (ANN) data structures like ball trees, KD-trees, HNSW graphs, or IVF indexes. Modern vector databases like Faiss, Pinecone, Weaviate, and pgvector are essentially industrialised k-NN engines. They let you query over billions of vectors in milliseconds. The retrieval layer in any RAG (retrieval-augmented generation) system is exactly this: a big ANN index that pulls the most relevant documents by embedding similarity. When you hear “vector search,” think “k-NN at scale.”

Second, the curse of dimensionality. In high-dimensional feature spaces, the notion of “nearest neighbour” becomes statistically weak. Almost all pairs of points have similar distances to a given query, so the “10 nearest” are not meaningfully closer than the “100 nearest.” This is a deep property of high-dimensional geometry and it’s why naive k-NN performs poorly on, say, raw pixel representations of images. The cure is learned embeddings: instead of running k-NN on raw features, you train a model (often a neural network) to map raw inputs to a lower-dimensional space where meaningful similarity structure exists, and then run k-NN in that space. This is exactly how modern semantic search works.

Third, sensitivity to feature scaling. Because k-NN is based on distances, features with larger numeric ranges dominate features with smaller ranges. If one feature is “income in pounds” (range 0 to 500,000) and another is “years tenure” (range 0 to 30), income completely dominates any Euclidean distance calculation. The fix is to standardise all features to zero mean and unit variance before running k-NN. This is the first thing you do, always. Forgetting it is a rite of passage for every new data scientist.

And fourth, no natural interpretability beyond the neighbours themselves. You can’t write down “the k-NN model” as a set of weights or rules. The best explanation you can give for a prediction is “this new customer is similar to these ten past customers, and here’s why they were each labelled this way.” In many contexts this is actually the most useful form of explanation, but in regulated contexts where model risk teams expect parametric explainability, it can be awkward.

If asked: “Why is k-NN still useful in a world of deep learning?”

Optimise by observing the loss

What is every learning algorithm secretly doing?

There is a beautiful unification waiting for you in this section, and once you see it, you cannot unsee it.

Take any of the five algorithms from Chapter 3. Strip away the historical baggage, the names, the cute analogies. What’s left? In every case, it’s three things: a way of measuring how wrong the model is on a single example, a way of summing that wrongness across the training set, and a way of nudging the parameters to make the sum smaller. That’s the entire shape of supervised machine learning. If you can write down those three things for any model, you can train it.

Here’s the table.

The first three rows have explicit losses, criteria, and optimisers. The last two are the historical accidents we mentioned: invented by intuition, with criteria reverse-engineered later. Every modern algorithm follows the explicit-three-part pattern. Every neural network you will meet in Chapter 6 fits the same template.

Why does this matter for an enterprise architect at a bank? Because when you sit in a model design review and someone proposes a new approach, you can ask three precise questions: What’s the loss function? What’s the cost function? What’s the optimiser? If the answers are clear, the proposal is grounded. If any of them is vague, push back. “We’re going to use a deep learning approach” is not an answer; it’s a category. “We’re going to minimise binary cross-entropy across a dataset of past defaults using Adam with learning rate scheduling, trained via a managed training service Training on the selected cloud platform or a managed training service Training Jobs on the selected cloud platform” is an answer. The three-part decomposition is your filter for distinguishing real proposals from buzzwords.

It also matters for governance. When the PRA’s SS1/23 supervisory statement on model risk management asks how a model is fitted, the right answer is structured: loss function, cost function, optimisation method, hyperparameters. A model risk management team that has internalised this structure will produce documentation that satisfies regulators. A team that has not will write essays full of hand-waving and end up rewriting them.

If asked: “What are the three building blocks of every supervised learning algorithm?”

Why do we ever need an iterative optimiser at all?

Here’s a fair question. In Chapter 3 we saw that linear regression has a closed-form solution. You don’t need to iterate. You compute one matrix inversion and you have the answer. So why do machine learning engineers spend so much time talking about gradient descent?

Three reasons.

First, most useful models do not have closed-form solutions. Logistic regression, almost every flavour of neural network, regularised models with non-smooth penalties, models with custom losses or constraints, all of these require iterative optimisation. Closed-form is the lucky exception, not the rule.

Second, even when a closed form exists, it can be impractical at scale. The closed-form linear regression solution requires computing (X^⊤X)⁻¹, which is a D × D matrix inversion. For D = 100 this is fine. For D = 10, 000 it’s painful. For D = 1, 000, 000 it is impossible on a single machine. Iterative methods like gradient descent have memory and compute costs that scale much more gently with model size, which is why they are the only realistic option for modern large models.

Third, the closed form does not generalise. As soon as you add a custom regularisation term that has no clean derivative, or you use a non-standard loss, or you have constraints on the parameters, the closed form vanishes. Iterative optimisation, by contrast, is general-purpose: tell me how to compute the loss and its gradient at any point, and I can find the minimum.

So the practical situation is: closed forms are nice when you can use them, but you can’t usually use them, and even when you can, you should still understand iterative optimisation because the next thing you do will probably need it.

The algorithm is humble in a way that more clever methods often are not. It does not try to plan a clever route. It does not look ahead. It just feels the immediate slope and steps downhill. And it works, more or less, for almost every loss function in machine learning, including ones with millions or billions of dimensions where no human or analytical method could plan a route through the space.

How does gradient descent actually work, line by line?

Gradient descent alternates observation and correction. It computes predictions, measures residual error, differentiates the aggregate loss and applies a bounded parameter update. The implementation below makes the data contract and update direction explicit.

def linear_gradient_step(x, y, weight, bias, rate):
    prediction = x @ weight + bias
    residual = prediction - y
    grad_weight = (2 / len(x)) * (x.T @ residual)
    grad_bias = (2 / len(x)) * residual.sum()
    return weight - rate * grad_weight, bias - rate * grad_bias

The code is intentionally small. A release-tested trainer must also define batching, stopping, numerical checks, random seeds, checkpointing and the evidence recorded for each run.

The setup. We have 200 data points. Each is a (spending on radio advertising in millions, units sold) pair. We want to fit a single-feature linear regression to predict sales from spending. The model is ŷ = wx + b, with two scalar parameters w and b to learn.

The loss is mean squared error:

$$L(w, b) = \frac{1}{N} \sum_{i=1}^{N} (y_i - (w x_i + b))^2$$

To minimise L via gradient descent, we need its partial derivatives with respect to w and b. Apply the chain rule:

$$\frac{\partial L}{\partial w} = \frac{1}{N} \sum_{i=1}^{N} -2 x_i (y_i - (w x_i + b))$$

$$\frac{\partial L}{\partial b} = \frac{1}{N} \sum_{i=1}^{N} -2 (y_i - (w x_i + b))$$

Reading these in plain English: for each training point, compute the residual (true minus predicted), multiply by negative two and (for the w derivative) by the feature value, sum across all points, average. The result is a number telling you how the loss would change if you nudged w (or b) up or down.

The gradient descent update rule is:

$$w \leftarrow w - \alpha \frac{\partial L}{\partial w}, \quad b \leftarrow b - \alpha \frac{\partial L}{\partial b}$$

We subtract because the partial derivative points uphill and we want to go downhill. The learning rate α controls how big each step is.

Walk through it line by line.

dl_dw = 0.0 and dl_db = 0.0 initialise accumulators for the gradient with respect to each parameter. We will sum contributions from every training example into these.

N = len(spendings) gets the number of training examples, which we’ll use to average the gradient later.

The for loop iterates over every training example. For each example, it computes the residual (sales[i] - (w*spendings[i] + b)) and uses it to update the two accumulators. The expressions -2*spendings[i]*residual and -2*residual are exactly the per-example terms in the partial derivative formulas above. After the loop, dl_dw and dl_db contain the sum (not yet the mean) of the gradient contributions.

w = w - (1/float(N))*dl_dw*alpha updates w by subtracting the average gradient times the learning rate. The (1/float(N)) converts the sum to a mean. The same thing happens for b.

return w, b hands back the updated parameters so the caller can use them in the next epoch.

The training loop wraps this in a counter:

def train(spendings, sales, w, b, alpha, epochs):
    for e in range(epochs):
        w, b = update_w_and_b(spendings, sales, w, b, alpha)

        # log the progress
        if e % 400 == 0:
            print("epoch:", e, "loss: ", avg_loss(spendings, sales, w, b))

    return w, b

For each epoch, we update the parameters once and occasionally log the loss so we can watch the model improve. The avg_loss helper just computes the mean squared error directly:

def avg_loss(spendings, sales, w, b):
    N = len(spendings)
    total_error = 0.0
    for i in range(N):
        total_error += (sales[i] - (w*spendings[i] + b))**2
    return total_error / float(N)

And the prediction function, once training is done, is one line:

def predict(x, w, b):
    return w*x + b

That’s the entire gradient descent training procedure for linear regression in pure Python. About thirty lines. No external libraries.

epoch: 0     loss: 92.32
epoch: 400   loss: 33.79
epoch: 800   loss: 27.99
epoch: 1200  loss: 24.33
epoch: 1600  loss: 22.03
...
epoch: 2800  loss: 19.08

The loss starts at 92, drops fast in the first few hundred epochs as the algorithm finds the broad shape of the solution, then settles into slow refinement as it homes in on the exact bottom. After 15,000 epochs the parameters are essentially converged, and predicting on a new spending of 23.0 gives 13.97 units. This is the same answer the closed-form solution would give. We just got there by stumbling downhill instead of by matrix inversion.

Read this as one full training loop. Most modern training, no matter how complex the model, is some elaboration of this picture. Even GPT-class language model training is, at its core, this loop applied at extreme scale with fancy variants of step E and F.

Why does this work? Because at every point on the loss surface, the negative gradient is the direction of steepest descent, so each step makes the loss smaller (provided α is small enough). Repeated steps drive you toward a local minimum. For convex losses like linear regression and logistic regression, the only local minimum is the global minimum, so you are guaranteed to find the optimal parameters. For non-convex losses like neural networks, you find some local minimum, which is usually good enough in practice.

Where this goes wrong, in three places. First, the learning rate. As we saw in the failure specimen at the top of this section, if α is too large, each step overshoots and the loss bounces or diverges. If α is too small, training crawls and may never finish. Picking the right learning rate is a problem-specific judgement and there’s a small industry of techniques (learning rate finders, schedulers, warm-up periods) to help.

Second, the scale of the features. If one feature ranges from 0 to 1 and another ranges from 0 to 1,000,000, the gradient with respect to the second one will be vastly larger than with respect to the first. A single learning rate cannot serve both. The fix is to standardise features to comparable scales before training, which we covered in Chapter 3 in the k-NN section and which applies even more strongly here.

Third, initialisation. We start with w = 0, b = 0 in the linear case and it works fine because the loss surface is convex. For neural networks with many parameters, all-zeros initialisation is catastrophic because it makes every neuron in a layer compute the same thing, and they never differentiate. The standard fix is small random initialisation (e.g., Xavier or He initialisation), tailored to the architecture. Bad initialisation in deep networks can mean the difference between converging to a useful model and producing garbage. We’ll come back to it in Chapter 6.

If asked: “Walk me through one iteration of gradient descent on a logistic regression model.”

Why is full-batch gradient descent too slow for real datasets?

The procedure above computes the gradient using every training example before taking a single step. For 200 advertising data points, that’s instant. For the 180,000 SME loans we’ve been imagining throughout this edition, it’s still fine on a laptop. For ImageNet’s 14 million images or a language model trained on hundreds of billions of tokens, it is utterly impractical. A single epoch would take hours or days, and you typically need many epochs to converge.

The fix, invented in 1951 by Herbert Robbins and Sutton Monro at Rutgers and a managed cachecovered countless times since, is stochastic gradient descent (SGD). The idea is delightfully cheap: instead of computing the gradient on the whole training set, compute it on just one randomly chosen example. Take a tiny, noisy step in that direction. Then move to the next random example and do it again. Each step is now ridiculously cheap, but each step is also a much worse estimate of the true gradient.

The remarkable empirical fact is that this works. The noisy steps don’t cancel out the descent direction; they just add jitter. On average, you still descend, just with a wiggly path instead of a smooth one. And because each step is so cheap, you can take far more of them in the same wall-clock time, and you usually end up converging faster overall.

In practice, pure SGD (one example at a time) is almost never used. The compromise that nearly everyone uses is minibatch SGD: compute the gradient on a small randomly chosen subset (a “minibatch”) of examples, typically 32 or 64 or 256 or some power of two, take a step, and move on. This gives you most of the speed of pure SGD while reducing the noise to a manageable level. Modern GPUs are also brutally efficient at computing matrix operations on batches of moderate size, so minibatch SGD lines up well with the hardware as well as the math.

Read this as three speeds along a spectrum. Full-batch is slow and accurate. Pure SGD is fast and noisy. Minibatch is the sweet spot, and is what every production ML training loop on the planet actually does.

The Merehaven Bank context. When the bank trains a fraud detection model on tens of millions of historical transactions, it does not load all of them into memory and compute one giant gradient. It streams batches off disk, one minibatch at a time, computes the gradient on each batch, takes a step, and moves on. This is also why distributed training works: you can have multiple workers each computing gradients on different batches, then averaging them. Modern training frameworks like PyTorch, JAX, and TensorFlow are built around this minibatch loop, and they make distributed SGD as easy as setting a configuration flag.

The variance/speed trade-off controlled by batch size is one of the few hyperparameters that has solid theoretical understanding. Smaller batches give more frequent updates but noisier gradients. Larger batches give better gradient estimates but slower updates. The right batch size depends on memory budget, model size, and (counter-intuitively) on whether you want slightly more generalisation noise to escape sharp minima. For neural networks specifically, very small batches (32-128) often generalise better than very large ones, even though large batches train faster per epoch. This is one of those empirical findings that makes deep learning feel more like a craft than a science.

If asked: “Why use minibatch SGD instead of full-batch gradient descent?”

What makes Adam better than vanilla SGD?

Once you have minibatch SGD, you have a working training algorithm. But there are several stubborn problems with it that researchers have spent thirty years fixing, and the modern descendants of SGD that come out of that work are materially better at training the models we actually care about.

Three problems, three fixes.

Problem one: features have different curvatures. Some features have steep gradients (small change in parameter, big change in loss). Others have shallow gradients. With a single learning rate, the steep ones get over-corrected at every step while the shallow ones get under-corrected, and convergence is slow because you have to set the learning rate small enough not to blow up the steep ones, which means the shallow ones barely move.

The fix: per-parameter adaptive learning rates. Adagrad (Duchi et al., 2011) keeps a running sum of squared past gradients for each parameter and divides the current update by the square root of that sum. Parameters with historically large gradients get small effective learning rates. Parameters with historically small gradients get large effective learning rates. The result is that different parameters move at different speeds, all calibrated to their own gradient history, and you don’t have to hand-tune them.

Problem two: gradients are noisy and oscillate. With minibatch SGD, the gradient bounces around the true descent direction. In a long valley with gentle gradient along the valley floor and steep gradients across the sides, vanilla SGD oscillates side to side and crawls along the floor, wasting most of its motion on the cross-direction oscillations.

The fix: momentum. Polyak’s momentum (1964, but popularised in deep learning much later) maintains an exponentially weighted moving average of past gradients and uses that as the update direction instead of the current gradient alone. The effect is exactly like a heavy ball rolling down a hill: it builds up speed in directions where the gradient is consistent and damps out direction-flipping noise. In the long-valley analogy, the momentum cancels out the side-to-side oscillation and accelerates motion along the valley floor.

Problem three: gradients vanish or explode in deep networks. When you stack many layers, each layer’s gradient depends on the layers above it via the chain rule, and the per-layer factors can multiply together to give either tiny or enormous overall gradients. Tiny gradients mean the early layers don’t learn. Enormous gradients mean training diverges.

The fix (partial): adaptive optimisers like Adam. Adam (Kingma and Ba, 2015), which stands for “adaptive moment estimation,” combines per-parameter adaptive learning rates (like Adagrad) with momentum (like Polyak) and adds bias correction to handle the early steps of training when the running averages are unreliable. The result is an optimiser that “just works” on a wide range of problems without much tuning, and it has become the default for training neural networks in the 2020s. Variants include RMSprop (Hinton, 2012, never published as a paper but presented in a Coursera lecture), AdamW (Loshchilov and Hutter, 2019), and Lion (Chen et al., 2023).

Here is the abridged Adam update rule for a single parameter θ:

m_t = β₁m_t − 1 + (1 − β₁)g_t

v_t = β₂v_t − 1 + (1 − β₂)g_t²

m̂_t = m_t/(1 − β₁^t)

v̂_t = v_t/(1 − β₂^t)

$$\theta \leftarrow \theta - \alpha \cdot \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$$

Don’t memorise this. Just notice the structure. m_t is a moving average of the gradient (momentum). v_t is a moving average of the squared gradient (per-parameter scale). The hat versions are bias corrections that matter only at the start. The final update divides the smoothed gradient by the smoothed scale, giving each parameter its own effective learning rate. The hyperparameters β₁, β₂, ϵ have well-known defaults (0.9, 0.999, 10⁻⁸) that work for almost everything, which is why “use Adam with the defaults” is a perfectly reasonable starting point for a new project.

Read this as the family tree of optimisers. Each step adds one mechanism to fix one problem with the previous step. By the time you reach Adam, you have an optimiser that handles the most common pathologies and trains most things “out of the box.”

Important nuance. Adaptive optimisers like Adam are fantastic for neural networks but sometimes worse than plain SGD with momentum on simple convex problems. If you train a logistic regression with Adam, you may converge to a slightly worse solution than you would with plain gradient descent or with L-BFGS (a quasi-Newton method that is still the standard for medium-scale convex problems). Use the right tool for the job. For simple convex models, scikit-learn’s defaults usually pick a good solver automatically. For neural networks, Adam or AdamW with default hyperparameters is the right starting point.

importantly, gradient descent and its variants are not machine learning algorithms. They are general-purpose minimisation tools. They solve any minimisation problem where the function is differentiable. The same Adam optimiser that trains GPT-class language models will, with no modification, also train your logistic regression for credit default prediction or your neural network for fraud scoring. It’s the loss function and the model architecture that make the model what it is. The optimiser is just the engine that pushes the parameters around.

If asked: “Why has Adam become the default optimiser for neural networks?”

What other particularities make algorithms differ in practice?

Categorical features. Some algorithms accept categorical features natively. Decision trees in particular can split on categorical features like “sector code” or “product type” without any preprocessing. You hand them the strings and they figure out which categories belong on which side of the split. Other algorithms, linear regression, logistic regression, SVM, k-NN, require all features to be numerical. To use them with categorical features, you need to convert categories into numbers, usually by one-hot encoding (creating a new 0/1 feature for each category) or by target encoding (replacing each category with the mean of the target for that category). We’ll cover both in Chapter 5. For now, just know that “this algorithm needs numerical inputs” is a real constraint that influences which algorithm fits which problem.

Class weighting. Some algorithms, including SVM, scikit-learn’s logistic regression, and most tree ensembles, let you specify a weight per class. If you set the weight of the positive class to 10 and the weight of the negative class to 1, the optimiser is told that misclassifying a positive example is ten times as costly as misclassifying a negative one. This is enormously useful for imbalanced problems like fraud detection, where positives are rare and you do not want the model to ignore them by defaulting to always predicting “not fraud.” Class weighting is a cheap fix that often improves recall on the rare class without much code change. We’ll come back to imbalance handling in Chapter 5.

Probability outputs versus class outputs. Some classification algorithms naturally output a probability or score in (0, 1). Logistic regression and decision trees do this by construction. Others, including the original SVM and naive k-NN, only output a class label, not a probability. If you need probabilities (and in regulated banking you often do, because expected loss calculations need them), you have two choices. One is to use an algorithm that produces probabilities natively. The other is to use a calibration technique like Platt scaling (fit a logistic regression on the model’s raw scores) or isotonic regression (fit a non-parametric monotone calibration curve). Calibration is a separate post-processing step that is critical for any regulated ML model where the output is interpreted as a probability of default or fraud.

Online versus batch learning. Some algorithms can be incrementally updated as new data arrives, without retraining from scratch. Others must be retrained on the full dataset every time. This matters when your data is changing fast and you cannot afford to wait days for a retraining run. Online algorithms include online SGD variants of linear and logistic regression, certain neural networks with online updates, and a few specialised techniques. Batch algorithms include trees, SVM (in its standard formulation), and most kernel methods. In banking, the typical compromise is periodic retraining (weekly or monthly) on the full dataset, with monitoring in between to catch drift early.

Interpretability per the regulator. This isn’t really an algorithmic property, it’s a business constraint, but it shapes algorithm choice profoundly. Algorithms vary on how easy it is to explain their predictions to a non-technical audience or a model risk team. Decision trees are at one end (you can read the path). Linear and logistic regression are next (you can read the weights). Tree ensembles and kernel SVMs are in the middle (you can use feature importances and SHAP values). Deep neural networks are at the far end (interpretability requires post-hoc explainers and is often unsatisfying). For credit decisions on retail customers, interpretability is required by the decision policy under applicable customer-outcome obligations rules, which is a major reason that logistic regression remains the most common algorithm in retail credit risk despite being decades old.

Read this as a multi-axis decision matrix. Each axis is one practical question you ask when choosing an algorithm. The right answer is whichever algorithm sits at the intersection of acceptable answers across all axes. In banking that’s often one of the five from Chapter 3, sometimes augmented by ensemble methods we’ll meet later.

How do you know when to stop training?

Here’s a question that sounds simple and turns out to be the most important practical decision in any training run. You start gradient descent. The loss goes down. At what point do you stop?

The naive answer is “when the loss stops decreasing.” This is partly right and very dangerous. Two refinements turn it into a real engineering practice.

First refinement: which loss are we watching? The training loss often continues to decrease as long as the model has enough capacity to keep memorising the training set. If you stop when the training loss stops decreasing, you stop when the model has overfit completely, which is the worst possible time. The right loss to watch is the validation loss: the loss computed on a held-out set that the optimiser is not allowed to use during training. Validation loss decreases at first as the model learns generalisable patterns, then plateaus, then often starts increasing as the model begins memorising training-specific quirks that hurt generalisation. The right time to stop is around the minimum of validation loss, before it starts climbing.

Second refinement: when has it really plateaued? Validation loss is noisy. It can wiggle up and down by a few percent from epoch to epoch even on a healthy training run. If you stop the first time it ticks up, you’ll stop too early on most runs. The standard trick is patience: keep training for p more epochs after each new best validation loss, and stop only if no improvement appears in those p epochs. Typical patience values are 5 to 20 epochs, depending on the noise level of the validation curve. This procedure is called early stopping and it’s one of the most reliable forms of regularisation in machine learning.

Read this as the typical training trajectory. Both losses fall together at first, the validation loss bottoms out, the training loss keeps falling because the model is now memorising, and you should have stopped at that bottom. Early stopping with patience automates this decision.

Concrete numbers from a Merehaven Bank-style training run. Suppose you’re training a small classifier and you log losses every epoch. The validation loss curve looks like:

With patience set to 10 epochs, training would stop at epoch 30 (10 epochs after the last improvement at epoch 20) and the best model is the one saved at epoch 20. The training loss at epoch 40 is much lower (0.05 vs 0.16), but the validation loss is noticeably worse (0.32 vs 0.26). The model at epoch 40 has overfit. The model at epoch 20 is the one you ship.

This procedure has saved more banking ML projects than any other single technique. It costs you nothing and it produces models that generalise better than the same architecture trained for a fixed number of epochs.

Where this goes wrong. The validation set itself can mislead you. If your validation set is too small, the validation loss is too noisy to give you a reliable stopping signal. If your validation set is not representative of production traffic (for example, sampled randomly when production has temporal drift), the validation loss can look great while the production performance is poor. The fix for the first problem is to use a larger validation set or to use cross-validation. The fix for the second is to use time-based validation splits, which we covered in Chapter 1: train on 2012-2018, validate on 2019, test on 2020-2021. Always validate on a slice that mimics how the model will be deployed.

If asked: “What’s the difference between early stopping and other forms of regularisation?”

What does a single gradient descent step actually look like for a neural network?

We’ve seen the linear regression case in detail. Let’s take one more concrete walkthrough, this time for a tiny neural network, because the same loop powers everything from a 13,000-parameter classifier to a 400-billion-parameter language model. Once you’ve seen the steps for a small network, you’ve seen them for any network.

Picture the smallest possible neural network for binary classification: two input features, one hidden layer with two neurons, ReLU activation, one output neuron, sigmoid activation. The network has 2 × 2 + 2 + 2 × 1 + 1 = 9 parameters in total. We’ll train it on one example with input x = (1.0, 0.5) and label y = 1. Set the initial weights randomly, say $W_1 = \begin{pmatrix} 0.3 & -0.2 \\ 0.1 & 0.4 \end{pmatrix}$, b₁ = (0.0, 0.0), w₂ = (0.5, −0.3), b₂ = 0.1. Learning rate α = 0.1.

Forward pass. Compute the hidden layer pre-activations: z₁ = W₁x + b₁ = (0.3 ⋅ 1 + (−0.2) ⋅ 0.5, 0.1 ⋅ 1 + 0.4 ⋅ 0.5) = (0.2, 0.3). Apply ReLU: both are positive, so h = (0.2, 0.3). Compute the output pre-activation: z₂ = w₂ ⋅ h + b₂ = 0.5 ⋅ 0.2 + (−0.3) ⋅ 0.3 + 0.1 = 0.11. Apply sigmoid: p̂ = σ(0.11) ≈ 0.5275. Compute the loss: cross-entropy −log (0.5275) ≈ 0.640.

Backward pass. This is where the chain rule from Chapter 2 earns its keep. We need the gradient of the loss with respect to every parameter in the network. The trick is to start from the output and work backwards, reusing computations as we go. This is exactly what backpropagation does.

The gradient of the cross-entropy loss with respect to the output pre-activation z₂ is p̂ − y = 0.5275 − 1 = −0.4725. (This clean form is one reason cross-entropy and sigmoid pair so beautifully.) The gradient with respect to b₂ is the same: −0.4725. The gradient with respect to each component of w₂ is the gradient with respect to z₂ times the corresponding hidden value: (−0.4725 ⋅ 0.2, −0.4725 ⋅ 0.3) = (−0.0945, −0.1418).

Propagate back to the hidden layer. The gradient with respect to each component of h is the gradient with respect to z₂ times the corresponding w₂: (−0.4725 ⋅ 0.5, −0.4725 ⋅ (−0.3)) = (−0.2363, 0.1418). ReLU’s derivative is 1 for positive inputs and 0 for negative, and both pre-activations were positive, so the gradient with respect to z₁ is the same: (−0.2363, 0.1418).

Now for W₁ and b₁. The gradient with respect to b₁ is the same as with respect to z₁: (−0.2363, 0.1418). The gradient with respect to each entry of W₁ is the gradient with respect to the corresponding entry of z₁ times the corresponding input feature. For example, ∂L/∂W₁^(1, 1) = −0.2363 ⋅ 1 = −0.2363. The full gradient matrix is $\begin{pmatrix} -0.2363 & -0.1182 \\ 0.1418 & 0.0709 \end{pmatrix}$.

Update step. Subtract α times each gradient from each parameter. New W₁: $\begin{pmatrix} 0.3 - 0.1 \cdot (-0.2363) & -0.2 - 0.1 \cdot (-0.1182) \\ 0.1 - 0.1 \cdot 0.1418 & 0.4 - 0.1 \cdot 0.0709 \end{pmatrix} = \begin{pmatrix} 0.3236 & -0.1882 \\ 0.0858 & 0.3929 \end{pmatrix}$. Similarly update b₁, w₂, and b₂. That’s one step. Repeat with the next training example.

Why bother walking through the arithmetic? Two reasons. First, when something goes wrong in a neural network training run, the diagnostic skill of “I can imagine what’s happening at the parameter level” is the difference between a five-minute fix and a five-day fix. Second, every modern deep learning framework (PyTorch, JAX, TensorFlow) automates this whole calculation through automatic differentiation. You define the forward pass; the framework computes all the gradients. But the framework is doing exactly the chain-rule walk we just did, scaled up to billions of parameters and parallelised across a GPU. The mental model is the same.

Read this as the universal training loop for any neural network. The forward pass produces a prediction. The loss measures wrongness. The backward pass propagates gradients via the chain rule. The optimiser updates the parameters. Every framework, every architecture, every model in this edition and out of it follows this loop. The differences are in what goes inside each box: how many layers, what activations, what loss, what optimiser. The skeleton is fixed.

Where the chain rule earns its keep. The reason backpropagation is feasible at all is that it reuses intermediate computations. Without it, computing the gradient of a loss through a 100-layer network would require 100 separate forward passes, one per layer. With backpropagation, you do one forward pass and one backward pass, and you get all the gradients in essentially the same time it takes to compute the loss. The trick was independently discovered several times in the 1970s and 1980s, most famously a managed cachecovered and popularised by Rumelhart, Hinton, and Williams in their 1986 Nature paper, which is widely credited with kicking off the second wave of neural network research. Without that paper, deep learning as we know it does not happen. The whole edifice of modern AI rests on the chain rule, applied carefully, plus a lot of GPUs.

If asked: “What is backpropagation, and why is it more efficient than computing gradients naively?”

Part III: Treat data and evaluation as the product

Most apparent model failures begin in representation, leakage, sampling or metric choice. This part makes those choices visible before any score reaches a decision route.

Prepare data before judging models

Feature engineering changes the hypothesis space

Andrew Ng, the Stanford professor who has done as much as anyone to bring machine learning to the wider world, has been quoted in countless talks saying some version of: “Coming up with features is difficult, time-consuming, requires expert knowledge. Applied machine learning is basically feature engineering.” Every working data scientist nods at this and every introductory ML course minimises it, because feature engineering is hard to teach in a textbook and easy to gloss over with toy datasets. In real banking work, feature engineering is where most of the value is created and most of the time is spent.

Here’s a concrete example. Imagine the head of commercial banking comes to your team and says: “We want to predict which of our SME customers are likely to need additional working capital over the next quarter, so analysts can have proactive conversations.” Sounds simple enough. The dataset is “everything we know about every SME customer In the synthetic Merehaven case,” which is several hundred fields across multiple systems. The label is some operationalisation of “needed additional working capital,” which doesn’t exist as a column anywhere; you have to construct it.

What does feature engineering for this problem actually look like?

Step one: invent the label. “Needed additional working capital” is a business concept, not a database column. You have to translate it into something measurable. Candidates: drew on overdraft to within 5% of limit at any point in the next 90 days; took up a new term loan in the next 90 days; had a missed direct debit followed by a manual transfer from a related account. Each candidate captures part of the concept and misses part. You discuss with the business, pick one or two, and document why.

Step two: pick the time horizon and snapshot date. You need to define, precisely, what “now” means for each training example. Is the snapshot the first day of each month? The end of each quarter? The day of each customer’s annual review? Each choice affects which features are available and which would leak. You also need to define how far forward to look for the label (90 days? 180?), and you have to make sure your training labels respect that horizon: a customer in the December 2022 snapshot can only be labelled if you have data through March 2023.

Step three: extract features from raw transactional data. This is where the real work happens. For each customer at each snapshot, you need to compute features like:

• Average end-of-day current account balance over the last 30 days, 60 days, 90 days
• Standard deviation of end-of-day balance over each window
• Number of distinct counterparties paid in the last quarter
• Fraction of days in the last 30 with overdraft usage above 50% of limit
• Trend (slope) in monthly aggregate inflows over the last six months
• Days since last late direct debit
• Ratio of payroll outflows to total outflows
• Sector-relative size percentile

Each of these is a piece of code, possibly hundreds of lines, that walks the customer’s transaction history and computes a number. Each one needs to be unit-tested. Each one needs a precise definition that matches between the training-time pipeline and the serving-time pipeline. Each one is a potential source of bugs.

Step four: handle the categorical features. Customer’s primary sector code, region, banking platform (legacy mainframe vs. modern stack), product mix, analyst team. None of these are numbers. Most ML algorithms need numbers. We’ll cover the techniques in a moment.

Step five: handle missing values. Some customers don’t have a sector code because of a data migration in 2017 that lost it. Some don’t have a director’s Experian score because the director is foreign. Some don’t have a confirmed turnover because they joined recently. Each missingness pattern needs a deliberate decision: drop the example, impute with a sensible value, or add an explicit “missing” indicator feature.

Step six: scale and normalise. Once you have all your numerical features, they will be on wildly different scales. Account balance in pounds. DSCR as a ratio. Age in years. Without scaling, gradient descent and distance-based methods will misbehave.

That’s all just the engineering. None of it is choosing an algorithm. None of it is tuning hyperparameters. None of it is “machine learning” in the popular sense. All of it is the prerequisite for any of those steps to work.

Read this as the pipeline that exists in production for any nontrivial banking ML system. Each box is owned by an engineer, instrumented for observability, and version-controlled. The whole thing runs on a schedule (nightly, hourly, or in real time depending on the use case) and is the substrate on which the models actually live.

If asked: “What’s the most important skill for an ML engineer in banking?”

How do you turn red, yellow, and green into numbers?

Most learning algorithms speak only the language of numbers. If you have a categorical feature like sector code or region, you need to translate it before you can feed it to logistic regression, SVM, k-NN, or a neural network. The standard translation is one-hot encoding.

The idea is exquisitely simple. For each possible category, create a new binary feature that is 1 when the original feature equals that category and 0 otherwise. A “colour” feature with three possible values becomes three new features:

red = [1, 0, 0]

yellow = [0, 1, 0]

green = [0, 0, 1]

A customer who is “red” gets the vector [1, 0, 0] in those three new positions. A customer who is “yellow” gets [0, 1, 0]. The original categorical feature is dropped and replaced by the binary expansion.

Why not just map red to 1, yellow to 2, green to 3? Because that imposes an order and a distance that don’t exist. By labelling them 1, 2, and 3, you are telling the model that yellow is “between” red and green, that the gap from red to yellow equals the gap from yellow to green, and that “twice red equals yellow.” None of these is true for unordered categories. If you give the algorithm this fake structure, it will try to find regularity in it, and any regularity it finds is noise that will hurt generalisation.

The cost of one-hot encoding is dimensionality. A categorical feature with 50 possible values becomes 50 new features. A sector code field with 1,200 SIC codes becomes 1,200 new features. A merchant ID field in payments data can have hundreds of thousands of distinct values, each of which would generate its own column. This is called the high-cardinality problem and it hits banking ML hard, because real banking data is full of high-cardinality categorical fields.

The fixes for high cardinality include:

• Grouping rare categories into an “other” bucket, keeping only the most common N values as their own categories
• Target encoding, where you replace each category with the mean target value for that category in the training data (a powerful but leakage-prone technique that requires careful out-of-fold computation)
• Embedding learning, where the categories are mapped to dense low-dimensional vectors by a small neural network trained jointly with the main model, which is what every modern recommender system does for product IDs and user IDs
• Frequency encoding, where each category is replaced by how often it appears in the training set

When ordinal encoding is fine. If your categorical variable has a genuine order, like “poor, decent, good, excellent” for an article quality rating, then mapping to 1, 2, 3, 4 is appropriate. The order is real and the algorithm can use it. The rule is: ordered categories get integer encoding; unordered categories get one-hot.

A Merehaven Bank example. When building a credit scoring model for SME loans, your categorical features might include: SIC code (high cardinality, no order, group rare codes and one-hot the rest), region (low cardinality, no order, one-hot), entity type (low cardinality, no order, one-hot), credit rating grade if from internal scorecards (low cardinality, ordered, integer encode), and originating channel (low cardinality, no order, one-hot). Each gets a different treatment based on its properties.

If asked: “When would you not use one-hot encoding for a categorical feature?”

When is a number not really a number?

Sometimes the opposite of one-hot encoding is what you want. You have a continuous numerical feature, but you suspect that what really matters is which “bucket” the value falls into, not the exact value. Binning (also called bucketing) chops a continuous variable into ranges and creates a categorical feature out of those ranges, which you can then one-hot encode.

Why would you ever do this? Three reasons.

First, if the relationship between the feature and the target is highly non-linear, binning gives a linear model the freedom to learn a different effect for each bin. A logistic regression with raw age can only learn “default risk grows linearly with age” or something equally constrained. With age bins, it can learn “default risk is high for 18-24, low for 25-45, medium for 46-65, high for 65+.” Each bin has its own weight and the model can shape the relationship freely.

Second, if the feature has sharp thresholds in the business logic, binning makes those thresholds explicit. Annual turnover thresholds for SME segmentation (under £2m, £2-10m, £10-50m, over £50m) are defined by the bank, not by some smooth relationship in the data. Binning along those thresholds means the model’s representation matches the business’s.

Third, binning can help with small datasets. If you have only a few hundred examples and a feature with a complex underlying relationship, binning is a form of regularisation: it forces the model to use a coarser representation that is harder to overfit.

The cost of binning. You throw away information. Two customers with ages 30 and 45 may end up in the same bin and be treated identically by the model, even though they’re very different in reality. The art is in choosing bin boundaries that preserve the structure that matters and discard the noise that doesn’t. Common choices include equal-width bins, equal-frequency bins (each bin contains the same number of training examples), and decision-tree-derived bins (let a small decision tree find the best split points).

When binning hurts. For modern flexible models like gradient boosted trees, binning is usually unnecessary because the model can find non-linearities itself. For deep neural networks, binning is rare because you typically want the gradient to flow continuously through the input. Binning shines for linear models, especially when interpretability for regulators is important: a credit scorecard that says “20 points for ages 25-34, 15 points for 35-44, 10 points for 45-54” is exactly what a credit officer wants to see, because it maps directly to the way they think.

If asked: “When would you bin a continuous feature?”

Why does feature scaling matter so much?

We touched on this in Chapter 4 with the failure specimen about the credit card churn model whose loss wouldn’t decrease until the features were normalised. Now let’s do it properly.

The problem: real banking features come on wildly different numerical scales. Customer income in pounds (range 0 to 500,000). Age in years (range 18 to 100). Number of products held (range 0 to 15). Days since last late payment (range 0 to 3,650). Debt service coverage ratio (range 0.5 to 5). If you feed these straight into a model that depends on distances or gradients, the high-magnitude features will dominate everything.

There are two standard fixes: normalisation and standardisation. They are different, and the difference matters.

Normalisation (also called min-max scaling) rescales each feature to a fixed range, usually [0, 1] or [−1, 1]. The formula is:

$$\bar{x}^{(j)} = \frac{x^{(j)} - \min^{(j)}}{\max^{(j)} - \min^{(j)}}$$

where min^(j) and max^(j) are the smallest and largest values of feature j across the training set. After this transformation, every value of feature j is in [0, 1], with the smallest training value mapped to 0 and the largest to 1.

Standardisation (also called z-score normalisation) rescales each feature to have mean 0 and standard deviation 1. The formula is:

$$\hat{x}^{(j)} = \frac{x^{(j)} - \mu^{(j)}}{\sigma^{(j)}}$$

where μ^(j) is the mean and σ^(j) is the standard deviation of feature j across the training set. After standardisation, the feature has the properties of a standard normal distribution: centred at zero, spread of one. Values can be negative or positive and are not bounded.

1. Unsupervised learning (clustering, PCA, etc.) often benefits more from standardisation than from normalisation.

2. Standardisation is preferred for features that are roughly normally distributed (the classic bell curve). Normalisation forces them into a bounded range that distorts the natural shape.

3. Standardisation is preferred for features with extreme values or outliers. Normalisation crushes the typical values into a tiny range while the outliers occupy the rest of the [0, 1] interval. A feature where 99% of values are between 0 and 100 but one value is 100,000 will, after normalisation, have almost all values clustered near zero, which destroys the model’s ability to use them.

4. In all other cases, normalisation is fine, and is often slightly faster because it doesn’t require recomputing the mean and standard deviation from scratch when you add new data.

The critical operational rule. You compute the scaling parameters (min , max , μ, σ) on the training set only. Then you use those same parameters to transform the validation set, the test set, and any production data. Never recompute on the validation or test set, and never recompute on each batch of incoming production data. If you do, you leak information from the holdout into the training process and your evaluation becomes optimistic. Modern feature stores enforce this discipline by versioning the scaling parameters as part of the feature definition. On the selected cloud platform, a managed training service Feature Store handles this via its offline/online store architecture; on the selected cloud platform, a managed training service Feature Store does the same with an analytical warehouse as the offline backing store.

Why does scaling help? Two reasons. First, gradient descent. If one feature is on a scale 1000x larger than another, the gradient with respect to the larger feature is much larger, and a single global learning rate cannot serve both: it has to be small enough not to blow up the large feature, which means the small feature barely moves. Scaling equalises the gradient magnitudes and lets training converge much faster. Second, distance-based methods like k-NN and clustering measure feature differences using Euclidean distance, which is dominated by features with large numeric ranges. Without scaling, the largest-range feature is effectively the only one that matters for finding neighbours.

Where scaling doesn’t matter. Tree-based methods (decision trees, random forests, gradient boosted trees) don’t need feature scaling. They split on individual features one at a time, and the choice of split point is invariant to monotonic transformations like scaling. So if you’re using an XGBoost model for credit risk, you can skip scaling entirely and the model will be unaffected. This is one of the small but real reasons tree ensembles are easier to deploy than neural networks: less preprocessing fragility.

If asked: “What’s the difference between normalisation and standardisation, and when do you use each?”

What do you do when the data has holes?

Real banking data has missing values everywhere. A customer’s industry code is blank because the migration in 2017 lost it. A director’s Experian score is missing because the director is non-UK resident. A management accounts file shows no EBITDA because the company hasn’t filed yet. Missing data is the rule, not the exception.

You have three families of strategies for handling missing values, and the right choice depends on how the values are missing and how much data you have.

Strategy one: drop the example. If you have lots of data and only a small fraction has missing values, you can simply drop the rows with any missingness. This is clean but throws away information. You should never drop more than a small percentage of your data this way, and you should never drop without first checking whether the missingness is random or systematic. If non-UK residents are systematically dropped, you’ve just biased your model against non-UK residents, which is both an analytical error and a fair-lending risk.

Strategy two: use an algorithm that handles missing values natively. Decision trees, gradient boosted trees, and some implementations of k-NN can deal with missing values directly. C4.5 and its descendants have explicit machinery for routing examples with missing values down both branches of a split with appropriate weighting. XGBoost learns which direction to send missing values at each split. If you can use one of these algorithms, you avoid the imputation problem entirely.

Strategy three: impute the missing values. Fill in the holes with some sensible substitute, then train as if the data were complete. There are several imputation techniques.

Mean imputation. Replace missing values with the average value of that feature in the training set. Simple, fast, often surprisingly effective. Loses information about variability and can systematically bias the model toward the centre.

Out-of-range imputation. Replace missing values with a value outside the normal range, like −1 or 999. The model can then learn to treat that special value differently. Works well with tree-based models, dangerous with linear models because the special value will bias the linear coefficient.

Mid-range imputation. Replace missing values with a value in the middle of the range, like 0 for a feature on [−1, 1]. The idea is that the imputed value won’t strongly affect predictions in either direction.

Regression imputation. Treat the missing feature as the target of a small regression problem and predict its value from the other features. This is more sophisticated and can capture relationships in the data, but it’s slow and has its own failure modes.

Indicator augmentation. Add a binary feature alongside the original that is 1 when the value was missing and 0 when it was present. This way the model can learn that “missing” is itself informative, which it often is in banking. A blank Experian score is not random; it tells you something about the director.

Read this as a decision tree for handling missingness. The right branch depends on the model family and the nature of the missingness.

A Merehaven Bank-style example. Suppose you’re building a default model for SME loans and 12% of your training examples are missing the director’s Experian score. First question: is the missingness random? You discover that almost all the missing scores belong to non-UK directors, who legitimately don’t have UK credit files. The missingness is highly informative: it tells you the director is non-UK, which is itself a meaningful credit signal. Strategy: add a binary “missing_experian” indicator and impute the original feature with the mean (or with a sentinel value for tree models). The model learns to use the indicator and to treat the imputed value as a placeholder. Performance is better than dropping the rows and better than imputing without the indicator.

The critical operational rule. Whatever imputation strategy you use at training time, you must use the same strategy at prediction time, with the same fitted parameters (e.g., the same mean computed on the training set). If you fit a new mean on incoming production data, you’ve leaked information and you’ve broken the assumption that training and serving see the same feature distributions. Modern feature stores version the imputation parameters along with the feature definition.

If asked: “How do you handle a feature that is missing for 30% of your training examples?”

How do you split data so you don’t lie to yourself?

Once your features are engineered, the next decision is how to split the data into training, validation, and test sets. This sounds like a procedural box-tick. It is not. Get it wrong and every other thing you do downstream is poisoned.

The principle is simple. You want to estimate how well your model will perform on data it has never seen before. The only honest way to estimate that is to actually measure it on data you genuinely held out from the training process. If you peek at the test data even once during training or hyperparameter tuning, you’ve contaminated the estimate. The test set is sacred. You touch it at the very end, once.

The standard split is into three subsets:

Training set. The largest. Used by the optimiser to fit the model parameters. Typically 70% to 95% of the data depending on dataset size.

Validation set (sometimes called the dev set). Used for choosing hyperparameters, comparing model variants, and deciding when to stop training. Typically 5% to 15%.

Test set. Used once, at the end, to produce an unbiased estimate of how the chosen model will perform in production. Typically 5% to 15%.

Why two holdout sets and not one? Because when you tune hyperparameters by repeatedly evaluating on a single holdout set, you are implicitly fitting your hyperparameters to that holdout set. The performance you see on that set becomes optimistic, even though you never used it for training. The fix is to have a separate test set that is never used for any decision, only for the final evaluation. The validation set is for iterating; the test set is for reporting.

The classic mistake is to use the validation set for both hyperparameter tuning and final evaluation. After tuning fifty hyperparameter combinations against the same validation set, the best one will look about 1-2% better than its true performance, just from the noise of the multiple comparisons. If you ship the model on that basis, you’ll be disappointed when you see the real production numbers. The test set protects you from this self-deception.

Read this as the discipline that separates honest ML from wishful ML. Training data fits the parameters. Validation data picks among models. Test data, used exactly once, tells you what you actually have.

For temporal data, split by time. This is critical in banking. If your data is loans originated between 2012 and 2022, do not split randomly. Random splits put 2022 loans in the training set and 2015 loans in the test set, which gives you an optimistic estimate because the model has effectively been allowed to see the future. The right split is chronological: train on 2012-2019, validate on 2020, test on 2021-2022. This mimics how the model will actually be used in production: trained on the past, deployed on the future. The performance estimate from a temporal split is much more honest than a random split, and it usually looks worse, which is the truth and you should believe it.

How big should each set be? It depends on dataset size. The old rule of thumb was 70/15/15. For modern large datasets with millions of examples, 95/2.5/2.5 is fine, because 2.5% of a million is still 25,000 examples, which is more than enough to estimate performance precisely. For small datasets with a few thousand examples, you may need to use cross-validation (which we cover later) instead of a fixed validation split, to avoid wasting examples.

Stratified splitting is a refinement worth knowing. If your classes are imbalanced (say 4% positive in a fraud detection problem), a random split might leave the validation or test set with no positive examples at all, or with very few, which makes the metrics noisy. Stratified splitting ensures that each set has roughly the same class proportions as the original data. Most ML libraries support stratified splits with one parameter.

If asked: “Why do the worked design uses three splits instead of just two?”

What does it mean for a model to underfit or overfit?

Two failure modes, opposite shapes. Knowing the difference is half the diagnostic skill of an ML engineer.

Underfitting is when the model can’t even predict the training data well. Its training error is high, and the validation error is also high. The model is too simple, or the features are too uninformative, to capture the patterns in the data. The technical term is high bias: the model is biased toward simple functions and can’t reach the true relationship.

The fixes for underfitting are: try a more flexible model (polynomial regression instead of linear, neural network instead of logistic regression, deeper tree instead of shallower), add more informative features, or remove regularisation if you’ve been using it.

Overfitting is the opposite. The model predicts the training data extremely well, even perfectly, but predicts validation data poorly. The technical term is high variance: the model’s predictions vary too much from training run to training run, depending on which specific examples ended up in the training set. The model has memorised noise in the training data, including its idiosyncrasies, mislabellings, and sampling artefacts.

The classic example: fitting a polynomial of degree 15 to 20 data points. The polynomial can wiggle through every training point exactly, achieving zero training error, but the wild oscillations between training points produce wild predictions that are nowhere near the truth. New data, which lands at slightly different positions than the training data, gets misclassified.

The fixes for overfitting are: try a simpler model, gather more training data, add regularisation, use early stopping, apply dropout (for neural networks), or do feature selection to reduce the number of inputs the model has to play with.

Read this as the universal diagnosis flowchart. Look at training error and validation error side by side. Their relative magnitudes tell you which fix to reach for.

A worked example with numbers. Imagine you’ve trained three models on the same SME default dataset:

The first model has both train and val low. It can’t even fit the training data well, suggesting the features are too few or the model too simple. The second has train high and val noticeably lower, the classic overfit signature: it has memorised some training-specific noise. The third has both train and val high and roughly equal: regularisation has prevented the overfit while still allowing the model to learn the real signal. This is the model you ship.

The bias-variance trade-off. This is the formal name for the underlying tension. As model complexity increases, bias decreases (the model can fit more shapes) but variance increases (the model is more sensitive to which specific data points it saw). The total error is the sum of bias squared plus variance plus an irreducible noise term. The sweet spot is somewhere in the middle, where the sum is minimised. Every single regularisation technique, every form of cross-validation, every hyperparameter knob is, at heart, a way of finding that sweet spot for the problem at hand.

If asked: “How do you tell whether your model is underfitting or overfitting?”

How do you penalise a model for being too clever?

Regularisation is the toolkit for preventing overfitting by adding a penalty to the training objective that discourages complex models. The idea is elegant: you tell the optimiser “minimise the training error, but also keep the model simple, and here’s how I’ll measure simple.” The optimiser then balances accuracy on the training data against simplicity, and the result is a model that’s a little less accurate on training but much more accurate on new data. Bias goes up slightly. Variance goes down a lot.

The two most common forms of regularisation are L1 and L2.

L2 regularisation adds a penalty proportional to the sum of squared parameters. For linear regression, the regularised objective is:

$$\min_{\mathbf{w}, b} \left[ C \|\mathbf{w}\|^2 + \frac{1}{N} \sum_{i=1}^{N} (f_{\mathbf{w}, b}(\mathbf{x}_i) - y_i)^2 \right]$$

where $\|\mathbf{w}\|^2 = \sum_{j=1}^{D} (w^{(j)})^2$ is the sum of squared weights and C is a hyperparameter controlling how much we care about the penalty relative to the data fit. The penalty term grows whenever any weight grows, so the optimiser is pushed toward solutions with small weights everywhere. Small weights mean the model’s predictions depend gently on each feature, which produces smoother decision boundaries that generalise better. L2 is also called ridge regularisation, and the regularised version of linear regression is called ridge regression.

L1 regularisation adds a penalty proportional to the sum of absolute values of the parameters:

$$\min_{\mathbf{w}, b} \left[ C |\mathbf{w}| + \frac{1}{N} \sum_{i=1}^{N} (f_{\mathbf{w}, b}(\mathbf{x}_i) - y_i)^2 \right]$$

where $|\mathbf{w}| = \sum_{j=1}^{D} |w^{(j)}|$ is the L1 norm, the sum of absolute values. L1 is also called lasso regularisation, and regularised linear regression with L1 is called lasso regression.

The important difference between L1 and L2. L2 shrinks all weights toward zero but rarely makes any of them exactly zero. L1, by contrast, tends to drive many weights all the way to zero. This means L1 produces sparse models: models where most features have no effect at all because their weights are exactly zero. L1 is therefore a form of automatic feature selection: the optimiser decides which features matter and silently drops the others. This is enormously useful when you have hundreds or thousands of candidate features and you want a small interpretable model.

Geometrically, the difference is that the L1 penalty has corners along the coordinate axes where weights can land exactly. The L2 penalty is a smooth sphere with no corners, so the optimal solution rarely lies on a coordinate axis. This is the technical reason for the sparsity difference, and it’s worth visualising once if you’ve never seen it.

Setting C. The hyperparameter C controls the strength of regularisation. With C = 0, you have no regularisation and the model is unconstrained (and likely overfits). As C grows, the penalty becomes stronger and the weights are pushed harder toward zero. With C very large, the model becomes too simple and underfits. The right value is somewhere in the middle, found by validation-set tuning. A typical search is over a logarithmic scale: C ∈ {0.001, 0.01, 0.1, 1, 10, 100, 1000}.

Beyond L1 and L2. Several other regularisation techniques exist and matter for specific model classes.

Dropout is a regularisation technique specific to neural networks. During training, randomly set a fraction (typically 20-50%) of neuron outputs to zero on each forward pass. The network is forced to learn redundant representations because it can’t rely on any specific neuron being available. At inference time, all neurons are used, but their outputs are scaled to compensate. Dropout was invented by Hinton’s group at Toronto in 2012 and has been a staple of deep learning ever since.

Batch normalisation normalises the activations within each layer of a neural network during training. It was introduced by Ioffe and Szegedy in 2015 and has both training and regularisation benefits, though its precise mechanism is still debated.

Data augmentation is the practice of creating new training examples by perturbing existing ones. In computer vision, you flip, rotate, crop, and colour-jitter images. In tabular data, you can add small Gaussian noise to numerical features or swap values between similar examples. Data augmentation acts as regularisation because it makes the model well-tested to small variations in inputs.

Early stopping, which we covered in Chapter 4, is also a form of regularisation. By stopping training before the validation error climbs back up, you prevent the model from memorising training-specific noise.

Read this as the menu of regularisation options. Pick one or several based on the model and the problem. For linear models, L1 or L2 is the default. For neural networks, dropout plus L2 plus early stopping is the standard recipe. For tree ensembles, regularisation comes via tree depth limits, minimum samples per leaf, and learning rate shrinkage.

If asked: “When would you use L1 regularisation instead of L2?”

How do you measure whether a classifier is actually any good?

Once you have a trained classifier, you need to evaluate it. For regression, this is mostly straightforward: compute MSE or some other loss on the test set and you’re done. For classification, things get complicated, and many production failures trace back to choosing the wrong metric.

The starting point is the confusion matrix. For a binary classifier, it’s a 2×2 table that breaks down predictions versus actuals into four cells:

A true positive is a fraud transaction correctly flagged as fraud. A false negative is a fraud transaction missed by the model. A false positive is a legitimate transaction wrongly flagged as fraud. A true negative is a legitimate transaction correctly passed through.

From the confusion matrix you can compute several metrics, each emphasising a different aspect of performance.

Accuracy is the simplest: the fraction of all predictions that are correct.

$$\text{accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

Accuracy is intuitive and works well when the classes are balanced. It is catastrophically misleading when classes are imbalanced. If your fraud rate is 0.5%, a model that predicts “not fraud” for every transaction achieves 99.5% accuracy and is utterly useless. Don’t use accuracy as the primary metric for any imbalanced problem in banking, and almost every problem in banking is imbalanced.

Precision is the fraction of predicted positives that are actually positive:

$$\text{precision} = \frac{TP}{TP + FP}$$

In words: of all the things I flagged as fraud, how many really were? High precision means few false alarms. The fraud operations team cares about precision because every false positive costs analyst time.

Recall (also called true positive rate or sensitivity) is the fraction of actual positives that the model caught:

$$\text{recall} = \frac{TP}{TP + FN}$$

In words: of all the real fraud, how much did I catch? High recall means few missed cases. Risk managers and regulators care about recall because every missed fraud is a customer loss and a compliance risk.

Precision and recall trade off. often, you can increase one only by decreasing the other. A very strict classifier (only flags transactions it’s very confident about) will have high precision but low recall: it catches only the obvious cases. A very lenient classifier (flags anything that looks slightly off) will have high recall but low precision: it catches most fraud but generates many false alarms. The right balance depends on the relative cost of the two error types.

Precision = 23/(23 + 12) = 23/35 ≈ 0.657. Recall = 23/(23 + 1) = 23/24 ≈ 0.958. Accuracy = (23 + 556)/592 ≈ 0.978.

The accuracy looks great. The recall is excellent: we catch 96% of spam. The precision is mediocre: only two-thirds of what we flag as spam actually is spam, meaning a third of our spam folder is misclassified legitimate mail. For email this is a big deal because legitimate mail being marked as spam is a serious user complaint. For fraud detection, we’d accept this trade-off because missing a fraud is much worse than annoying an analyst.

Cost-sensitive accuracy. When false positives and false negatives have different worked costs, accuracy can be replaced by a weighted version. Assign cost c_FP to each false positive and c_FN to each false negative, then compute:

$$\text{cost-sensitive accuracy} = \frac{TP + TN}{TP + TN + c_{FP} \cdot FP + c_{FN} \cdot FN}$$

Or, more usefully in practice, just compute the total cost and minimise it directly: total cost = c_FP ⋅ FP + c_FN ⋅ FN. In credit risk, the cost of approving a defaulter is the expected loss given default, which can be tens of thousands of pounds. The cost of declining a good customer is lost margin, perhaps a few hundred pounds. The asymmetry is real and the metric should reflect it.

The ROC curve and AUC. This is the most important visualisation in classifier evaluation. A classifier that outputs a continuous score (like logistic regression’s probability or an SVM’s signed distance) has a tunable decision threshold. At each threshold, you get a different confusion matrix, and therefore different precision, recall, true positive rate, and false positive rate.

Define true positive rate (same as recall) and false positive rate as:

$$TPR = \frac{TP}{TP + FN}, \quad FPR = \frac{FP}{FP + TN}$$

As you sweep the threshold from very high (almost no positive predictions) to very low (almost all predictions positive), TPR and FPR both increase from 0 to 1. The ROC curve plots TPR against FPR for all possible thresholds. The area under the ROC curve (AUC) is a single number summarising classifier quality.

Read this as the relationship between threshold, prediction volume, and the two rates. The ROC curve traces the trade-off as you sweep the threshold.

Interpretation of AUC. A perfect classifier has AUC = 1, hugging the top-left corner. A random classifier (coin flip) has AUC = 0.5, a diagonal line. A good banking model typically has AUC between 0.7 and 0.9, depending on the problem. AUC under 0.5 means your model is worse than random, usually a sign that you’ve inverted the labels somewhere.

Why AUC is popular. It’s threshold-independent: it summarises performance across the whole range of possible operating points, not just one. It’s easy to interpret: a higher number means a better model. It works for any classifier that outputs a score. And it allows direct comparison between different model families on the same problem.

Why AUC can mislead. It treats all parts of the ROC curve equally, which is rarely what you want. In fraud detection, you operate at very low false positive rates (because you can’t have analysts chasing thousands of false alarms a day), and you only care about the left edge of the ROC curve. A model with great average AUC but poor performance at low FPR is not what you want. The fix is to use partial AUC (area under the curve restricted to a specific FPR range) or to evaluate at the specific operating point you’ll use in production. Precision-recall AUC is another alternative that focuses on the positive class and is often more informative for highly imbalanced problems.

If asked: “Why is accuracy a bad metric for fraud detection?”

How do you systematically tune hyperparameters?

You’ve engineered features, split data, picked an algorithm, and you’ve identified a few hyperparameters that need to be set. SVM has C and the kernel choice. Decision tree has max depth. Logistic regression with regularisation has the regularisation strength. Gradient descent has the learning rate. None of these are learned from data; you have to set them. The systematic process for setting them is hyperparameter tuning.

Three standard approaches.

Grid search is the simplest. Define a set of candidate values for each hyperparameter, then train a model for every combination. Evaluate each on the validation set. Keep the best.

Why a logarithmic scale for C? Because the right value can vary by orders of magnitude across problems, and a linear grid would waste samples in regions where the model behaves the same. Powers of 10 (or 2) cover a wide range with few samples.

Grid search is conceptually simple and fully reproducible. It scales badly: with k hyperparameters and n values each, you have n^k combinations. For 5 hyperparameters with 7 values each, that’s 16,807 training runs. For a model that takes an hour to train, that’s about two years of compute on a single machine. Grid search is feasible only when you have few hyperparameters or few candidate values.

Random search is the smarter alternative. Instead of trying every combination on a grid, sample combinations uniformly at random from the space, and try as many as your compute budget allows. This sounds primitive but turns out to work surprisingly well, often better than grid search for the same number of trials, because most hyperparameters don’t matter equally. If only two of your five hyperparameters strongly affect the outcome, grid search wastes 95% of its trials varying the irrelevant three. Random search, by contrast, gives every hyperparameter equal coverage with each random draw, and concentrates its evaluation on the dimensions that matter.

The classic 2012 paper by Bergstra and Bengio at Université de Montréal titled “Random Search for Hyper-Parameter optimisation” demonstrated this convincingly and changed how the field thinks about hyperparameter tuning. For most practical problems, random search is the right default.

Bayesian hyperparameter optimisation is more sophisticated still. Instead of choosing the next combination randomly or on a grid, fit a probabilistic model to the results so far (typically a Gaussian process) and use it to predict which untried combinations are most likely to perform well. The next combination is chosen to maximise expected improvement. Bayesian optimisation is materially more sample-efficient than random search for problems where each trial is expensive (e.g., training a neural network for hours). Tools like Optuna, Hyperopt, and scikit-optimize implement it well.

Read this as a progression from simple to sophisticated. Use grid search for small problems where you want full coverage. Use random search for medium problems with several hyperparameters. Use Bayesian optimisation when each trial is expensive and you can’t afford to waste any.

Cross-validation is the technique you reach for when you don’t have enough data for a separate validation set. Instead of a single 70/15/15 split, you split your training data into k folds (typically 5 or 10), train k models, and validate each on a different fold. The final estimate is the average of the k validation scores.

Concretely with 5-fold cross-validation: you split your training data into 5 equal subsets {F₁, F₂, F₃, F₄, F₅}. Train model 1 on {F₂, F₃, F₄, F₅} and validate on F₁. Train model 2 on {F₁, F₃, F₄, F₅} and validate on F₂. And so on. You end up with five performance estimates and you average them. This gives you a much more stable estimate than a single 70/15 split, at the cost of doing five times the training work.

Read this as the cross-validation pattern. Each fold takes a turn as the validation set while the others train the model. The average across folds is your performance estimate. It’s the gold standard for small datasets and for hyperparameter tuning where stability matters.

Cross-validation combined with grid or random search is the most common production pattern: for each hyperparameter combination, run k-fold CV, get an averaged validation score, and pick the combination with the best average. Once you’ve picked, retrain on the full training set with the chosen hyperparameters and evaluate once on the held-out test set. This is the recipe used for nearly every well-tuned classical ML model in production In the synthetic Merehaven case and similar institutions.

If asked: “When would you use Bayesian hyperparameter optimisation instead of random search?”

Part IV: Match model structure to the evidence

Depth, ensembles, clustering and ranking solve different structural problems. Each section asks what additional capacity buys, what it hides and how it can be tested.

Use depth only when structure requires it

What is a neural network, really?

Strip away everything you think you know about neural networks. Forget the brain analogies. Forget the hype. Forget the pictures with circles and arrows. Here is what a neural network is, in one equation:

y = f_NN(x) = f₃(f₂(f₁(x)))

That’s a three-layer neural network. You take the input x, apply function f₁ to get an intermediate result, apply f₂ to that, apply f₃ to that, and the output is your prediction y. The whole thing is a composition of functions. That is the entire concept.

f_l(z) = g_l(W_lz + b_l)

Read this slowly. The function f_l takes an input vector z (which is the output of the previous layer, or the original input if l = 1), multiplies it by a matrix W_l, adds a bias vector b_l, and then applies a nonlinear function g_l (the activation function) element by element to the result. That’s one layer. The parameters W_l and b_l are learned from data. The activation function g_l is chosen by the engineer before training.

Stop and compare this to logistic regression. In logistic regression, the model is:

p̂ = σ(w^⊤x + b)

where σ is the sigmoid function. A linear combination, plus a bias, fed through a squashing function. That is exactly one layer of a neural network with the logistic function as the activation. If you put a single layer with a sigmoid activation into a neural network, you have reinvented logistic regression. If you stack two layers, you have a shallow neural network. If you stack ten, you have a deep network. The only difference between “logistic regression” and “a neural network” is how many times you apply the operation before emitting the output.

Why a matrix instead of a vector? Because each layer has multiple units, not one. In logistic regression, you compute a single number w^⊤x + b and squash it. In a neural network layer with 10 units, you want to compute 10 such numbers in parallel, each with its own weight vector and bias. Stack those 10 weight vectors as rows of a matrix W_l and stack the 10 biases as a vector b_l, and the single matrix-vector multiplication W_lz gives you all 10 linear combinations at once. The activation function is then applied element-wise to each of the 10 entries, producing a 10-dimensional output vector. That output becomes the input to the next layer, which may have a different number of units.

A concrete walkthrough with small numbers. Imagine a 2-layer network that takes a 3-dimensional input and produces a 1-dimensional output. Layer 1 has 4 units; layer 2 has 1 unit. So:

• W₁ is a 4×3 matrix (4 units, each with a weight vector of length 3)
• b₁ is a 4-dimensional vector (one bias per unit)
• W₂ is a 1×4 matrix (1 unit, weight vector of length 4)
• b₂ is a 1-dimensional vector (one bias)

Total parameters: 4 ⋅ 3 + 4 + 1 ⋅ 4 + 1 = 21. Small, but illustrative.

Suppose x = (1.0, 0.5, −0.3) and the parameters are whatever they are. Compute:

z₁ = W₁x + b₁ → a 4-dimensional vector of linear scores h₁ = g₁(z₁) → a 4-dimensional vector after activation z₂ = W₂h₁ + b₂ → a scalar y = g₂(z₂) → the output

Each layer takes in the previous layer’s output, combines it linearly with its own weights and bias, and applies its activation. The whole network is just that sequence of transformations. Nothing mystical.

Read this as the forward pass of a two-layer network. Input goes in, matrix multiplications and activations happen in sequence, output comes out. This is the fundamental shape. Every neural network in the world, including the one powering ChatGPT, is a version of this pattern with many more layers and many more units per layer.

Why do you need the nonlinearity? This is the single most important question in all of deep learning, and the answer is beautiful. Without the activation function, each layer would be a linear transformation. And the composition of two linear transformations is still a linear transformation. So a three-layer network with no nonlinearity, mathematically, collapses to a single linear transformation with a different weight matrix: W₃(W₂(W₁x + b₁) + b₂) + b₃ is just W^′x + b^′ for some W^′ and b^′. All that stacking buys you nothing. You might as well use a single linear layer, which is just linear regression.

The nonlinear activation is what breaks the collapse. When you apply a nonlinear function between layers, the composition of layers is no longer a linear function, and the network gains the ability to represent arbitrarily complex relationships. Without the nonlinearity, deep learning would be pointless. With it, deep learning can in principle approximate any continuous function (this is the universal approximation theorem, which Cybenko proved in 1989 for a single hidden layer with sigmoid activation). In practice, “can approximate any function” doesn’t mean “can learn any function from finite data,” so depth matters for different reasons we’ll get to shortly.

If asked: “Why can’t a neural network work with purely linear activation functions?”

What does a multilayer perceptron actually look like?

The multilayer perceptron (MLP) is the simplest and most general neural network architecture, and it is where you start when thinking about neural networks. An MLP consists of an input layer, one or more hidden layers, and an output layer. Every unit in each layer is connected to every unit in the previous layer. This all-to-all connection pattern is called fully connected or dense.

• Input x = (x⁽¹⁾, x⁽²⁾)
• Layer 1: each of 4 units computes y₁^(u) = g₁(w_1, u^⊤x + b_1, u) for u = 1, 2, 3, 4
• Layer 2: each of 4 units computes y₂^(u) = g₂(w_2, u^⊤y₁ + b_2, u) for u = 1, 2, 3, 4
• Layer 3: the single output unit computes y = g₃(w_3, 1^⊤y₂ + b_3, 1)

Each unit is identical in form to logistic regression: a dot product plus a bias, through an activation. The difference is that there are many of them, arranged in layers, and their outputs feed into one another. The parameters w_l, u and b_l, u are all learned jointly by gradient descent (or Adam, or whichever optimiser you pick) backpropagating through the whole thing.

Read this as a dense web of connections. The input feeds every hidden unit in layer 1. Each layer 1 unit feeds every layer 2 unit. Each layer 2 unit feeds the output. Every arrow carries a learned weight; every unit has a learned bias. The fully-connected pattern means the model has no inductive bias about which input features should combine with which, it learns whatever combinations make the loss go down.

Choosing the output activation based on the task. The activation function of the final layer determines what kind of model you have. If the final activation is linear (just the raw score, no squashing), the network is a regression model and its output can be any real number. If the final activation is sigmoid, the network is a binary classifier producing a probability. If the final activation is softmax (a generalisation of sigmoid to multiple classes), the network is a multiclass classifier producing a probability distribution over classes. Same network skeleton, different output activation, different kind of task. This is one of the lovely unifications that neural networks give you: the overall architecture is the same; only the last layer changes to match the problem.

A Merehaven Bank example: credit memo classification. Imagine you’re building a small network inside the Merehaven Credit Workbench that takes a 512-dimensional embedding of a draft credit memo (produced by an upstream language model) and classifies it into one of six memo types: new facility, facility increase, covenant waiver, annual review, watchlist, and decline. The architecture might be:

• Input: 512-dimensional embedding vector
• Hidden layer 1: 128 units, ReLU activation
• Hidden layer 2: 64 units, ReLU activation
• Output layer: 6 units, softmax activation

Total parameters: 512 ⋅ 128 + 128 + 128 ⋅ 64 + 64 + 64 ⋅ 6 + 6 ≈ 74, 000. Small by deep learning standards. Trained with cross-entropy loss and Adam optimiser. You’d hit perhaps 90%+ accuracy with decent training data. The result slots naturally into the product: each memo is auto-tagged with its predicted type, routing it to the right approval queue.

This is the kind of neural network that does real work in a bank every day. Not billions of parameters. Not months of training. Tens of thousands of parameters, trained in minutes, serving a specific focused purpose. Most production neural networks look like this, not like GPT-4.

Sigmoid / logistic function. σ(z) = 1/(1 + e^−z). Output range (0, 1). Historically the default. Now rarely used in hidden layers because of vanishing gradients (we’ll see why in a moment), but still used in output layers for binary classification.

Tanh (hyperbolic tangent). tanh (z) = (e^z − e^−z)/(e^z + e^−z). Output range (−1, 1). Centred at zero, which is nicer for optimisation than sigmoid. Used occasionally, particularly in RNNs.

ReLU (rectified linear unit). ReLU(z) = max (0, z). Output range [0, ∞). Negative inputs map to zero; positive inputs pass through unchanged. This is the activation function that unlocked modern deep learning. Popular because it’s trivial to compute, doesn’t saturate for positive inputs (so gradients flow nicely), and empirically trains faster and reaches better solutions than the smoother alternatives.

Variants of ReLU exist, including Leaky ReLU (allows a small negative gradient for negative inputs to avoid “dead” neurons), GELU (a smooth approximation to ReLU used in modern transformers), and Swish (another smooth variant). For most practical purposes, ReLU is the default. If you are unsure what activation to use in a hidden layer, use ReLU.

If asked: “What’s the difference between a multilayer perceptron and logistic regression?”

Why was deep learning hard for twenty years?

Here’s a historical puzzle worth sitting with. The basic neural network architecture has been around since at least the 1960s. Backpropagation, the algorithm for training it, was popularised in 1986 by Rumelhart, Hinton, and Williams. By the late 1980s, people knew what a neural network was and how to compute its gradient. And then, for about twenty years, nothing much happened. Neural networks were a backwater of machine learning. Support vector machines dominated classification. Random forests and gradient boosting dominated tabular data. Neural networks were viewed as finicky, hard to train, and not worth the effort. The people who kept working on them were regarded, mostly, as stubborn eccentrics.

What happened? The short answer is: the vanishing gradient problem prevented deep networks from being trained, and nobody knew how to get around it reliably until around 2010. The long answer is the rest of this section.

The vanishing gradient problem. Backpropagation computes the gradient of the loss with respect to each parameter by applying the chain rule backwards through the network. At each layer, the gradient flowing through the layer gets multiplied by the derivative of the layer’s activation function. For sigmoid and tanh, this derivative is always less than 1 (it maxes out at 0.25 for sigmoid and 1 for tanh at the origin, and goes to zero at the extremes). Multiply n numbers less than 1 together and you get a number that decreases exponentially with n. For a deep network, the gradient at the early (input-side) layers is vanishingly small by the time it finishes propagating backwards. The early layers barely update. Training stalls.

The exploding gradient problem. The other side of the same coin. If the activation derivatives are sometimes greater than 1, or if the weight magnitudes are large, the product of gradients can grow exponentially instead of shrinking. The early layers receive enormous gradient updates, blow up, and training diverges. Exploding gradients were easier to fix than vanishing ones, usually with gradient clipping (just cap the gradient norm at some threshold before applying the update) and regularisation.

Read this as the vanishing gradient pipeline. Each layer shrinks the gradient a little more. By the time backprop reaches the first layer, there’s essentially nothing left to work with, and those layers never learn.

The breakthroughs that fixed it. Between 2006 and 2012, a series of clever ideas accumulated to make deep networks trainable. None of them was a single magic bullet; they combined into an ecosystem of techniques.

ReLU activation. The derivative of ReLU is 1 for positive inputs and 0 for negative inputs. For positive inputs, the gradient flows through unchanged, so the product of derivatives doesn’t shrink as you go backwards. You get some dead neurons (those with permanently negative input), but on average the gradients reach the early layers intact. ReLU was used in neural networks by Glorot, Bordes, and Bengio in 2011, and by Krizhevsky, Sutskever, and Hinton in their landmark AlexNet paper in 2012, which won the ImageNet competition and kicked off the modern deep learning era.

Better initialisation. Random initialisation sounds trivial, but getting it right matters enormously. Xavier Glorot’s initialisation (2010) and Kaiming He’s initialisation (2015) carefully choose the scale of initial weights based on the number of inputs and outputs per layer, so that gradients neither explode nor vanish at the start of training. Modern frameworks use these as the default.

Skip connections and residual networks. Kaiming He and peers at Microsoft Research introduced ResNet in 2015, which allows gradients to flow around layers via direct shortcut connections. A residual block computes y = x + F(x) where F is the usual layer transformation. The “+x” shortcut means that even if F(x) has a vanishing gradient, the gradient can still flow through the identity path. This single trick made networks with 100+ layers trainable for the first time, and residual connections are now everywhere in deep learning.

Better optimisers. Adam and its descendants (which we met in Chapter 4) handle the per-parameter learning rate adaptation that makes gradient descent reliable on the complex loss surfaces of deep networks.

Batch normalisation. Ioffe and Szegedy (2015) normalise the activations within each layer during training, which keeps the gradient distributions stable across layers and depths.

GPUs. The hardware that made training a 100-million-parameter network in hours (rather than weeks) feasible. Without the compute, none of the algorithmic improvements would have mattered because nobody could run experiments fast enough to iterate on them.

Put it all together and you get the state of deep learning in 2016: trainable networks with hundreds of layers, billion-parameter models, and the beginnings of the foundation model era. The twenty-year dark age ended because many small problems got fixed at once, and the sum of fixes was greater than any individual one.

If asked: “Why did deep learning take off around 2012 and not earlier?”

How does a neural network see an image?

Here is a real problem. You want to classify 28×28 pixel images of handwritten digits (the classic MNIST dataset) with a neural network. Each image has 784 pixels. If you flatten it into a vector and feed it to a plain MLP, the first layer with 100 hidden units would have 784 ⋅ 100 = 78, 400 weights. That’s manageable. But now imagine the images are 1000×1000 pixels (a modest photo). Each image has 1,000,000 pixels. The first layer with 100 hidden units would have 100 million weights. That’s infeasible.

And even if you could afford it, an MLP on images treats every pixel as an independent input with no spatial relationship to its neighbours, which throws away the single most important fact about images: nearby pixels are correlated, edges and textures are local patterns, and the meaning of a patch depends on the arrangement of its pixels, not just their individual values.

The insight that led to convolutional neural networks came from thinking about how an image works. Pixels near each other tend to carry related information: all the pixels of a patch of sky are blue; all the pixels of a patch of grass are green. The interesting things happen at the boundaries: edges between objects, textures within objects, shapes formed by arrangements of edges. A good model should exploit locality: it should recognise patterns in small patches and combine them into larger patterns.

The second insight is parameter sharing. A pattern like “vertical edge” has the same mathematical structure regardless of where in the image it appears. If you train a small detector to recognise a vertical edge at position (5,5), the same detector should work at position (100,100). Instead of learning a separate detector for every position, learn one detector and slide it across the whole image.

These two insights, locality and parameter sharing, are the entire conceptual basis for convolutional neural networks. They were worked out by Yann LeCun and peers at Bell Labs in the late 1980s, applied to handwritten digit recognition for US Postal Service address parsing, and remained a specialist tool until AlexNet dominated ImageNet in 2012.

A convolution, intuitively. A small matrix of weights (called a filter or kernel) slides across the input image, computing a weighted sum of the pixels under it at each position. The result at each position is a single number indicating how much the image patch at that position resembles the filter. Do this for every position and you get a new matrix (the feature map) that tells you where the filter “activates” across the image. Train the weights of the filter to make the feature map useful for downstream prediction, and you have a convolutional layer.

$$P = \begin{pmatrix} 0 & 1 & 0 \\ 1 & 1 & 1 \\ 0 & 1 & 0 \end{pmatrix}, \quad F = \begin{pmatrix} 0 & 2 & 3 \\ 2 & 4 & 1 \\ 0 & 3 & 0 \end{pmatrix}$$

The convolution (technically, it’s a correlation, but everyone calls it convolution in the neural network world) is the element-wise product of P and F summed:

0 ⋅ 0 + 1 ⋅ 2 + 0 ⋅ 3 + 1 ⋅ 2 + 1 ⋅ 4 + 1 ⋅ 1 + 0 ⋅ 0 + 1 ⋅ 3 + 0 ⋅ 0 = 0 + 2 + 0 + 2 + 4 + 1 + 0 + 3 + 0 = 12

So at this position, the convolution outputs 12. The filter was designed to detect a cross pattern, and the patch happens to contain a cross, so the output is high. If the patch contained a T instead:

$$P' = \begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 0 \\ 0 & 1 & 0 \end{pmatrix}$$

The convolution with the same filter gives:

1 ⋅ 0 + 1 ⋅ 2 + 1 ⋅ 3 + 0 ⋅ 2 + 1 ⋅ 4 + 0 ⋅ 1 + 0 ⋅ 0 + 1 ⋅ 3 + 0 ⋅ 0 = 0 + 2 + 3 + 0 + 4 + 0 + 0 + 3 + 0 = 12

Sliding the filter. To process the whole image, you slide the filter across every position, compute the convolution at each position, and collect the results into an output matrix. This is the feature map for that filter. If you have many filters in a layer (say, 64 of them), each filter slides across the same input and produces its own feature map, giving you 64 feature maps at the output. These are stacked into a volume with 64 channels. The next layer then treats this volume as its input and slides its own filters across it.

Read this as the basic CNN operation. One filter produces one feature map. Many filters per layer produce a stack of feature maps. Multiple layers of this, and you end up with progressively higher-level representations: edges in early layers, textures in middle layers, object parts in late layers, whole objects near the end.

Stride and padding are two hyperparameters that control how the filter slides. Stride is how many pixels the filter moves at each step. Stride 1 gives the finest-grained feature map; stride 2 halves the size in each dimension, which reduces parameters and compute. Padding is adding zeros around the border of the input so the filter can be centred on the edge pixels and you don’t lose size at every layer. Stride and padding together determine the output dimensions of a convolution layer. You adjust them based on how much the network should shrink its spatial dimensions as it goes deeper.

Pooling is the other key ingredient. After a convolution layer, you often apply a pooling layer, which aggregates small neighbourhoods of the feature map into single values. The most common is max pooling: for each 2×2 patch, take the maximum value. Max pooling with stride 2 halves the spatial dimensions and makes the representation slightly translation-invariant (small shifts in the input don’t change the output much). Pooling has no learnable parameters; it’s a fixed operator with a moving window.

The typical CNN architecture alternates convolution and pooling layers. In production at both Merehaven Bank (a managed training service) and Merehaven Bank (a managed training service), CNNs for document understanding are usually pretrained ResNet or ViT models fine-tuned on internal cheque or document images via transfer learning, gradually reducing spatial dimensions while increasing the number of channels, and finally feeds the flattened result into one or more fully connected layers for the final prediction.

Concrete walkthrough: a document understanding CNN In the synthetic Merehaven case. Imagine you’re building a model to classify scanned cheque images as “valid,” “suspicious,” or “needs review.” The input is a 256×256 grayscale image. A plausible architecture:

• Conv layer 1: 32 filters of size 3×3, stride 1, padding 1, ReLU activation. Output: 32 × 256 × 256.
• Max pool 2×2, stride 2. Output: 32 × 128 × 128.
• Conv layer 2: 64 filters of size 3×3, stride 1, padding 1, ReLU. Output: 64 × 128 × 128.
• Max pool 2×2, stride 2. Output: 64 × 64 × 64.
• Conv layer 3: 128 filters of 3×3, stride 1, padding 1, ReLU. Output: 128 × 64 × 64.
• Max pool 2×2, stride 2. Output: 128 × 32 × 32.
• Flatten: 131,072-dimensional vector.
• Fully connected layer: 256 units, ReLU.
• Output layer: 3 units, softmax.

Total parameters: roughly 34 million, mostly in the final fully connected layer. Trained with cross-entropy loss and Adam. In production at a bank, you would also carefully engineer the preprocessing (cropping, deskewing, contrast normalisation) and the post-processing (confidence thresholds, routing logic), and you would rigorously validate the model against known failure modes. The CNN is a component in a larger system, not the whole system.

If asked: “Why use a CNN instead of a fully connected network for image classification?”

How does a neural network read a sentence?

Images have spatial structure. Text has temporal structure: words come in a specific order, and that order matters. “Dog bites man” and “man bites dog” have the same words but very different meanings. A model that processes text needs to respect the order. Plain MLPs and CNNs do not; they treat their input as an unordered bag of values, possibly with spatial arrangement but not with sequence.

The architecture designed for sequences is the recurrent neural network or RNN. The idea is beautifully simple: instead of processing all of the input at once, process it one element at a time, maintaining an internal state vector that carries information from the past into the present. At each time step, the network takes the current input and the previous state, combines them, and produces a new state (and optionally an output). At the next time step, the network takes the next input and the new state, and so on. The state acts as a memory.

The math. Let x_t be the input at time step t and h_t be the hidden state. A single layer of a vanilla RNN computes:

h_t = g(Wx_t + Uh_t − 1 + b)

Read this carefully. The new hidden state h_t depends on the current input (through W), the previous hidden state (through U), and a bias, all squashed through a nonlinearity g (typically tanh). The weight matrices W and U are shared across all time steps: the same W processes every input, and the same U processes every previous state. This is the recurrent structure. The network has memory, and the memory evolves step by step as the sequence is consumed.

At each time step the network can also produce an output y_t = g_o(Vh_t + c), which depends only on the current hidden state. Whether you use the output at every step or only at the end depends on the task. For sequence labelling (tag each word with its part of speech), you use the output at every step. For sequence classification (is this email spam or not spam?), you use only the output at the final step.

Read this as the unrolled view of an RNN. Time flows left to right. At each step, the network combines the current input with the previous hidden state to produce a new hidden state. The same weights are reused at every step. The final hidden state (or outputs at intermediate steps) feed into the prediction layer.

Training an RNN via backpropagation through time. To train an RNN, you apply backpropagation to the unrolled network, which looks like a very deep feedforward network with shared weights at every layer. This is called backpropagation through time (BPTT). The same weight appears in many layers of the unrolled network, and the gradient with respect to that weight is the sum of the contributions from every layer. The technique was described by Werbos in 1990 and remains the standard training approach for RNNs.

The long-term dependency problem. Here is where the trouble starts. Vanilla RNNs suffer from the same vanishing gradient problem as deep feedforward networks, but even more acutely. The “depth” of the unrolled network equals the length of the sequence. For a 50-word sentence, you’re backpropagating through 50 layers. The gradients vanish before they reach the early words. As a consequence, the model cannot learn dependencies that span long distances. If a pronoun at the end of a paragraph refers to a noun at the beginning, the RNN cannot learn to connect them, because the signal has decayed long before it gets back to the noun.

This was the RNN’s central failure. People knew about it in the 1990s, and several researchers worked on fixing it, and the fix that eventually worked was the long short-term memory (LSTM) network, invented by Sepp Hochreiter and Jürgen Schmidhuber in 1997. Their paper was initially viewed as niche and sat relatively unread for a decade before deep learning’s rise turned it into one of the most cited papers in machine learning. Schmidhuber has some opinions about this, which he shares freely whenever the opportunity arises.

The LSTM idea. Instead of a single hidden state that gets updated at every time step, an LSTM maintains a cell state that can be selectively read, written, or cleared by a set of gates. Each gate is a small neural network (essentially a logistic regression) that takes the current input and previous hidden state and produces a value between 0 and 1, indicating how much of some information to keep, let through, or overwrite. The architecture has three gates: the forget gate (how much of the cell state to erase), the input gate (how much of the new input to write), and the output gate (how much of the cell state to expose as the hidden state).

The magic of gates is that when a gate is at or near 1, information passes through unchanged, effectively via an identity function, and the gradient of the identity function is constant. Gradients can flow across many time steps without vanishing, and the network can learn long-term dependencies that would be invisible to a vanilla RNN.

h̃_l, u^t = g₁(w_l, ux^t + u_l, uh_l^t − 1 + b_l, u)

Γ_l, u^t = g₂(m_l, ux^t + o_l, uh_l^t − 1 + a_l, u)

h_l, u^t = Γ_l, u^th̃_l, u^t + (1 − Γ_l, u^t)h_l^t − 1

Don’t memorise the equations. Notice the structure. h̃ is a candidate new value for the memory cell. Γ is the gate, a sigmoid-squashed linear combination that lies in (0, 1). The update rule h = Γh̃ + (1 − Γ)h_prev interpolates between the new candidate and the old value, controlled by the gate. When the gate is 0, the cell retains its previous value exactly. When it’s 1, the cell takes the new candidate. Between these extremes, the cell blends the two. The gate itself is learned, which means the network decides, based on the current input and state, how much to update the memory.

Read this as the GRU update step. At every time step, the network computes both a candidate new value and a gate, and blends them with the previous state. The gate learns to decide when to update and when to preserve. This is the mechanism that lets gated RNNs handle long dependencies.

Banking applications of RNNs. Sequence models in banking are real. Fraud detection systems process sequences of transactions and flag suspicious patterns. AML systems look at customer behaviour over time to detect structured deposits or layering schemes. Time-series forecasting for liquidity planning and stress testing uses sequence models. Speech-to-text in customer contact centres uses sequence models. In natural language processing tasks within banks, RNNs and LSTMs were the standard until around 2019, when transformers (coming in a later chapter) largely replaced them for most tasks. RNNs still see use where the sequences are short, the compute budget is tight, or the deployment constraints rule out transformer-class models.

The sequence-to-sequence pattern. A particularly important application is translating one sequence to another. You use one RNN (called the encoder) to process the input sequence and produce a fixed-size representation. You use a second RNN (called the decoder) to generate the output sequence from that representation. This encoder-decoder architecture, introduced by Sutskever, Vinyals, and Le in 2014, was the basis for the first neural machine translation systems, and it’s the ancestor of the transformer architecture used by every modern language model. The fingerprints of sequence-to-sequence are visible in the Merehaven Credit Workbench today: the Copilot takes a conversation and generates a draft memo, which is exactly a sequence-to-sequence problem.

If asked: “Why do we need LSTMs or GRUs instead of vanilla RNNs?”

Match specialised tools to data structure

How do you fit a smooth curve through messy data without choosing a polynomial?

Linear regression draws a straight line. Polynomial regression draws a curve, but you have to choose the degree of the polynomial in advance, and getting it wrong leaves you either underfit (degree too low) or overfit with wild oscillations (degree too high). Neither feels right when you have messy worked data and you just want a smooth function that follows the trend wherever the data goes.

That’s kernel regression. Given training data {(x_i, y_i)}_i = 1^N and a new query x, the prediction is a weighted average of the training labels:

$$f(\mathbf{x}) = \frac{1}{N} \sum_{i=1}^N w_i y_i, \quad \text{where} \quad w_i = \frac{N \cdot k\left(\frac{\mathbf{x}_i - \mathbf{x}}{b}\right)}{\sum_{l=1}^N k\left(\frac{\mathbf{x}_l - \mathbf{x}}{b}\right)}$$

The function k(⋅) is the kernel. It plays the role of a similarity function. When x_i is close to x, the kernel value is large, so the weight w_i is large, and the corresponding training label contributes heavily to the prediction. When x_i is far from x, the kernel value is small and the weight is small. The denominator normalises the weights so they sum to one.

The most popular kernel is the Gaussian kernel:

$$k(z) = \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{z^2}{2}\right)$$

This is the bell curve. Inputs near zero produce values near $1/\sqrt{2\pi}$. Inputs far from zero in either direction produce values that decay smoothly toward zero. When you plug z = (x_i − x)/b into the kernel, you’re measuring how close x_i is to x in units of b, which is called the bandwidth.

The bandwidth b is the only hyperparameter and it controls everything. Small b means the kernel is sharply peaked: only the very nearest training points contribute meaningfully, and the regression curve hugs the training data tightly (overfit). Large b means the kernel is wide and flat: many training points contribute, and the regression curve is very smooth, possibly too smooth (underfit). Choosing b is a validation-set tuning problem, exactly like the hyperparameters we met in Chapter 5.

Concrete walkthrough. Suppose you have three training points for a 1D regression: (1, 10), (3, 20), (5, 30). You want the prediction at x = 2 with bandwidth b = 1.

Compute the kernel values: - $k((1 - 2)/1) = k(-1) = \frac{1}{\sqrt{2\pi}} \exp(-0.5) \approx 0.242$ - $k((3 - 2)/1) = k(1) = \frac{1}{\sqrt{2\pi}} \exp(-0.5) \approx 0.242$ - $k((5 - 2)/1) = k(3) = \frac{1}{\sqrt{2\pi}} \exp(-4.5) \approx 0.004$

Sum: 0.242 + 0.242 + 0.004 ≈ 0.488.

Normalised weights: w₁ = 0.242/0.488 ≈ 0.496, w₂ ≈ 0.496, w₃ ≈ 0.008.

Prediction: f(2) = 0.496 ⋅ 10 + 0.496 ⋅ 20 + 0.008 ⋅ 30 ≈ 14.96.

Almost exactly halfway between the labels of points 1 and 3, which is what you’d expect since the query is halfway between x = 1 and x = 3 and the third point is too far away to matter much. If you used b = 0.1 instead, the kernel for x = 5 would be effectively zero, and even the x = 3 contribution would be small, so the prediction would be dominated by x = 1 and would be near 10. If you used b = 5, all three points would contribute roughly equally and the prediction would be near 20. The bandwidth controls the smoothness of the resulting function.

Where this fits in banking. Kernel regression is rarely the production model of choice in modern banks because it requires storing all training data (like k-NN), which has memory and latency costs at scale, and because tree ensembles usually do better on complex tabular problems. But it’s a beautiful baseline for one-dimensional or low-dimensional smoothing problems. If you have a year of weekly unemployment data and you want a smooth trend line for stress testing, kernel regression with a Gaussian kernel will give you exactly that. It’s also the conceptual ancestor of many modern non-parametric methods, including Gaussian processes, which are used in Bayesian hyperparameter optimisation.

If asked: “What’s the difference between kernel regression and k-NN regression?”

How do you handle more than two classes?

Many of the algorithms from Chapter 3 are fundamentally binary. SVM separates two classes with a hyperplane. Logistic regression outputs a probability for one class. They were not designed for problems where the answer might be one of fifteen categories. Yet most real classification problems have more than two classes. A merchant categorisation model has hundreds of categories. A document routing model has dozens. A loan disposition model has at least four: approved, declined, refer, conditional approval.

Three approaches handle multiclass classification, each suited to a different situation.

Approach one: use an algorithm with native multiclass support. Some algorithms generalise naturally to many classes. Decision trees simply count the proportion of each class at each leaf and predict the majority. Logistic regression generalises to the softmax regression model, which we’ll see again in neural networks: the output layer has one unit per class, each computing a linear score, and the softmax function turns the scores into a probability distribution over classes. k-NN simply looks at the k nearest neighbours and takes the majority vote across however many classes appear. Each of these requires no special tricks for multiclass; you just train the model and it works.

The softmax function deserves a moment. Given C scores z₁, z₂, …, z_C (one per class), softmax computes:

$$\text{softmax}(z_c) = \frac{e^{z_c}}{\sum_{c'=1}^C e^{z_{c'}}}$$

The numerator exponentiates the score for class c. The denominator sums the exponentiated scores for all classes, normalising the result. The output is a vector of C values that are all positive and sum to one, a valid probability distribution. Softmax is to multiclass what sigmoid is to binary: the squashing function that turns raw scores into probabilities.

Approach two: one-versus-rest (OvR). For algorithms that are stuck being binary, like the original SVM, you train C separate binary classifiers, one per class. Each binary classifier is trained to distinguish “class c” from “everything else.” At prediction time, you run all C classifiers on the new input and pick the class whose classifier is most confident. For logistic regression you compare predicted probabilities. For SVM you compare the signed distances from the decision boundary.