Define the discipline at its interfaces

The definition that anchors this edition:

Machine learning system design is the complex, multistep process of designing, implementing, and maintaining ML-based systems. It combines techniques from ML, software engineering, project management, product management, and leadership.

Maintenance belongs in the system definition because data, policy, dependencies and ownership change after release.

Why intuition stops scaling

Each interface and feedback loop is a failure surface. The design dossier records its owner, observable signals and recovery route.

System design makes failure modes explicit and recoverable.

Design enough to learn safely

The roots are in classical software architecture. Building complicated software has consistently required someone to decide the structure: what are the high-level components, how do they communicate, what happens when one fails? That someone was traditionally called a software architect, an experienced engineer who had seen many systems and understood the patterns of success and failure.

This approach is associated with the Waterfall methodology and the Big Design Upfront paradigm. analyse everything, document everything, then write code. It was reliable but slow. In a world of rapid changes, projects could lose their purpose before they were finished.

The reaction was agile software development: ship fast, iterate, learn from production. Less documentation, more code, more user feedback. Agile gave teams speed but sometimes sacrificed coherence. The pendulum swung from “plan everything” to “plan nothing,” and many teams landed somewhere in the middle.

ML system design inherits this tension. You cannot fully plan an ML system upfront because you do not know if your model will work until you train it. But you cannot just start coding either, because the interactions between data pipelines, model training, serving infrastructure, and business logic are too complex to discover through trial and error.

The sweet spot, as the design team argue, is this: design the system’s skeleton thoroughly, but expect the flesh to change. Write a design document that covers the problem, the data, the metrics, the validation strategy, the integration plan, and the monitoring approach. Then iterate.

Five disciplines in one design review

ML system design requires simultaneous attention to statistical learning, software, product, delivery and operating ownership.

Treat it as wearing five hats, sometimes all at once.

Hat 1: The ML Researcher. You need to know which algorithms exist, what their strengths and weaknesses are, and when gradient-boosted trees will outperform a neural network (more often than you think). You need to read papers, understand loss functions, and have intuition about what 0.02 improvement in AUC actually means for the business.

Hat 2: The software engineer. The model runs inside tested, versioned and observable software with explicit interfaces and recovery behaviour.

Hat 3: The Project Manager. ML projects have timelines, budgets, and stakeholders. You need to estimate how long things will take (hint: longer than you think), communicate progress, and know when to cut scope.

Hat 4: The Product Manager. What problem are we solving? For whom? How will we know if we have succeeded? These are product questions, and they must be answered before a single line of training code is written.

Hat 5: The accountable lead. The lead aligns the team, resolves trade-offs and ensures that unresolved risk has an owner.

The system lifecycle as four evidence loops

The book follows the life cycle of an ML system, from birth through adolescence to maturity.

Part 1: Preparations (Chapters 1-4) covers what to do before you write any ML code. This is where you define the problem, research existing solutions, and create the design document, a central artefact in the entire process.

Part 2: Early Stage (Chapters 5-8) dives into the technical foundations: loss functions and metrics, datasets, validation schemas, and baseline solutions. This is where you set up the scientific infrastructure that will tell you whether your system is actually working.

Part 3: Intermediate Steps (Chapters 9-12) is where the system grows muscles. Error analysis, training pipelines, feature engineering, and A/B testing. This is where research gives way to implementation, and you spend more time writing code than reading papers.

Part 4: Integration and Growth (Chapters 13-16) is where the system gets its wisdom. Integration with the broader software ecosystem, monitoring for drift and degradation, inference optimisation, and the often-neglected topic of ownership and maintenance.

The four parts trace the lifecycle from problem framing through evidence, learning operations and accountable release.

When do these principles actually help?

There is a cynical reading of system design: it is bureaucracy dressed up as engineering. More documents, more meetings, more overhead, less actual building.

Design review has a cost. It earns that cost when complexity, team size, decision consequence or continuity risk makes undocumented assumptions expensive.

Scenario 1: The system is complex. If you are building a demand forecasting model for a grocery chain with 10,000 SKUs across 500 stores, with weekly seasonality, promotional effects, supply chain constraints, and a procurement team that needs predictions three days in advance, you cannot hold all the requirements in your head. You need a document.

Scenario 2: The team is large or distributed. If more than three people are working on the system, the communication overhead grows quadratically. A design document becomes the shared source of truth that prevents the conversation from restarting every Monday morning.

Scenario 3: The system must survive its creators. People leave. They get promoted, change teams, or join startups. If the system’s design lives only in someone’s head, it dies when they walk out the door. This is the continuity risk, a morbid but useful concept we will revisit in Chapter 16.

The cost of not designing

Count the failures:

1. The model was biased (uneven error distribution), but no owner had planned the relevant disparity analysis.
2. Executives needed manual overrides, but the system had no override mechanism.
3. The model solved a subtly different problem than what the business needed.
4. Data was drifting, but there was no monitoring.
5. The infrastructure could not handle peak load, but no load testing had been done.
6. Other teams were not consulted, leading to API mismatches.

The case shows is that the synthetic mobility operator employed plenty of people who knew about bias analysis, monitoring, load testing, and stakeholder alignment. They knew these things in the abstract. What they lacked was a systematic framework that forced them to ask the right questions at the right time.

That framework is ML system design.

A thought experiment before we move on

Consider you are handed a new project tomorrow. Your company wants to build a recommendation system for an e-commerce platform. The VP of Product has already chosen the model architecture (“I read about transformers; let’s use those”). The data team says they have “plenty of data.” The deadline is three months.

Before you write a single line of code, write down the ten questions you would ask. Not technical questions about attention mechanisms or embedding dimensions. Questions about the system: What problem are we solving? How will we know it is solved? What data do we actually have? What happens when the model is wrong? Who approves deployment? What does the fallback look like? How do we monitor? Who owns this after launch?

If you can write those ten questions confidently, you are already thinking like an ML system designer.

If you cannot, do not worry. That is what this edition is for.

Dossier check

Before leaving this chapter, record the assumption that matters most to design the system, not the estimator, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

Diagnose before prescribing

Consider you are a doctor. A patient walks in and says, “I need antibiotics.” Do you hand them a prescription? Of course not. You ask questions. Where does it hurt? How long has it been going on? What have you tried? Do you have allergies? The patient knows their symptoms; they do not know their diagnosis. Your job is to diagnose before you prescribe.

The same principle applies to ML system design. Stakeholders know their symptoms (“our sales team wastes time on bad leads”). They do not know the diagnosis (“we have no systematic way to score lead quality because our CRM does not capture behavioural signals”). And they definitely do not know the prescription (“a gradient-boosted classifier on engagement features, served via batch inference, with a fallback to a rule-based heuristic”).

The problem space is explored with “what?” and “why?” questions. There is a popular technique called Five Whys that recommends stacking your “why?” questions to dig to the root cause. For the Merehaven product engineer’s project, it might look like this:

1. Why do we need to prioritise leads? Because the sales team spends too much time on leads that do not convert.
2. Why do they spend time on bad leads? Because they have no way to distinguish good leads from bad ones.
3. Why can’t they distinguish them? Because the signals that indicate lead quality (engagement, company size, industry fit) are scattered across different systems.
4. Why are signals scattered? Because the CRM was set up for contact management, not lead intelligence.
5. Why hasn’t this been fixed? Because no owner held the cross-system data integration until now.

The questioning changes the problem boundary. We started with “build a lead scoring model” and arrived at “we need cross-system data integration.” The ML model is still part of the solution, but it is no longer the whole solution. Without fixing the data integration problem, the model would be scoring leads based on incomplete information, producing predictions that are technically correct but practically useless.

The solution space is explored with “how?” questions. How should we frame this as an ML problem? How should we collect data? How should we validate? These questions are important, but they are premature until the problem space is thoroughly mapped.

Find the decision behind the request

Consider three worked ML problems and watch how they transform when you dig into the problem space.

Problem 1: Fraud Detection. A fintech company says, “We need an ML model to detect fraudulent transactions.” Sounds clear. But what kind of fraud? Credit card fraud, identity theft, account takeover, money laundering? Each requires different data, different features, and different response times. A credit card fraud model needs to make a decision in milliseconds (before the transaction clears). An anti-money-laundering model might have days to analyse patterns. The “same” problem is actually five different problems with five different architectures.

Problem 2: Dynamic pricing. A pricing request may optimise short-term margin, inventory movement, fairness constraints or long-term retention. Those objectives can conflict, so the business owner must declare the priority and guardrails before modelling.

Problem 3: Recommendation System. “We want to recommend products to users.” Recommend based on what? Past purchases (collaborative filtering), product similarity (content-based), or something the user has not seen but might like (exploration)? What counts as a good recommendation: one the user clicks on, one the user buys, or one the user buys and does not return? The choice of success metric reshapes the entire system.

In each case, the surface-level problem statement masks a web of assumptions, tradeoffs, and implicit decisions that must be made explicit before engineering begins.

Translate a decision into a learning task

Once the problem space is thoroughly explored, you face a translation challenge: converting a messy, human-scale business problem into something an ML algorithm can optimise.

Treat an ML model as a bounded prediction service. It is a universal machine that can answer any properly formulated question. Your job is to find the right question and teach users how to ask it. Before mimicking the oracle’s behaviour with algorithms, you need to define what the oracle should do.

Some translations seem straightforward:

For the fraud problem, we want the oracle to label a user as a fraudster as soon as possible, ideally before they act. This sounds like classification.

For the pricing model, we want the oracle to estimate how much a customer is willing to pay without driving them to a competitor (short-term) or making them think “this company is too greedy” (long-term). This sounds like regression, but with constraints.

For the recommendation system, we want the oracle to suggest items that make the customer happy with the service. This sounds like ranking.

But notice the qualifications. Each “straightforward” translation required caveats. The fraud model needs temporal constraints. The pricing model has competing objectives. The recommendation system depends on how you define “happy.” A good ML system design acknowledges these complications upfront rather than discovering them in production.

A useful heuristic, popularized by Andrew Ng: imagine how a human expert would solve the problem. For medical diagnosis, a doctor examines symptoms, cross-references with patient history, and makes a judgment. The ML system should approximate this process. But the heuristic has limits. There are problems where machines outperform humans (detecting subtle patterns in high-dimensional data) and problems where human judgment is irreplaceable (ethical tradeoffs, novel situations).

Risks, limitations, and the constraints that must be named

Every ML system operates under constraints that the problem space exploration must surface. These constraints fall into three categories.

Technical limitations are the constraints imposed by the current state of technology. Some problems are simply too hard for existing algorithms. If your problem requires understanding sarcasm in multilingual text, you are pushing the boundaries of what NLP can do reliably. If your problem requires real-time inference on edge devices with 256MB of RAM, you are constrained in model size and complexity.

Data limitations are often the binding constraint. You might have the perfect algorithm for your problem, but if you do not have enough labelled data, or if your data is biased, or if the distribution of your data shifts over time, The model can fail. We will devote an entire chapter to datasets, but for now, the key insight is this: the best algorithm trained on bad data can lose to a mediocre algorithm trained on good data.

The cost of a mistake: Not all errors are created equal

The following contrast that separates ML system designers from ML researchers.

You build a spam filter. It makes two kinds of errors. A false positive labels a legitimate email as spam. The user misses it. A false negative lets a spam email through to the inbox. The user sees it and deletes it. Which error is worse?

Most people say false positives, because missing a legitimate email could mean missing a job offer or an urgent message. False negatives are annoying but recoverable.

Now change the context. You build a cancer detection model. A false positive tells a healthy patient they might have cancer. They undergo additional tests, experience anxiety, and eventually learn they are fine. A false negative tells a cancer patient they are healthy. They go home, the cancer progresses, and by the time it is detected, treatment options are limited.

Which error is worse now?

The answer depends on the domain, the stakes, and the downstream actions. A false positive in cancer detection causes stress and additional testing costs. A false negative can be fatal. But a false positive in criminal sentencing (imprisoning an innocent person) is arguably worse than a false negative (releasing a guilty person), because the justice system is designed to err on the side of caution.

The cost of a mistake is not a property of the model. It is a property of the system and the domain in which it operates.

This has direct implications for system design:

1. Asymmetric error costs require asymmetric loss functions. If false negatives are 10x more costly than false positives, your loss function should penalize them 10x more heavily. We will cover this in Chapter 5.

2. High-cost domains require human-in-the-loop. If a wrong prediction can kill someone, the model should flag cases for human review rather than making autonomous decisions.

Worked specimen: governed case prioritisation

Return to the Merehaven product engineer and do the problem exploration properly.

Step 1: Stakeholder Interviews (Problem Space)

The Merehaven product engineer goes back to the Merehaven sales lead and asks deeper questions.

What does “prioritise leads” mean? the Merehaven sales lead says the team has 500 new leads per week and can only contact 100. They need to pick the best 100.

What makes a lead “good”? the Merehaven sales lead pauses. “One that becomes a customer.” the Merehaven product engineer probes further: one that becomes a customer this quarter, or one that eventually becomes a customer? The answer matters because leads that convert quickly look different from leads that need months of nurturing.

What data do you have on past leads? the Merehaven sales lead says the CRM has contact information, industry, company size, and whether the lead eventually converted. the Merehaven product engineer asks about behavioural data: email opens, website visits, content downloads. the Merehaven sales lead says “some of that is in our marketing automation platform, but it’s not linked to the CRM.”

What happens when the model is wrong? If a good lead is ranked low, the sales team misses an opportunity. If a bad lead is ranked high, a salesperson wastes an hour on a call that goes nowhere.

Who will use the output? Individual salespeople, who will see a ranked list in their CRM dashboard every morning.

Step 2: Problem Definition

The Merehaven product engineer now has enough context to define the problem precisely: “Given a set of leads with demographic features (industry, company size, role) and behavioural features (email opens, page visits, content downloads), predict the probability that each lead will convert to a paying customer within 90 days. Present the top 100 leads daily in the CRM dashboard.”

Step 3: Risk Assessment

• Data risk: behavioural features are in a separate system. Integration is non-trivial.
• labelling risk: Historical conversion labels may be noisy (did a lead not convert because it was bad, or because no salesperson called them?).
• Selection bias: The sales team already makes subjective decisions about who to call. The model will be trained on data generated by these decisions, inheriting their biases.
• organisational risk: Salespeople may resist using the model if it contradicts their gut feelings.

Step 4: Cost of Mistake Analysis

A false negative (good lead ranked low) costs a potential sale. If the average deal is $50,000 and the model causes the team to miss 5 deals per quarter, the cost is $250,000.

A false positive (bad lead ranked high) costs a salesperson’s time. If a salesperson spends an hour per lead and their fully loaded cost is $100/hour, ranking 20 bad leads per day costs $2,000/day, or roughly $130,000 per quarter.

Both costs are significant, but false negatives are about 2x more expensive. This suggests the model should err on the side of including marginal leads rather than excluding them.

The formal framework: Problem space vs. solution space

Formalise what the Merehaven product engineer did.

The problem space answers: What are we building and why?

• Why does the problem exist?
• Who is affected?
• What are the constraints (budget, timeline, data, regulations)?
• What does success look like?
• What are the risks of failure?
• What is the cost of different types of mistakes?

The solution space answers: How will we build it?

• What ML approach fits (classification, regression, ranking)?
• What data do we need?
• What features should we engineer?
• What model architecture should we use?
• How will we validate?
• How will we deploy and monitor?

This does not mean you need perfect information. You not have perfect information. It means you need to have asked the right questions, even if some answers are “we don’t know yet.” Knowing what you do not know is itself valuable information that shapes how you design the system.

The questions that change the design

After his stakeholder deep-dive, the Merehaven product engineer returned to the Merehaven sales lead with a very different conversation than his first one.

“the Merehaven sales lead, before I build anything, I need to flag three things. First, the behavioural data in our marketing platform needs to be linked to the CRM. Without it, the model will be working with one hand tied behind its back. That’s a data engineering project that will take two to three weeks. Second, our historical conversion labels are noisy because leads that were not contacted look identical to leads that were contacted and rejected. We’ll need to filter the training data carefully. Third, I want to run this as a pilot with one sales team before rolling it out company-wide. If the salespeople don’t trust the rankings, the model is worthless regardless of its accuracy.”

The no-model option

There is a failure mode that trips up even experienced ML engineers, and it is worth naming explicitly: solving a problem that does not exist.

Sometimes the problem has already been solved adequately by a non-ML approach. A transparent eligibility rule may capture enough value to make an ML route unnecessary. Sometimes the problem is real but unsolvable with current data. If you do not have the features that distinguish good leads from bad ones, no algorithm will help. Sometimes the problem is real and solvable, but the organisation is not ready. If the sales team will ignore the model’s output, building it is a waste of resources.

The design document, which we will build in Chapter 4, exists in part to catch these cases. A well-constructed design document makes it possible to kill a project early, which is one of the most valuable outcomes in engineering. Every dollar not spent on a doomed project is a dollar available for a promising one.

Dossier check

Before leaving this chapter, record the assumption that matters most to write the decision problem first, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

Search for structural analogues

The design implication is that problems in different domains often share the same mathematical structure. A recommendation system for an e-commerce platform and a recommendation system for a music streaming service are solving the same underlying problem: given a user’s history, predict which items they will engage with next. The items are different (products vs. songs), the features are different (price vs. tempo), but the core architecture, a two-stage pipeline with candidate generation and ranking, is remarkably similar.

This principle extends further than you might expect. Fraud detection and medical diagnosis share a common structure: both are rare-event classification problems with extreme class imbalance, high costs for false negatives, and a need for explainability. Demand forecasting and weather prediction both involve time-series models with strong seasonal patterns and external covariates. Image search and code search both use embedding-based retrieval with contrastive learning.

The practical implication: before you design your ML system, spend a few days studying how similar problems have been solved in other domains. The sources are abundant.

Academic papers are the primary source. ArXiv is free and searchable. Google Scholar lets you trace citation chains forward and backward. If you find a foundational paper, look at what cited it to see how the field evolved.

Papers with Code (paperswithcode.com) is particularly valuable because it links papers to implementations. Theory without code is often useless; code without theory is often dangerous. Papers with Code gives you both.

Blog posts from ML teams at major tech companies (Google AI Blog, Meta AI, Netflix Tech Blog, Spotify Engineering) describe production systems with a level of practical detail that academic papers rarely provide. They tell you what actually worked, including the parts that failed.

benchmark competitions are underrated as a source of architectural inspiration. The winning solutions in benchmark competitions are widely evaluated against real data and real evaluation metrics. They often contain tricks and heuristics that not make it into papers.

Build or buy: The most expensive question in ML engineering

Every ML system design process eventually arrives at a fork in the road. Do you build the component yourself, or do you buy (or adopt) an existing solution?

The intuition of most engineers is to build. Building is fun. Building is creative. Building gives you complete control. Building is also, often, more expensive than you think.

Consider a concrete example. Your team needs a vector database for a semantic search feature. You have three options:

Option A: Build from scratch. Write your own approximate nearest neighbor search using HNSW (Hierarchical Navigable Small World) graphs. You will understand every line of code. You will also spend three months on indexing, querying, persistence, and concurrency, reinventing solutions that existing libraries have already optimised over years.

Option B: Use an open-source solution. Deploy Faiss (large technology and industrial organisations’s vector search library), Qdrant, Milvus, or Weaviate. You get production-quality ANN search immediately. You lose some customization flexibility and take on a dependency.

Option C: Buy a managed service. Use Pinecone, the selected cloud platform OpenSearch, or Google a managed training platform Matching Engine. You get everything from Option B, plus managed infrastructure, automatic scaling, and 24/7 support. You pay a recurring fee and are locked into a vendor.

The right choice depends on your constraints:

The authors’ rule of thumb: buy when you can, build when you must. Build only when the component is so core to your competitive advantage that owning it provides a strategic edge, or when no existing solution meets your specific constraints (like the on-device face verification example).

There is a middle path that experienced teams use: start with a bought or open-source solution, then replace it with a custom one if and when the limitations become binding. This gives you the speed of buying with the option to build later. The key is to design your system so that components can be swapped. If your vector database is accessed through a clean abstraction layer, replacing Pinecone with a custom solution later is a refactor, not a rewrite.

Problem decomposition: separate the interfaces

The principle is the same one that makes all complex engineering possible: abstraction. Build low-level blocks that encapsulate complexity, treat them as black boxes, use them to build higher-level blocks, repeat.

Consider a product recommendation system. At the highest level, it is a single function: recommend(user) → [items]. But inside that function, there is an entire ecosystem.

The candidate generation stage retrieves a broad set of potentially relevant items (hundreds to thousands) using fast, low-precision methods. The ranking stage scores these candidates using a richer, slower model that considers more features. The post-processing stage applies business rules (do not recommend items the user already bought), diversity constraints (do not show five blue shirts in a row), and generates explanations (“Because you bought X”).

Each of these components is itself decomposable. The scoring model has a feature pipeline. The feature pipeline has data sources. The data sources have ETL jobs. And so on, down to individual database queries.

The art of decomposition is knowing where to draw the boundaries. Good boundaries have three properties:

1. Clear interfaces. Each component takes well-defined inputs and produces well-defined outputs. The candidate generator takes a user ID and returns a list of item IDs with scores. The ranker takes a user ID and a list of candidate item IDs and returns a ranked list.

2. Independent development. Different teams or engineers can work on different components simultaneously. The team improving the ranker does not need to coordinate daily with the team improving the candidate generator, as long as both respect the interface contract.

3. Independent testing. You can evaluate each component’s quality in isolation. Is the candidate generator retrieving relevant items? Is the ranker ordering them correctly? Is the post-processor applying business rules faithfully?

Poor decomposition creates hidden dependencies. If a ranker assumes a recency rule that the candidate generator later changes, performance can degrade without a model-code defect. The failure sits at the interface contract.

Choosing the right degree of innovation: The bounded innovation

There is a spectrum of innovation in ML system design, and both extremes are dangerous.

On one end, you have excessive conservatism: using only techniques you have personally used before on similar problems. This is safe but limiting. You miss opportunities to leverage advances in the field, and you build systems that are obsolete on arrival.

At the other extreme, a research-stage technique may lack operating evidence, debugging knowledge and a recovery history. Use it only in an isolated experiment until the evidence supports a wider role.

The sweet spot depends on two factors: the cost of failure and the time horizon.

If the cost of failure is low (a recommendation system for a low-traffic blog) and the time horizon is long (you have six months), you can afford to experiment with newer techniques. If the cost of failure is high (a medical diagnosis system) and the time horizon is short (you have six weeks), stick to proven approaches.

A useful mental model: place every technique on an innovation spectrum, from “proven in production at multiple companies” to “proposed in a recent paper, not tested at scale.” For each component of your system, choose a technique appropriate to the component’s criticality.

A practical example: You are building a search engine for a photo stock company. The candidate retrieval stage should be widely evaluated (CLIP embeddings with ANN search). The ranking model can be established (a fine-tuned transformer). The query expansion mechanism can be emerging (LLM-based query rewriting). But nothing in the production pipeline should be research-stage.

A concrete example: Solving the solution space for a taxi fare estimator

The following traces the solution space exploration for one of the examples from Chapter 2: a dynamic pricing system for a synthetic mobility operator.

Step 1: Find similar problems. Airline ticket pricing, hotel room pricing, surge pricing at Uber/Lyft, electricity spot pricing. All involve dynamic pricing with supply-demand dynamics.

Step 2: Study existing solutions. Airline pricing uses a combination of dynamic programming and demand forecasting. Uber’s surge pricing uses a real-time supply-demand model. Hotel pricing uses time-series forecasting with event-based adjustments.

Step 3: Identify shared patterns. All these systems share a common structure: (1) forecast demand, (2) assess supply, (3) compute a price that optimizes an objective function (revenue, utilization, customer satisfaction).

Step 4: Choose an approach. Given our constraints (real-time pricing, multiple vehicle types, geographical variation), we choose a two-stage approach: a demand forecasting model (time-series regression) feeding into a pricing optimisation model (constrained optimisation).

Step 5: Decompose. The demand model needs features: time of day, day of week, weather, events, historical demand. The pricing model needs constraints: minimum fare, maximum surge multiplier, competitor pricing bounds.

Step 6: Assess innovation level. Demand forecasting with gradient-boosted trees is widely evaluated. Constrained pricing optimisation with linear programming is established. Using reinforcement learning for pricing (as some papers suggest) is emerging and not appropriate for the first version.

The failure mode: Over-engineering the first version

We have all seen it. An engineer reads three papers about graph neural networks, gets excited, and proposes using them for a problem that could be solved with logistic regression. The GNN solution is more interesting to build, more impressive on a resume, and completely unnecessary.

Early designs face the greatest risk of over-engineering because the constraints are still uncertain. Prefer the simplest candidate that satisfies the declared decision and service requirements, then add complexity against measured evidence.

We will formalise this principle in Chapter 8, where we discuss baselines. The idea is simple: consistently start with the simplest possible approach. If it works well enough, ship it. If it does not, you now have a calibration point that tells you exactly how much improvement the more complex approach needs to deliver.

Dossier check

Before leaving this chapter, record the assumption that matters most to search before you invent, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

Four myths that keep teams from writing design documents

Before we discuss what a design document should contain, we need to dismantle the myths that prevent teams from writing them.

Myth #1: Design documents work only for big companies.

Small teams have little redundancy for architectural mistakes. A concise design dossier is therefore a continuity and risk-control instrument, not a proxy for organisation size.

Myth #2: Design documents are efficient only for complex projects.

Even simple projects benefit from explicit problem statements and success criteria. The design document for a simple project might be one page. That one page forces you to articulate what you are building, why, and how you will know it works. If you cannot fill one page, you do not understand the project.

Myth #3: Every design document should be based on a template.

Templates are useful starting points, but they can become straitjackets. A template designed for a recommender system will have sections that make no sense for a demand forecasting system. The structure of your design document should follow from the structure of your problem, not from a one-size-fits-all form. Use templates as checklists, not as constraints.

Myth #4: Every design document should lead to a deployed system.

A design review may correctly conclude that a project should not be built. Discovering an unsound problem, missing evidence or a weak business case before implementation preserves budget for stronger opportunities.

Goals and antigoals: What you are building, and what you are NOT building

Every design document starts with two lists: goals and antigoals.

Goals are the specific outcomes the system must achieve. They should be measurable, time-bound, and connected to business value. “Improve recommendation quality” is not a goal. “Increase click-through rate on recommended products by 15% within 3 months of launch” is a goal.

Antigoals are the things you are explicitly choosing NOT to do. They are just as important as goals, because they prevent scope creep, a source of hidden error of ML projects.

Treat antigoals as fences. Without them, every stakeholder meeting becomes an opportunity for someone to suggest “while we’re at it, we should also…” And each “while we’re at it” adds weeks of work, additional failure modes, and integration complexity.

The following synthetic example. You are designing a demand forecasting system for a grocery chain (the Merehaven Market case study that runs throughout the book).

Goals: - Predict daily sales per SKU per store for the next 7 days - Reduce out-of-stock events by 20% compared to the current heuristic-based system - Reduce overstock waste by 15% - Integrate with the existing procurement system via batch API

Antigoals: - We will NOT build a real-time pricing optimisation system (that is a separate project) - We will NOT predict demand for new products with zero sales history (cold start is out of scope for v1) - We will NOT build a custom dashboard; we will use the existing reporting infrastructure - We will NOT optimise for individual customer demand; we optimise at the store level

The antigoals are doing real work here. Without the first antigoal, someone would inevitably suggest adding dynamic pricing “since we are already predicting demand.” Without the second, the team would spend weeks on the cold-start problem instead of shipping a system that works for 95% of products. Without the third, a front-end engineer would be pulled into the project for three months. Without the fourth, the feature engineering would become enormously more complex.

The anatomy of a design document

A good design document has a shape, and that shape roughly follows the sequence of decisions you need to make. A useful dossier follows the sequence of decisions that another reviewer must reconstruct.

I. Problem Definition

This section answers four questions:

Origin: Where did this problem come from? Is it a pain point surfaced by users, a strategic initiative from leadership, or a competitive response?

Relevance and reasons: Why does this problem matter right now? What is the cost of not solving it? What is the opportunity cost of solving it instead of something else?

Previous work: Has anyone attempted to solve this before, inside or outside the company? What happened? If a previous attempt failed, why?

Other issues and risks: What could go wrong? What are the dependencies? What are the assumptions that, if violated, would invalidate the design?

II. Goals and Antigoals (as described above)

III. Metrics and Losses (Chapter 5 will cover this in depth)

IV. Dataset (Chapter 6)

V. Baseline Solution (Chapter 8)

VI. Validation Schema (Chapter 7)

VII. Training Pipeline (Chapter 10)

VIII. Feature Engineering (Chapter 11)

IX. Measuring and Reporting (Chapter 12)

X. Integration (Chapter 13)

XI. Monitoring (Chapter 14)

XII. Serving and Inference (Chapter 15)

The structure shows that this structure mirrors the book’s own structure. Each chapter adds a new section to the design document. By the end of the book, you will have seen two complete design documents, one for Merehaven Market (demand forecasting) and one for Merehaven Images (image search), built section by section.

Reviewing the design document: The art of constructive destruction

Writing the design document is half the battle. Reviewing it is the other half.

The review process serves three purposes:

1. Error detection. Reviewers catch mistakes, gaps, and implicit assumptions that the author missed.
2. Knowledge sharing. Reviewers learn about the project, creating redundancy and reducing the continuity risk.
3. Alignment. The review process ensures that all stakeholders agree on the direction before significant resources are committed.

The authors have strong opinions about how reviews should work.

Invite reviewers with diverse backgrounds. An ML engineer will catch modelling mistakes. A software engineer will catch infrastructure problems. A product manager will catch misalignment with business goals. A data engineer will catch data pipeline issues. A review panel drawn from one discipline creates a blind spot.

Beware the “looks good to me” response. When a reviewer says “LGTM” without specific feedback, they either did not read the document or did not feel empowered to criticize it. Both are problems. If you are the author, follow up: “Thanks for reviewing. Were there any sections where you had concerns or where the reasoning was unclear?” Force specificity.

Require alternatives with critique. A useful review names the failing assumption, proposes a feasible alternative and compares the consequences of each route.

The following synthetic example from the book that illustrates both good and bad reviews.

The original design proposes using a single model for all product categories in the demand forecasting system.

Bad review: “This looks fine, but maybe consider using separate models per category?”

This is a suggestion with no reasoning. It does not help the author make a decision.

Good review: “A single model for all categories assumes that the relationship between features and demand is consistent across categories. This may not hold for perishable goods (where weather is a strong predictor) vs. electronics (where promotional events dominate). I suggest running an experiment: train one unified model and one model per top-5 category, and compare performance on a holdout set. If the per-category models win by more than 5% MAE, the additional maintenance cost of separate models is justified.”

This review identifies the assumption, explains why it might be wrong, proposes a concrete experiment to test it, and specifies a decision criterion. This is the kind of review that makes systems better.

A design doc is a living thing

A design document is versioned and remains open to evidence-led revision.

A design document is not a contract. It is a living record of your understanding at a given point in time. As you build the system, you learn things that were unknowable during the design phase. The model performs worse on weekends (you did not anticipate weekend demand patterns). The data pipeline has a 3-hour lag that breaks your real-time serving requirements. A new paper proposes a technique that would simplify your feature engineering.

When these discoveries happen, you update the design document. You add a section explaining what changed and why. The document becomes a chronological record of choices, evidence, reversals and rationale that a future maintainer can reconstruct.

The analogy is to biological evolution. A design document at v1 is a blueprint. At v5, it is a fossil record. Both are valuable. The blueprint tells you the architect’s intent. The fossil record tells you what actually happened.

Concrete walkthrough: Writing the problem definition for Merehaven Market

The following drafts the opening section of the design document for Merehaven Market, the fictional grocery chain case study.

I. Problem Definition

i. Origin

Merehaven Market operates 500 grocery stores across 12 regions. Each store carries approximately 10,000 SKUs. The procurement department currently uses a heuristic-based system (moving averages with manual adjustments) to forecast demand and place orders. This system was built 8 years ago and has not been significantly updated.

ii. Relevance and Reasons

iii. Previous Work

A data science team attempted to build a forecasting model 18 months ago using Prophet. The model outperformed the heuristic on aggregate metrics but failed on individual SKUs with high variance (e.g., promotional items). The project was abandoned when the lead data scientist left the company. The code exists in a Jupyter notebook on a shared drive but has not been maintained.

iv. Other Issues and Risks

• Data quality: Historical sales data has gaps during system migrations (Q2 2021). Some stores report with a 1-day lag.
• organisational: The procurement team has developed heuristics over years and may resist model-based recommendations.
• Technical: The existing procurement system accepts batch uploads only (no real-time API). Integration requires a daily batch job.
• Compliance: Food safety regulations require maintaining minimum stock levels for certain categories regardless of demand predictions.

A useful problem definition explains why the work exists, what has already been tried, what evidence is missing and which failure modes could invalidate the case.

The thought experiment: Would you kill this project?

After drafting the problem, goals and exclusions, pause and decide whether the evidence still supports building the project. A stop decision is a valid design outcome.

If the honest answer is “I’m not sure,” that is fine. Document your uncertainty. What would change your answer to “yes”? What would change it to “no”? These conditions become your project’s kill criteria, conditions that, if met, should trigger a project review and possible cancellation.

Kill criteria for Merehaven Market might include: - If the ML model cannot beat the heuristic baseline by at least 10% on MAE after 8 weeks of development, revisit the approach. - If the data quality issues in Q2 2021 are more extensive than initially assessed, the training data may be insufficient. - If the procurement team refuses to adopt model-based recommendations after the pilot, the project has no path to impact.

These criteria are not signs of weakness. They are signs of maturity. An engineer who can articulate the conditions under which their own project should be canceled is an engineer who understands the difference between building something and building something useful.

The iteration principle: Designing things twice

Draft at least two materially different designs before selection. The second exposes assumptions that are difficult to see while the first remains implicit.

In practice, this means:

1. Write a first draft of the design document quickly, in a day or two. It will have gaps, wrong assumptions, and overly optimistic timelines.
2. Review it yourself. Mark the parts you are uncertain about.
3. Share it with 2-3 trusted peers. Ask them to critique, not to approve.
4. Incorporate feedback. Some sections will be rewritten entirely.
5. Share the revised version with a broader group, including stakeholders.
6. Iterate until the document stabilizes, which usually happens after 2-3 rounds.

People who prefer building over thinking may find this process irritating. “Come on, first you suggest writing docs instead of code, and now you suggest rewriting the doc multiple times?” The authors’ response: it makes little sense to iterate once you are no longer receiving new information. But designing things twice is often a fair tradeoff between agility and preparedness.

And the cost comparison is stark. Rewriting a design document takes hours. Rewriting a deployed system takes months.

Dossier check

Before leaving this chapter, record the assumption that matters most to freeze the design dossier, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

Loss functions: The language your model speaks

A loss function (also called an objective function or cost function) is a mathematical function that quantifies how wrong the model’s predictions are. During training, the model adjusts its parameters to minimize this function. The loss function therefore defines the model’s entire worldview: what it pays attention to, what it ignores, and what tradeoffs it makes.

Changing the loss while holding data, features and architecture fixed can materially change fitted behaviour.

A synthetic demand-forecasting specimen makes the distinction concrete. You are predicting how many units of each product a grocery store will sell tomorrow.

Mean Squared Error (MSE): The model is penalized proportionally to the square of its error. Predict 100 when the truth is 110? Penalty of 100. Predict 100 when the truth is 120? Penalty of 400. Squaring the error means the model is terrified of large mistakes. It will sacrifice accuracy on easy predictions to avoid blowing up on hard ones.

MSE = (1/n) * Σ(predicted - actual)²

Mean Absolute Error (MAE): The model is penalized proportionally to the absolute value of its error. Predict 100 when the truth is 110? Penalty of 10. Predict 100 when the truth is 120? Penalty of 20. Linear penalty. The model treats all errors as equally important per unit.

MAE = (1/n) * Σ|predicted - actual|

Quantile Loss: Instead of predicting the expected value, the model predicts a specific quantile of the distribution. The 90th percentile quantile loss penalizes under-prediction much more heavily than over-prediction. Why? Because in demand forecasting, under-predicting means running out of stock (lost sales, unhappy customers), while over-predicting means excess inventory (waste, but manageable).

The synthetic Merehaven design compares lower, median and upper quantile forecasts. Each quantile answers a different planning question, so the chosen levels follow from stock-out, waste and capacity costs rather than a universal list.

A worked example: How an objective changes behaviour

Consider you are forecasting demand for milk at a single store. Here are the actual sales for the past 7 days: [100, 95, 110, 105, 200, 98, 102]. Day 5 was a holiday, causing a spike to 200.

Model A (trained with MSE) predicts: [101, 98, 108, 103, 180, 100, 104]. It is trying hard to capture the spike, sacrificing accuracy on normal days to reduce the squared error on Day 5.

Model B (trained with MAE) predicts: [100, 96, 109, 104, 130, 99, 103]. It partially captures the spike but does not distort normal-day predictions to do so, because the linear penalty does not amplify the spike as much.

Model C (trained with 90th percentile quantile loss) predicts: [115, 110, 125, 118, 210, 112, 117]. It consistently over-predicts, because under-prediction is penalized 9x more heavily than over-prediction at the 90th percentile.

Which model is “best”? That depends entirely on the business context. If stockouts cost 10x more than overstock per unit, Model C is best. If stockouts and overstock cost the same, Model B is best. If a single catastrophic stockout (like running out of milk on a holiday) could damage the store’s reputation, Model A might be best despite its higher average error.

The model did not change. The data did not change. Only the loss function changed, and the system’s behaviour changed entirely.

Loss tricks for deep learning models

When working with deep neural networks, loss function design becomes even more consequential, because deep models have enough capacity to exploit any weakness in the loss function.

Label smoothing replaces hard class targets with a declared distribution, such as 0.1 and 0.9 in a binary learning specimen. It can reduce overconfident fitting under some objectives, but the smoothing value is a tuned design choice and does not substitute for calibration tests.

Treat label smoothing as teaching humility. You are telling the model, “Even the training data might be wrong sometimes, so do not bet your entire probability mass on any single answer.”

Focal loss addresses class imbalance in a different way than class weighting. Instead of upweighting the minority class, focal loss downweights the easy examples. If the model already predicts a negative example correctly with 99% confidence, focal loss reduces the gradient from that example nearly to zero. The model’s attention is redirected toward the hard examples, the ones near the decision boundary.

The analogy: a soccer coach who spends practice time on the drills the team is worst at, not the ones they have already mastered.

Contrastive loss and triplet loss are used when the goal is to learn embeddings (vector representations) rather than class labels. The model is shown pairs (positive: same class, negative: different class) or triplets (anchor, positive, negative) and learns to pull similar items together and push dissimilar items apart in embedding space.

This is how models like CLIP learn to associate images with text. The loss function says: “Make the embedding of a photo of a cat closer to the embedding of the text ‘a photo of a cat’ than to any other text.” The result is a shared embedding space where semantically similar items, regardless of modality, are nearby.

Metrics: What you measure after training

The following distinction that trips up junior engineers: a loss function is what the model optimizes during training. A metric is what you use to evaluate the model after training. Every loss function can be used as a metric, but not every metric can be used as a loss function (because loss functions must be differentiable).

The gap between loss and metric is where most model evaluation failures occur.

Consider the credit risk example from the opening. The model optimised cross-entropy loss (good for classification). The team evaluated with accuracy (bad for imbalanced data). The loss and the metric were measuring different things.

The following taxonomy of metrics for the two most common problem types:

For classification:

• Accuracy: Fraction of correct predictions. Useless for imbalanced data.
• Precision: Of all items predicted positive, how many were actually positive? High precision = few false positives.
• Recall: Of all actually positive items, how many did we catch? High recall = few false negatives.
• F1 Score: Harmonic mean of precision and recall. A compromise.
• AUC-ROC: Area under the Receiver Operating Characteristic curve. Measures the model’s ability to rank positive examples above negative ones, across all possible thresholds.

For regression:

• MAE: Average absolute error. well-tested to outliers.
• RMSE: Square root of average squared error. Sensitive to large errors.
• MAPE: Mean Absolute Percentage Error. Scale-independent.
• WAPE: Weighted Absolute Percentage Error. Better than MAPE when actual values can be zero.
• Quantile metrics: Error at specific quantiles, capturing the distribution of errors.

The choice of metric is not a technical decision. It is a business decision expressed in technical language. When you choose precision over recall, you are saying “I would rather miss some positive cases than flag false positives.” When you choose RMSE over MAE, you are saying “large errors are disproportionately worse than small ones.”

Consistency metrics: When small changes cause big problems

There is a category of metrics that most textbooks ignore but that matters enormously in production: consistency metrics.

Consider a search engine. You search for “best Italian restaurant” and get a list of results. Now you search for “best Italian restaurants” (adding an “s”). If the results change completely, the search engine feels broken, even if both result sets are individually reasonable.

Consistency metrics measure how stable the model’s output is under small perturbations of the input. For search engines, this is query perturbation sensitivity. For recommendation systems, it is temporal consistency (do the recommendations change wildly between sessions?). For demand forecasting, it is prediction stability (does tomorrow’s forecast change significantly just because we received one more day of data?).

Consistency is not captured by traditional metrics like accuracy or MAE. A model can have excellent accuracy and terrible consistency if it is overfit to noise in the data.

The hierarchy of metrics: From model to business

The authors introduce a framework that is immediately useful in practice: the hierarchy of metrics.

At the bottom are offline metrics: AUC, MAE, F1 score, the numbers you compute on a holdout set during development. These are precise, fast to compute, and completely controlled. They are also the furthest from what the business actually cares about.

In the middle are online metrics: click-through rate, conversion rate, session duration. These are measured during A/B tests in production. They are closer to business impact but noisy and slow to measure (you need statistical significance, which takes days or weeks).

At the top are business metrics: revenue, profit, customer lifetime value, churn rate. These are what the CFO tracks. They are the ultimate measure of success but are influenced by dozens of factors beyond your model.

The design task is connecting the levels. If your offline AUC improves by 0.02, what does that mean for online CTR? If CTR improves by 5%, what does that mean for revenue? These connections are rarely linear and often require historical data from previous A/B tests to calibrate.

Proxy metrics are the bridge between offline and online. They are offline metrics that have been shown (through past experiments) to correlate with online metrics. If you have run 10 A/B tests and found that a 1% improvement in offline NDCG consistently produces a 0.5% improvement in online CTR, then NDCG becomes a reliable proxy metric that you can use to make go/no-go decisions without running a full A/B test.

The failure mode: Goodhart’s Law in action

“When a measure becomes a target, it ceases to be a good measure.” This is Goodhart’s Law, and it is a common failure mode in metric selection.

A recommendation system optimised for click-through rate will learn to recommend clickbait: sensational titles, misleading thumbnails, anything that gets a click. The click rate goes up, but customer satisfaction goes down, and eventually churn increases. The metric improved while the system got worse.

Percentage errors become unstable or undefined when actual demand is near zero and can give slow-moving items disproportionate influence. A forecasting review should therefore inspect absolute, scaled and decision-cost measures rather than trusting one aggregate percentage score.

A defence against Goodhart’s Law is a hierarchy of measures. Do not optimise one proxy in isolation; track decision and outcome measures, and accept a weaker model metric when the predeclared outcome evidence supports the trade-off.

Dossier check

Before leaving this chapter, record the assumption that matters most to align loss, metric and consequence, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

Where does data come from?

You can use any source to find data for your dataset. The availability and quality of these sources depend on your work environment, your company’s history, and the specific goals of your ML system. But the sources fall into predictable categories.

Internal transactional data is the richest source for most companies. Every click, purchase, search query, and support ticket generates data. The advantage is volume and relevance; this is your actual user behaviour. The disadvantage is messiness: missing fields, schema changes, duplicate records, and the accumulated detritus of years of engineering decisions made by people who have since left.

Internal operational data includes logs, metrics, and system events. Server response times, error rates, deployment timestamps. This data is often overlooked for ML purposes but can be invaluable. If your model’s predictions degrade every time the backend is updated, server logs will tell you.

External public data includes government datasets, academic datasets, and open data initiatives. Census data, weather data, economic indicators. Useful for enriching internal data with contextual features, but rarely sufficient on its own.

Third-party commercial data is purchased from vendors. Market research, consumer behaviour panels, satellite imagery. Expensive, often licensing-restricted, and requires careful validation because you do not control the collection methodology.

Synthetic data is generated artificially, either by augmenting existing data (rotating images, paraphrasing text) or by using generative models to create new examples. Useful for addressing class imbalance or privacy constraints, but dangerous if the synthetic distribution does not match the real distribution.

Human-labelled data is created by annotators who manually classify, tag, or rate examples. a strong causal design when its assumptions hold for supervised learning, but expensive, slow, and subject to annotator disagreement. We will return to labelling in detail shortly.

Cooking the dataset: From raw ingredients to a meal

Raw records require declared extraction, filtering, transformation and labelling before they can support a training claim.

ETL (Extract, Transform, Load) is the process of pulling data from its source, transforming it into a usable format, and loading it into your data store. Think of it as grocery shopping: you go to different stores (sources), buy ingredients (extract), wash and chop them (transform), and put them in your fridge (load).

ETL sounds mechanical, but it is where many ML projects encounter their first serious problems. The sales data is in a SQL database with a schema that changed three times in the last two years. The clickstream data is in a Kafka stream that requires parsing JSON blobs with inconsistent field names. The weather data is in CSV files downloaded monthly from a government website that occasionally changes its column headers without warning.

Filtering removes data that should not be in your training set. Duplicate records, bot traffic, test accounts, data from time periods with known quality issues. Filtering decisions are consequential because they change the distribution of your training data, and the model will learn from whatever distribution you give it.

A subtle but common filtering mistake: removing outliers without understanding why they are outliers. If your demand forecasting data shows a spike to 10x normal volume on a particular day, and you remove it as an outlier, you have also removed the information about how demand behaves during promotional events. The model will not learn to handle promotions.

Feature engineering transforms raw data into the inputs your model will actually use. We will devote an entire chapter to this (Chapter 11), but at the dataset stage, the key decisions are: what granularity to aggregate at (hourly? daily?), what time windows to use (last 7 days? last 30 days?), and what derived features to compute (ratios, differences, rolling averages).

labelling is the process of attaching ground truth to your data. For classification, this means assigning a class label. For regression, it means associating a numerical target. For ranking, it means establishing an ordering.

labelling is where the most expensive mistakes happen, and they happen quietly.

The labeling trap: How your labels lie to you

Consider a fraud detection system. Your labels come from chargebacks: if a customer files a chargeback, the transaction is labelled “fraud.” Otherwise, it is labelled “legitimate.” This seems straightforward.

But chargebacks are a lagging indicator. They arrive days or weeks after the transaction. Some fraudulent transactions do not result in chargebacks because the victim does not notice or does not bother to report. Some legitimate transactions result in chargebacks because the customer changed their mind or did not recognize the merchant name.

Your labels are noisy, biased, and delayed. And your model, which treats these labels as ground truth, will inherit every one of these flaws.

There are several approaches to label acquisition, each with its own tradeoffs.

Explicit labels come from direct user actions: clicks, purchases, ratings, thumbs-up/down. They are cheap and abundant but biased toward the items the system already shows (you cannot observe a click on a product that was not recommended).

Implicit labels are inferred from behaviour: dwell time, scroll depth, return visits. They are even cheaper than explicit labels but noisier. Did the user stare at the product page for 30 seconds because they were interested, or because they were confused?

Expert labels come from domain specialists: doctors labelling X-rays, lawyers labelling contracts, linguists labelling sentiment. They are expensive and slow but high-quality. The bottleneck is usually expert availability.

Crowdsourced labels come from platforms like Amazon Mechanical Turk or Scale AI. They are faster than expert labels but lower quality. The standard approach is to have multiple annotators label each example and use majority voting or inter-annotator agreement to assess quality.

Programmatic labels come from heuristic rules or weak supervision frameworks like Snorkel. They scale infinitely but require careful calibration against a small set of ground-truth labels.

Active learning is a hybrid approach where the model identifies the examples it is most uncertain about and sends them for human labelling. This is maximally efficient with the labelling budget but requires an existing model (creating a chicken-and-egg problem).

Data vs. metadata: A distinction that saves debugging hours

The following distinction that seems pedantic until it saves you from a production incident: the difference between data and metadata.

Data is what the model uses for prediction: feature values, target variables, input sequences.

Metadata is information about the data: when it was collected, from which source, what version of the collection pipeline produced it, which annotator labelled it, what the distribution of labels is.

Metadata is not used by the model, but it is used by the engineer debugging the model. When your model’s accuracy drops by 3% on Tuesday morning, metadata helps you answer: Did the data source change? Did the labelling pipeline fail? Did a schema migration break a feature?

The practical recommendation: store metadata alongside data from the beginning. It costs almost nothing to add a timestamp, source ID, and pipeline version to every record. The alternative, trying to reconstruct this information after a production incident, costs enormously.

How much data is enough?

This is one of a common questions in ML, and the honest answer is unsatisfying: it depends.

It depends on the complexity of the problem. A linear model separating two Gaussian blobs might need 100 examples. A vision transformer classifying 1,000 object categories needs millions.

It depends on the signal-to-noise ratio. If the relationship between features and target is strong and clear, you need less data. If it is weak and noisy, you need more.

It depends on the model. A decision tree with 5 splits has fewer parameters to learn than a neural network with 100 million parameters.

But there are practical heuristics.

The learning curve. Train your model on increasing fractions of the data (10%, 20%, 50%, 100%) and plot performance versus data size. A rising curve may justify collecting more representative data. A flat curve calls for tests of features, labels, objective and capacity before assuming that volume is the constraint.

The annotation budget approach. Calculate the cost per label (annotator time × hourly rate). Calculate the marginal improvement per additional label (from the learning curve). When the marginal cost exceeds the marginal value, stop labelling.

The chicken-and-egg problem: No data, no model; no model, no data

Some of the most valuable ML applications face a cold-start problem: you need labelled data to train a model, but you need a model to generate the behaviour that produces labelled data.

A recommendation system needs user interaction data to learn preferences. But without a recommendation system, there are no recommendations to interact with. A fraud detection system needs labelled fraud examples. But without a detection system, fraud goes undetected and unlabeled. A search engine needs click data on search results. But without a search engine, there are no search results to click on.

There are several strategies for breaking the chicken-and-egg cycle.

Heuristic bootstrapping. Start with a rule-based system (e.g., “recommend the most popular items”) and collect interaction data. Use that data to train a first-generation model. Use the model to collect better data. Iterate.

Transfer learning. Use a model trained on a related problem or dataset. A search engine for a photo stock company can start with a general-purpose CLIP model and fine-tune it on company-specific data as it accumulates.

Synthetic data generation. Create artificial training examples that approximate the real distribution. For demand forecasting, you can simulate demand patterns based on known seasonal factors.

Manual labelling. Pay experts to label a small seed dataset. Even 500-1,000 high-quality labelled examples can be enough to train a first-generation model that collects more data.

The design implication: the first version of your system does not need to be good. It needs to be good enough to generate data for the second version. This is why baselines matter (Chapter 8). A simple baseline that generates useful data is more valuable than a complex model that not launches.

Properties of a healthy data pipeline

A data pipeline is the plumbing that moves data from its sources to your model. Like actual plumbing, you do not think about it until it breaks. And when it breaks, everything downstream is affected.

A healthy data pipeline has five properties.

Reliability. The pipeline runs on schedule and handles failures gracefully. If a data source is temporarily unavailable, the pipeline retries, falls back to cached data, or alerts the team, rather than crashing silently and producing a truncated dataset.

Freshness. The data arrives within the time window your model requires. A real-time fraud detection model needs data within seconds. A weekly demand forecast needs data within hours. If the pipeline delivers data too late, the model makes predictions on stale information.

Completeness. All expected data arrives. If your model uses features from 10 sources and one source fails, the model receives 9 features instead of 10. The model does not know a feature is missing; it interprets the absence as a feature value (usually zero or null), which can produce systematically wrong predictions.

Consistency. The same data, processed by the same pipeline, produces the same output every time. Nondeterminism in data pipelines (caused by race conditions, floating-point rounding, or random sampling) makes debugging nearly impossible.

Observability. You can inspect the pipeline at every stage to understand what data is flowing, what transformations are applied, and where problems occur. This means logging, data quality checks at each stage, and schema validation.

The failure mode: Distribution mismatch

The most insidious data problem is not missing data or noisy labels. It is distribution mismatch: when the data you train on does not represent the data you serve on.

This happens more often than you think. You train on historical data from the last 12 months, but the world has changed since then (COVID-19 changed purchasing patterns overnight). You train on data from your most active users, but the model serves all users, including casual ones with very different behaviour. You train on data from one geography, but the model serves globally.

Distribution mismatch is the data problem that makes experienced ML engineers paranoid. It cannot be caught by standard model evaluation (because the test set has the same distribution as the training set). It can only be caught by monitoring in production, which is why Chapter 14 exists.

Dossier check

Before leaving this chapter, record the assumption that matters most to build the dataset as a product, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

Why reliable evaluation is so hard

The fundamental problem of evaluation is this: you want to know how your model will perform on data it has not seen. But by definition, you cannot test on unseen data, because once you see it, it is no longer unseen.

Validation schemes are clever approximations of this impossible ideal. They work by hiding a portion of the available data from the model during training and using it for evaluation. The hope is that performance on the hidden data predicts performance on truly new data.

This hope is justified under one critical assumption: the hidden data has the same distribution as the future data. When this assumption holds, your validation score is a reliable estimate of production performance. When it does not, your validation score is a lie that sounds convincing.

Standard validation schemas

Holdout validation is the simplest approach. Split your data into training (e.g., 80%) and test (e.g., 20%). Train on the training set, evaluate on the test set. Done.

The holdout approach is fast and easy to understand. Its weakness is variance: depending on which 20% you hold out, the score can change significantly. If the held-out portion happens to be easy, your score is optimistic. If it happens to be hard, your score is pessimistic.

Cross-validation addresses this by repeating the evaluation K times. Split the data into K equally sized folds. For each fold, train on the remaining K-1 folds and evaluate on the held-out fold. Average the K evaluation scores.

Cross-validation gives you both a mean performance estimate and a standard deviation, which tells you how stable the estimate is. If your 5-fold CV yields scores of [0.82, 0.81, 0.83, 0.80, 0.84], you can be fairly confident the true performance is around 0.82. If it yields [0.72, 0.91, 0.65, 0.88, 0.79], something is wrong: the variance is too high, likely indicating that the data is heterogeneous or the folds are unbalanced.

The choice of K matters more than most practitioners realize. K=5 and K=10 are a common choices. Smaller K (e.g., K=3) means smaller training sets per fold, leading to pessimistic estimates (the model trains on less data than it will in production). Larger K (e.g., K=20) means smaller test sets per fold, leading to high-variance estimates. The extreme case, leave-one-out cross-validation (K = N, where N is the dataset size), uses the largest possible training set but is computationally expensive and produces extremely high-variance individual scores.

Time-aware validation is needed when deployment has temporal structure. Training evidence must not cross the declared prediction-time boundary.

In time-series validation, the split is chronological. You train on data up to time T and evaluate on data from time T+1 to T+k. This mimics the production scenario, where the model consistently predicts the future using only the past.

The expanding window approach (train on all data up to the test period) is most common, but some practitioners use a sliding window (train on the most recent N months only) to give the model more recent data and less stale historical data. The right choice depends on whether older data is still relevant to current predictions.

Nontrivial schemas: When standard approaches fail

Standard validation works well when the data is independently and identically distributed (i.i.d.), meaning each row is drawn from the same distribution independently of every other row. In practice, this assumption is frequently violated.

Nested validation (also called double cross-validation) is required when you are tuning hyperparameters. The outer loop evaluates model performance. The inner loop selects hyperparameters. Without nesting, you contaminate the evaluation: you choose hyperparameters that look good on the test set, which means the test set is no longer truly held-out.

Adversarial validation is a diagnostic tool that detects distribution mismatch between your training and test sets. The idea is counterintuitive: train a classifier to distinguish between training and test examples. If the classifier succeeds (AUC > 0.5), there is a systematic difference between the two sets. The features the classifier uses to distinguish them tell you where the distributions differ.

This is especially useful in benchmark competitions (where the public and private test sets may have different distributions) and in production ML (where the training data from last year may differ from the serving data today).

Quantifying dataset leakage exploitation is a technique for detecting whether your model has accidentally accessed information it should not have. The classic example is temporal leakage: a feature that encodes information about the future. If you are predicting whether a customer will churn next month, and one of your features is “number of support tickets filed next month,” the model will have perfect accuracy and zero practical value.

Leakage is insidious because it produces models that look excellent in evaluation but fail completely in production. The best defense is a combination of domain knowledge (knowing which features could possibly contain leakage) and adversarial validation (detecting whether the model relies on features that distinguish train from test).

Split updating: When your validation scheme ages

Your validation splits are not permanent fixtures. They need to be updated as new data arrives, the world changes, or your understanding of the problem evolves.

When to update splits:

• When you accumulate significantly more data, so the previous test set is too small relative to the new dataset.
• When the data distribution changes (a new product category, a market shift, a pandemic).
• When you change the problem formulation (e.g., from predicting next-day demand to predicting next-week demand).

How to update splits:

For time-series data, the update is natural: slide the training and test windows forward.

For non-temporal data, the update requires care. You want to ensure that the new split is consistent with previous splits in terms of class distribution, feature distribution, and difficulty. Otherwise, improvements in your model might be artefacts of an easier test set, not genuine progress.

One design pattern maintains a “golden” test set that is not used for hyperparameter tuning and is only evaluated once per major model version. This golden set gives you an uncontaminated estimate of true performance. Think of it as the control group in your ongoing experiment with the model.

A concrete walkthrough: Validation for Merehaven Market

The following specifies the validation scheme for the Merehaven Market demand forecasting system.

The problem: Predict daily sales per SKU per store for the next 7 days.

Why standard cross-validation fails: The data is temporal. Sales on Monday depend on sales on Sunday. Random cross-validation would let the model “see” Tuesday’s sales while predicting Monday, which is impossible in production.

The chosen scheme: Time-series expanding window validation.

• Training window: All data up to T
• Prediction horizon: T+1 through T+7
• Evaluation windows: We slide T through the last 12 months, creating approximately 50 evaluation windows (one per week).

For each window, we compute: - Quantile metrics (1.5th, 25th, 50th, 75th, 95th, 99th percentiles) - MAE and WAPE at the SKU level and the store level - Bias (average signed error) to detect systematic over- or under-prediction - Seasonal breakdown (weekday vs. weekend, holiday vs. non-holiday)

The golden test set: The most recent 4 weeks of data are reserved as a final evaluation that is not used during development. Model candidates must beat the baseline on this set before deployment.

Confidence intervals: We use bootstrap resampling (1,000 iterations) on each evaluation window to compute 95% confidence intervals for all metrics. A model is only considered better than the baseline if the improvement is statistically significant.

The failure mode: Overconfidence from evaluation leakage

a common validation failure in practice is not dramatic like time leakage. It is subtle: multiple hypothesis testing.

Repeated reuse of the same test set turns it into part of the selection process. Under an illustrative assumption of 50 independent null tests at a 5% false-positive rate, the chance of at least one false positive is $1 - 0.95^{50}$, about 92%; dependent model variants require a more careful analysis.

In other words, if you try enough things, something will look good on the test set by accident. This is why the golden test set matters. It gives you one final, clean measurement that is not contaminated by the selection process.

Dossier check

Before leaving this chapter, record the assumption that matters most to validate against future use, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

Baselines expose the value of complexity

A baseline is the simplest decision or prediction route that creates a valid comparison. It establishes the bar that additional modelling and operating complexity must clear.

Baselines provide the first controlled evidence about where modelling can add value.

A baseline tells you:

1. How hard the problem is. If a constant prediction achieves MAE of 10, and your neural network achieves MAE of 8, the problem is mostly solved by simple approaches, and your model is providing incremental value. If the constant prediction achieves MAE of 100 and your model achieves MAE of 8, the model is doing real work.

2. Where the model’s value lies. By comparing the baseline’s errors to the model’s errors, you can see exactly which cases the model handles better. Maybe it is better on weekdays but worse on weekends. Maybe it handles popular items well but fails on rare ones. This error analysis (Chapter 9) is only possible with a baseline as a reference point.

3. What the deployment complexity is worth. A baseline often has near-zero latency, zero training cost, and zero maintenance burden. If the complex model is only 5% better, is the additional infrastructure, monitoring, and on-call burden justified?

4. Whether the problem is solvable at all. If your best baseline is already close to the theoretical optimum (e.g., predicting a random process), no amount of model complexity will help. The problem may be fundamentally noisy, and the right answer is to accept uncertainty rather than overfit to noise.

The hierarchy of baselines

Baselines are not all the same. They form a ladder of increasing complexity, and you should climb the ladder one rung at a time.

Constant baselines predict a single value for every input. For regression, this is the mean or median of the target. For classification, this is a common class. For the Merehaven Market demand forecasting system, the constant baseline is: “predict that tomorrow’s sales will equal last week’s same-day sales.”

Why bother? Because the constant baseline establishes the floor. If your model cannot beat “predict the historical average,” your model has negative value. This happens more often than you would think, especially with poor features or noisy targets.

An advanced constant baseline adds a bit of nuance. For Merehaven Market, this might be: “compute the 50th percentile of sales over the last year, weighted by recency.” This captures seasonality and trend without any modelling.

Heuristic baselines encode domain knowledge as rules. “If it is Monday, predict 20% less than the weekly average” (because Mondays are slow at grocery stores). “If there is a promotional event, multiply the prediction by 2.5.” Heuristic baselines are powerful because they leverage the accumulated expertise of the business, expertise that took years to develop and that a model might take months to learn from data.

Model baselines are the simplest ML models. Logistic regression for classification. Linear regression for regression. A decision tree with a small depth. These models have few hyperparameters, train in seconds, and produce interpretable results. If your model baseline already achieves acceptable performance, you may not need anything more complex.

Feature baselines keep the model simple but enrich the features. The insight here is that in many worked problems, the features matter more than the model. A logistic regression with 100 carefully engineered features will often outperform a random forest with 10 raw features. Feature baselines tell you how much performance you can extract from better feature engineering alone.

Deep learning baselines are the most complex but also the most expensive. They are appropriate when the problem involves unstructured data (images, text, audio) or when simpler models have been exhausted.

A worked example: Baselines for demand forecasting

The following builds the baseline ladder for Merehaven Market.

Constant baseline: Predict that demand on September 8, 2022, equals demand on September 1, 2022 (same weekday, one week back). This captures weekly seasonality with zero sophistication.

Result: MAE = 15.3 units, WAPE = 22%.

Advanced constant baseline: Compute the median of the same weekday over the last 52 weeks, excluding holidays. Apply a recency weight so recent weeks count more.

Result: MAE = 12.1 units, WAPE = 18%.

Model baseline: Train a gradient-boosted tree (LightGBM) on simple features: day of week, month, year, is_holiday, last_week_sales, last_month_average.

Result: MAE = 9.4 units, WAPE = 14%.

Feature baseline: Same LightGBM model, but add 50 additional features: rolling averages at multiple windows (7, 14, 28, 90 days), promotional indicators, weather data, price changes, category-level aggregates.

Result: MAE = 7.8 units, WAPE = 11%.

Deep learning baseline: A temporal fusion transformer trained on all features with attention over multiple time scales.

Result: MAE = 7.2 units, WAPE = 10%.

Now look at the improvement trajectory:

The synthetic comparison shows diminishing marginal improvement as model and operating complexity increase. The accepted candidate depends on whether that improvement changes a decision enough to pay for its continuing burden.

For a real business decision, the question is: does the 8% improvement in MAE translate to enough additional revenue to justify the additional complexity? Sometimes yes. Often no.

Baseline comparison: The dimensions that matter

When comparing baselines, accuracy is not the only dimension. Five properties determine which baseline to deploy.

Prediction quality is the obvious one: how well does the baseline predict the target?

Effort includes development time, compute cost, and the specialized skills required. A deep learning baseline requires GPU infrastructure, hyperparameter tuning expertise, and potentially weeks of experimentation. A heuristic baseline requires a few hours and a spreadsheet.

Deployment complexity covers serving infrastructure, latency requirements, and integration work. A constant baseline can be served from a static file. A deep learning model requires a model serving framework, GPU provisioning, and monitoring.

Maintenance includes debugging effort, retraining frequency, and the work needed when something breaks. A linear regression is trivial to debug. A 100-layer neural network is not.

Interpretability determines whether you can explain the model’s predictions to stakeholders, regulators, or users. A decision tree is self-explanatory. A neural network is a black box. In regulated industries (finance, healthcare), interpretability is not optional.

The right baseline depends on where you are in the project lifecycle. In the first iteration, prioritise low effort and high interpretability. Use the simplest baseline that produces acceptable results. In later iterations, when the simple approaches have been exhausted and the business case for improvement has been validated, invest in more complex models.

The failure mode: Skipping baselines and losing months

The failure mode is predictable. A team starts with a complex model because “we know we’ll need it eventually.” They skip baselines. They spend three months training and tuning the model. They achieve MAE of 7.5. They are pleased.

Then someone runs the constant baseline. MAE = 8.0.

The team’s three months of work improved upon the simplest possible approach by 6%. Most of that improvement is not statistically significant. The complex model is harder to maintain, more expensive to serve, and more difficult to explain to stakeholders.

If the team had started with baselines, they would have known from week one that the problem is 80% solved by simple approaches. They would have focused their complex modelling efforts on the 20% of cases where simple approaches fail (e.g., promotional events, new product launches), rather than trying to improve performance uniformly.

A thought experiment: The baseline test

Use this review before model deployment: the baseline test.

1. Compute the performance of the simplest possible baseline.
2. Compute the performance of your proposed model.
3. If the proposed model is less than 2x better than the baseline, ask yourself: is the additional complexity worth it?

No universal improvement ratio justifies complexity. The review should compare decision value, uncertainty, service cost, recovery burden and maintenance capacity under the accepted operating conditions.

Dossier check

Before leaving this chapter, record the assumption that matters most to make the baseline earn its defeat, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

Learning curves expose data and capacity limits

The loss curve plots training loss and validation loss at each epoch of training. It is the first thing you should look at after training a model, and it tells you one of three stories.

Story 1: Underfitting. Both training loss and validation loss are high and converging to a plateau. The model is not complex enough to capture the patterns in the data. Think of a linear model trying to fit a curved relationship. Adding more data will not help; adding more model capacity (features, layers, parameters) will.

Story 2: Overfitting. Training loss continues to decrease, but validation loss starts increasing. The model is memorizing the training data rather than learning generalizable patterns. Think of a student who memorizes exam answers but cannot solve new problems. More data, regularization, or simpler models will help.

Story 3: Good fit. Both losses decrease together and converge to a similar value. The model has found the right balance between capacity and generalization. More training will not help much; if you want improvement, you need better features, more data, or a fundamentally different approach.

The sample-wise learning curve plots performance versus training set size. Train on 10%, 20%, 50%, 100% of your data and measure validation performance at each point. This curve answers the critical question: “Will more data help?”

A rising curve is evidence that additional representative data may help. A flat curve suggests testing representation, objective, label quality and model capacity before paying for more data.

Double descent is a phenomenon that challenges the classical view of overfitting. In highly parameterized models (especially deep neural networks), as you increase model complexity, performance initially improves, then degrades (the classical bias-variance tradeoff), and then improves again. The model passes through a “interpolation threshold” where it has just enough capacity to memorize the training data, and performance is worst. Beyond that threshold, additional capacity improves generalization.

This is counterintuitive. Classical statistics says “more parameters = more overfitting.” Double descent says “much more parameters = sometimes less overfitting.” The practical implication: do not assume that making your model larger will automatically make it worse. Test it.

Residual analysis: What the errors tell you

Residuals are the differences between predicted and actual values. They are the errors, expressed as numbers. analysing their distribution, patterns, and correlations is where most actionable insights live.

Residual distribution. Plot a histogram of residuals. If the distribution is centered at zero and roughly symmetric, the model has no systematic bias. If the distribution is shifted (mean > 0), the model systematically over-predicts. If it is skewed, the model’s errors are asymmetric.

Residual vs. predicted value. Plot residuals on the y-axis, predicted values on the x-axis. If the residuals are uniformly distributed around zero for all predicted values, the model is well-calibrated. If the residuals fan out (heteroscedasticity), the model is less accurate for certain ranges. If you see a curve, there is a nonlinear pattern the model has missed.

Residuals across relevant groups. Compare residual direction and magnitude across declared populations. A persistent disparity requires investigation of data, label, policy and model effects, followed by the applicable legal and risk review for the decision context.

For Merehaven Market: if the model consistently under-predicts demand in stores located in lower-income neighborhoods, those stores will experience more out-of-stock events. The customers in those neighborhoods bear the cost of the model’s bias, even though they did nothing to cause it.

Elasticity curves measure how the model’s prediction changes as a single feature varies, holding everything else constant. This is closely related to partial dependence plots and SHAP values. Elasticity curves tell you: “For every 1-degree increase in temperature, the model predicts 5 more units of ice cream sold.” If this relationship makes domain sense, the model is likely correct. If it does not (e.g., the model predicts more ice cream sales when temperature decreases), something is wrong with the feature, the model, or the data.

Finding commonalities in errors: Where does the model struggle?

The most powerful technique in error analysis is grouping errors and looking for patterns.

Worst-case analysis. Sort examples by error magnitude (largest first). Look at the top 50 worst predictions. What do they have in common? Are they from a specific time period? A specific product category? Do they involve unusual feature values?

In the Merehaven Market case study, worst-case analysis revealed that the 20 largest errors all occurred on the day after a major promotional event. The model had not learned to predict the “hangover” effect: demand drops sharply the day after a promotion because customers stocked up.

Adversarial validation on errors. Train a classifier to distinguish between examples where the model performs well and examples where it performs poorly. The features this classifier uses tell you what makes predictions hard. If the classifier uses “is_promotional_day” as its top feature, you know that promotional dynamics are the model’s blind spot.

Group analysis. Compute metrics for slices of the data: by region, by product category, by day of week, by customer segment. A model that performs well on average might perform terribly on a critical slice. A recommendation system that averages 0.85 AUC might have 0.60 AUC for new users (who have no interaction history).

Corner-case analysis. Identify edge cases where the model’s behaviour is unexpected or harmful. What happens when all features are zero? What happens when a feature has an extreme value? What happens when two features that are usually correlated become decorrelated?

A worked example: Error analysis for Merehaven Images

Merehaven Images is building an image search engine. The model takes a text query and returns relevant images, ranked by predicted relevance.

Baseline performance: NDCG@10 = 0.72 (where 1.0 is perfect ranking).

Residual analysis reveals: - The model performs well on concrete queries (“red sports car,” “sunset over ocean”) with NDCG@10 = 0.85. - The model performs poorly on abstract queries (“loneliness,” “success,” “freedom”) with NDCG@10 = 0.45. - The model performs poorly on queries with domain-specific terminology (“bokehlicious portrait,” “rule of thirds landscape”) with NDCG@10 = 0.38.

Worst-case analysis reveals: - The 30 worst queries all involve either abstract concepts or photography jargon. - The underlying CLIP model was not trained on photography-specific language.

Actionable improvements: 1. Fine-tune the CLIP model on photography-specific (image, text) pairs. 2. Build a query expansion module that translates domain jargon into general terms (“bokehlicious” → “shallow depth of field portrait with blurred background”). 3. Create a curated taxonomy of abstract concepts with manually tagged images to improve recall on abstract queries.

Each of these improvements targets a specific failure mode identified by error analysis. Without the analysis, the team might have spent weeks randomly tuning hyperparameters, hoping to improve overall NDCG. With the analysis, they know exactly where the value lies.

The failure mode: Averaging away problems

a common error analysis failure is not performing one.

The second most common is performing one at the wrong level of aggregation. If you only look at aggregate metrics (overall MAE = 8.0), you miss the fact that MAE = 4.0 for weekday predictions and MAE = 16.0 for weekend predictions. The aggregate looks acceptable; the weekend performance is terrible.

Slice-based analysis prevents an aggregate from hiding decision-relevant failures. Compute measures for the populations, times and conditions that can change the release decision. For Merehaven Market, that means per-category, per-region, and per-day-of-week. For Merehaven Images, that means per-query-type.

Dossier check

Before leaving this chapter, record the assumption that matters most to study errors as structured evidence, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

What is a training pipeline?

A training pipeline is a directed acyclic graph (DAG) of steps that takes a dataset as input and produces a trained model as output. Each step takes some input, performs some action, and produces output that feeds into the next step.

The word “pipeline” is overloaded in ML. The model itself is a pipeline (preprocessing → model → postprocessing). The inference system is a pipeline. The data pipeline is a pipeline. To avoid confusion, here is the terminology used in this edition:

A training pipeline produces a trained model from data. It runs periodically (daily, weekly, or on-demand).

An inference pipeline produces predictions from raw inputs. It runs continuously in production.

The training pipeline has seven stages:

Data fetching downloads data from sources and makes it available for processing. This is where you interface with data engineering infrastructure.

Preprocessing prepares data for training. Feature selection, normalization, augmentation, handling missing values. Some preprocessing is training-specific (feature selection), and some is shared with inference (normalization).

Model training is the core. It takes preprocessed data and produces a trained model. In deep learning systems, this is the longest and most resource-intensive step.

Evaluation and testing answers “how good is this model?” Evaluation computes metrics. Tests run automated checks: Does the model produce predictions for all input types? Are predictions within expected bounds? Are metrics above minimum thresholds?

Postprocessing prepares the model for deployment: format conversion (to ONNX or CoreML), quantization, compilation for specific hardware.

Report generation produces human-readable summaries: validation metrics, error analysis visualizations, comparison with previous model versions. No responsible engineer deploys a model without reviewing reports.

artefact packaging bundles everything (model weights, configuration files, preprocessing parameters) into a deployable format. The output should be as agnostic as possible to the training pipeline, so changes in training do not cascade to deployment.

Tools and platforms: There is no one-size-fits-all

Choosing the right tools for your training pipeline is determined by your problem scale and organisational infrastructure. FAANG-level companies build custom platforms. Smaller companies assemble open-source tools and cloud services.

The essential capabilities a platform needs:

Dependency resolution: Steps run in the right order, with upstream outputs feeding downstream inputs.

Reproducibility: Given the same parameters and code version, the pipeline produces identical results.

Compute integration: Run steps on appropriate hardware (CPUs for preprocessing, GPUs for training).

artefact storage and experiment tracking: Store model weights, configurations, and metrics. Track which experiment produced which result.

Caching: Expensive intermediate results are cached so unchanged steps are not re-run.

Tool selection starts with required capabilities: orchestration, experiment lineage, artefact storage, isolated execution, observability, access control and recovery. Compare supported implementations against those requirements at release time.

The key advice: choose tools based on your current needs, not your aspirational needs. A startup training one model per month does not need a workflow orchestrator. A team training 100 models per day does.

Scalability: Vertical vs. horizontal

When your training pipeline takes too long, you have two options.

Vertical scaling means getting a bigger machine: more CPUs, more RAM, more GPUs. It is simple, requires no code changes, and has a hard ceiling (the biggest machine money can buy).

Horizontal scaling distributes work across multiple machines. It can extend beyond one machine, but requires coordination and introduces communication, synchronisation and partial-failure costs. It also introduces new failure modes: network latency between machines, synchronization overhead, and the possibility that one machine fails mid-training.

For most ML teams, vertical scaling is sufficient. A single machine with 8 A100 GPUs can train most models that non-FAANG companies need. Horizontal scaling becomes necessary when your dataset or model is too large for a single machine, which is increasingly common with LLMs but still uncommon for most applied ML.

Expose only governed configuration

How configurable should your pipeline be?

Too rigid: Every parameter is hardcoded. To change the learning rate, you edit source code. To use a different dataset, you change a file path buried in a script. Any change requires a developer and a code deployment.

Too flexible: Everything is configurable. The pipeline reads from a 500-line YAML file with parameters for every conceivable option, most of which interact in undocumented ways. The interactions are not understood, and changing one often breaks something else.

The sweet spot: make the things that change often easily configurable. Hardcode the things that change rarely. In practice, this means:

• Configurable: learning rate, batch size, number of epochs, dataset path, feature list, evaluation metrics.
• Hardcoded: data processing logic, model architecture (at the level of code), evaluation pipeline structure.

The boundary between configurable and hardcoded shifts as the project matures. Early in the project, you are experimenting with architectures (keep them in code, easy to modify). Later, when the architecture is stable, you might make it configurable to support A/B tests between variants.

Testing your pipeline: Not optional

Testing ML pipelines is often neglected because “it’s not production code, it’s just training.” This is wrong. A training pipeline that produces a bad model is worse than no model, because a bad model gets deployed and makes bad predictions that affect users.

Property-based testing is particularly useful for ML pipelines. Instead of testing specific outputs, you test properties that should hold:

• “The model’s predictions should be between 0 and 1 for probability estimates.”
• “The training loss should decrease over the first 10 epochs.”
• “The model’s output shape should match the expected shape.”
• “The preprocessed dataset should have no null values in required columns.”

These tests exercise the contract of each pipeline stage alongside its specific input-output examples.

A concrete walkthrough: Merehaven Images’s training pipeline

The Merehaven Images search engine uses a fine-tuned CLIP model. Here is their training pipeline design:

Toolset: - PyTorch for model training - PyTorch Lightning for structured training loops - Flyte for workflow orchestration (already used for data engineering) - a managed training platform for compute - TensorBoard for visualization - Docker for reproducibility

Pipeline output: - Two ONNX models: text encoder and image encoder - Configuration files for inference (prompt templates, preprocessing parameters) - A training report with metrics

Active experimentation areas: - What to fine-tune (full model, specific layers, LoRA adapters) - Augmentation techniques (for images: random crop, color jitter; for text: paraphrase, synonym substitution) - Loss functions (contrastive loss with different margin values) - Backbone architectures (convolutional vs. transformer-based image encoders)

The pipeline is designed to make these experiments reproducible. Every experiment is tracked with its configuration, data version, and results. When someone asks “why did we choose the transformer backbone?”, the answer is in the experiment logs, not in someone’s memory.

The failure mode: The irreproducible result

a common training pipeline failure is irreproducibility. You train a model on Monday and get MAE of 7.2. You retrain on Tuesday with “the same” configuration and get MAE of 8.1. What changed?

Possible culprits: random seed not fixed, data shuffle order changed, GPU numerical precision differs, a dependency was updated overnight, the data source added new records between Monday and Tuesday.

Each of these is individually fixable. Together, they create a combinatorial nightmare. The defense is systematic: pin random seeds, version data, pin dependencies, log hardware configurations, and test that re-running the pipeline produces identical results.

Dossier check

Before leaving this chapter, record the assumption that matters most to turn training into a reproducible system, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

Feature engineering as an iterative process

Feature engineering is not a one-time step. It is an iterative cycle that runs throughout the project’s life.

The cycle works like this: you start with domain knowledge (which signals should matter?), create features, train a model, analyse errors, and discover that the model fails on specific cases. Those failure cases suggest new features. You create them, retrain, and the cycle continues.

For Merehaven Market, the first iteration might use simple features: day of week, last week’s sales. Error analysis reveals that the model fails on promotional days. This suggests promotional features: is there a promotion today, what type (buy-one-get-one, percentage discount), how many days has the promotion been running. These features improve performance on promotional days, and the cycle continues.

Feature importance: Why does the model use what it uses?

After training, the first question is: which features actually matter?

Feature importance analysis serves two purposes. First, it validates that the model is using features that make domain sense. If a demand forecasting model places its highest importance on “store ID” rather than “day of week,” something is suspicious (the model might be memorizing store-specific patterns rather than learning generalizable ones). Second, it identifies features that can be removed without losing performance, simplifying the model and reducing serving latency.

Permutation importance is the most model-agnostic method. Shuffle a single feature’s values randomly, re-evaluate the model, and measure how much performance drops. If performance drops a lot, the feature is important. If it barely changes, the feature is redundant.

SHAP (SHapley Additive exPlanations) values estimate per-feature contributions for an individual prediction and can also be aggregated across a review population. Their interpretation depends on the explainer, background distribution and feature dependence assumptions. For a specific prediction of “150 units of milk tomorrow,” SHAP might tell you: “day of week contributes +30, last week’s sales contributes +50, promotional event contributes -10 (it’s a competitor’s promotion), temperature contributes +20.”

The analogy: permutation importance tells you which musicians matter in an orchestra. SHAP values tell you what each musician contributes to each specific performance.

Feature selection: Less is often more

More features is not necessarily better. Adding features has costs:

Training cost increases linearly (or worse) with feature count. Training with 1,000 features takes significantly longer than training with 100 features.

Serving latency increases because every feature must be computed at prediction time. If a feature requires a database query, that query runs for every prediction.

Maintenance burden increases because every feature has a data source, a computation pipeline, and a set of assumptions that can break. A feature derived from a third-party API is a dependency on that API’s reliability and schema stability.

Overfitting risk increases with the number of features, especially when training data is limited. More features means more ways for the model to find spurious correlations.

Feature selection methods include:

• Filter methods: Remove features with low variance, high correlation with other features, or low correlation with the target. Fast and simple.
• Wrapper methods: Train models with different subsets of features and compare performance. Effective but expensive.
• Embedded methods: Use models that perform feature selection as part of training (L1 regularization, tree-based importance). A good default.

A practical heuristic: start with all features, train a model, compute importance, remove the bottom 20% of features, retrain, and check if performance is maintained. If it is, repeat. Stop when removing features hurts performance.

Shadow features: A clever trick for feature selection

One useful technique that deserves more attention: shadow features.

Create random copies of each feature by shuffling their values. Train the model on both the original features and the shuffled copies. Any original feature that is less important than the best random feature is effectively noise and should be removed.

The intuition: if a real feature is less informative than random noise, it is not contributing to the model’s predictions. It is either irrelevant or its information is already captured by other features.

Feature stores: The infrastructure investment that pays off

A feature store is a centralised system for computing, storing, and serving features. It ensures that the same feature computation used in training is used in serving (avoiding training-serving skew) and allows features to be shared across teams and models.

Feature stores are most valuable when:

• Multiple models use the same features. If five models all use “user’s average order value,” computing it once and storing it is more efficient than computing it five times.
• Real-time serving requires precomputed features. Computing “user’s rolling 30-day average” at prediction time requires scanning 30 days of data per request. Precomputing and storing it makes serving fast.
• Training-serving consistency is critical. If the training pipeline computes “time since last purchase” differently from the serving pipeline (due to timezone handling, for example), the model will perform worse in production than in evaluation.

Feature stores are less valuable for pure deep learning problems where the model operates directly on raw inputs (images, text). The “feature engineering” in these cases happens inside the model (through learned representations) rather than in a preprocessing pipeline.

Model predictions as features: Power and peril

One of the most powerful feature engineering techniques is using the output of one model as a feature in another. This is called model stacking.

The simplest example: target encoding. For a categorical feature like “city,” replace each city name with the average target value for that city. The model now sees a numerical signal that directly captures the relationship between city and target. But target encoding is dangerous without proper safeguards: if you compute the encoding on the same data you train on, you leak information and the model overfits.

A more sophisticated example: using a weather forecast model’s predictions as a feature in a demand forecasting model. Weather obviously affects demand (people buy more ice cream when it is hot), and a weather forecast captures this signal.

The risks:

Data leakage. If the encoding uses information from the future or from the test set, the model learns to cheat.

Dependency fragility. If the weather forecast API changes its format, your demand model breaks.

Drift amplification. If the upstream model drifts (the weather forecast becomes less accurate), your downstream model inherits and amplifies the drift.

Mitigation requires careful engineering: nested cross-validation for target encoding, version-pinning for external models, and monitoring for upstream drift.

Dossier check

Before leaving this chapter, record the assumption that matters most to govern the feature contract, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

Offline evaluation: The hypothesis

Before deploying your model to real users, you need an estimate of its impact. This is offline evaluation, and it answers: “Based on historical data, what effect can we expect?”

Offline evaluation uses the validation schemes from Chapter 7 to compute metrics on held-out data. The result is a hypothesis: “Based on offline metrics, we expect this model to improve CTR by approximately 5%.”

The qualification is “approximately.” Offline metrics are proxies for online metrics, not substitutes. The relationship between offline improvement and online improvement is rarely linear and often surprising. A model that improves offline NDCG by 10% might improve online CTR by 2%, or by 15%, or not at all.

This is why A/B tests exist.

Simulated environments offer a middle ground. You can replay historical data through the new model and compare its recommendations to what the old model actually served. The limitation is that you can only evaluate on items that were actually shown; you cannot observe user reactions to items the new model would recommend but the old model did not.

Human evaluation is useful when automated measures cannot represent the judgement of interest. Its validity depends on assessor guidance, sampling, blinding, agreement analysis and a clear link to the operating decision.

Controlled experiments and their assumptions

An A/B test splits users into two groups: a control group (using the existing system) and a treatment group (using the new system). By comparing the metrics of both groups over the same time period, you isolate the effect of the change from external factors (seasonality, marketing campaigns, economic shifts).

A/B testing is a strong causal design when its assumptions hold because it answers the causal question: “Did the new model cause the improvement, or would it have happened anyway?”

Reporting results: The art of communicating evidence

When A/B testing is not possible

Some domains make classical A/B testing infeasible: legal restrictions (healthcare, finance), logistical limitations (small user bases), or ethical concerns (cannot withhold a potentially life-saving treatment).

Alternative approaches include:

• Causal inference methods: Difference-in-differences, synthetic control, regression discontinuity.
• Interrupted time-series analysis: Compare performance before and after the change.
• Quasi-experiments: Use natural variation as a proxy for random assignment.

These methods are less rigorous than A/B tests but better than no measurement at all.

The failure mode: Peeking at results

a common A/B testing mistake is peeking: checking the results before the planned experiment duration has elapsed.

If you check daily and stop the experiment the first time the p-value drops below 0.05, your effective false positive rate is far higher than 5%. This is because statistical significance fluctuates over time. A result that looks significant on day 3 might reverse by day 7.

The fix: define the experiment duration in advance, and do not make a decision until the planned duration has elapsed. If you must monitor during the experiment, use sequential testing methods that adjust the significance threshold for multiple looks.

Dossier check

Before leaving this chapter, record the assumption that matters most to experiment without fooling yourself, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

API design: The contract between your model and the world

Your ML system communicates with the outside world through an API. The API is a contract: it specifies what inputs the system accepts, what outputs it produces, and what sets service objectives it makes about latency, availability, and error handling.

The authors emphasize several API design practices that ML engineers often overlook:

Version your API. Models change. Input features change. Output formats change. Without versioning, every model update is a potential breaking change for every consumer.

Decouple model updates from API updates. The API contract should be stable even as the underlying model is swapped. This requires an abstraction layer between the API and the model. The API consistently accepts the same inputs and returns the same output format; internally, the model can be updated, rolled back, or A/B tested without the consumer knowing.

Design failure routes. Specify timeout, invalid-input, dependency and internal-error behaviour for each interface. Return meaningful error codes. Distinguish between “the model failed” (return a fallback prediction) and “the input is invalid” (return a 400 error with a helpful message).

Document response times. If your model takes 500ms per prediction, your consumers need to know. If they are building a real-time application that requires sub-100ms responses, they need to know before they integrate, not after.

Release cycle: How to ship models without breaking things

The release cycle for ML models is different from the release cycle for regular software, because model updates are inherently stochastic. A code change has deterministic effects; you can predict what will change. A model update has probabilistic effects; you hope things improve, but you cannot be certain until you measure.

Shadow deployment runs the new model alongside the production model. Both receive the same inputs; only the production model’s outputs are served to users. The new model’s outputs are logged and compared. This catches cases where the new model produces unexpected behaviour (predicting negative prices, returning empty result sets) without affecting users.

Canary deployment serves the new model to a small, predeclared fraction of eligible traffic. If metrics hold, the fraction increases gradually. If metrics degrade, the deployment is automatically rolled back. This is similar to the A/B testing framework from Chapter 12, but the goal is safety rather than measurement.

Blue-green deployment maintains two complete production environments. The “blue” environment runs the current model; the “green” environment runs the new model. Traffic is switched instantly from blue to green. If problems arise, traffic is switched back instantly. The downside is cost: you maintain two full environments.

Operating the system: Connections you did not plan for

Once your ML system is live, it connects to other systems in ways you did not anticipate. These connections create dependencies, and dependencies create failure modes.

Tech-related connections: Your model consumes features from a feature store, which consumes data from a warehouse, which consumes events from a streaming platform. If any link in this chain breaks, your model receives bad input and produces bad output. The chain is only as strong as its weakest link.

Non-tech-related connections: Your model’s predictions influence human decisions, which generate new data, which feeds back into the model. This feedback loop can be virtuous (the model improves over time) or vicious (the model’s biases are reinforced by the data it generates). A recommendation system that shows mostly popular items generates click data biased toward popular items, which trains the next model to recommend popular items even more heavily. This is the echo chamber effect, and it requires active intervention (exploration, diversity constraints) to prevent.

Overrides and fallbacks: When the model should not have the last word

Every ML system needs a mechanism for human override and automatic fallback.

Human overrides allow business stakeholders to manually adjust the model’s predictions. For Merehaven Market, the procurement team might override the demand forecast for a specific product because they know something the model does not (a local event, a competitor’s promotion, a supply disruption). The system should make overrides easy to apply, easy to track, and easy to analyse (so you can later determine whether the override improved or degraded the outcome).

Automatic fallbacks activate when the model is unavailable or producing obviously wrong results. If the model server goes down, the system should fall back to a simpler model (the baseline from Chapter 8) or a cached set of recent predictions. The fallback should be invisible to the user; they should get a slightly less accurate prediction, not an error page.

The design of overrides and fallbacks is a test of your system architecture. If overrides and fallbacks are afterthoughts bolted onto the system after deployment, they will be fragile and untested. If they are designed from the beginning, they will be reliable and well-tested.

The failure mode: Integration as an afterthought

The worst integration failures happen when the ML team builds in isolation and tries to connect to the rest of the organisation at the end.

Common symptoms: the API format does not match what the frontend expects, the model’s latency is too high for the user experience, the output format is incompatible with the downstream decision system, and the security team has not approved the data flows.

Every one of these problems is trivially avoidable if integration is considered from Day 1 of the design document. Every one becomes a multi-week emergency if discovered at deployment time.

Dossier check

Before leaving this chapter, record the assumption that matters most to integrate through explicit contracts, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

Why monitoring is not optional

An ML system has four points where things can go wrong, and each requires different monitoring.

Incoming data can degrade in quality (missing fields, corrupted values), change in distribution (new user segments, shifted behaviour), or stop arriving entirely (upstream pipeline failure).

The model can fail technically (out-of-memory errors, exceeding latency SLAs), silently produce degraded results (because the input data changed), or become stale (the world evolved but the model did not).

Model output can develop anomalous patterns: predictions clustering in unexpected ranges, confidence scores that are suspiciously uniform, or outputs that violate business constraints (negative prices, probabilities > 1).

Postprocessing and decision-making can amplify model errors through business rules, or human overrides can mask model problems (if humans consistently override the model, the model might be broken).

Software system health: The foundation

Before monitoring anything ML-specific, monitor the basics. Is the server running? Is it responding within SLA? Is memory usage stable? Are there error spikes?

These are standard software monitoring concerns (CPU, memory, latency, error rate, throughput), and mature tools exist for them (Prometheus, Grafana, Datadog, CloudWatch). The ML-specific monitoring sits on top of this foundation.

If your model server runs out of memory every Tuesday at 3 AM because the batch prediction job overlaps with a data pipeline, no amount of model monitoring will help. Fix the infrastructure first.

Data quality and integrity

Data problems are a common cause of model degradation, and they come in several flavors.

Processing problems include corrupt data (invalid values, encoding errors), missing data (null fields that should be populated), and schema changes (a feature renamed or restructured without updating the model pipeline).

Data source corruption occurs when an upstream data source changes its behaviour: a third-party API modifies its output format, a database migration alters column types, or a partner stops sending data without notification.

Cascade/upstream model problems happen when your model depends on another model’s output (e.g., using a weather forecast model’s predictions as a feature). If the upstream model degrades, your model inherits the degradation.

Training-serving skew is a source of hidden error. If the feature computation in your training pipeline differs from the feature computation in your serving pipeline (different timestamp handling, different null imputation, different aggregation windows), the model sees different features in production than it saw during training. Performance degrades for reasons that are invisible in offline evaluation.

How to monitor: Compute statistics on incoming data at every stage: means, medians, standard deviations, min/max values, null rates, unique counts. Compare these statistics to historical baselines. Alert when any statistic deviates beyond a threshold.

Use a version-pinned data-quality framework or custom checks inside the pipeline. Run the checks automatically at the cadence declared by the data contract.

Model quality and relevance

Even if the data is perfect, the model can degrade because the relationship between features and target changes over time.

Data drift (also called covariate drift) occurs when the distribution of input features changes. Users age, markets shift, products are added or removed. The model’s feature distributions in production diverge from what it saw during training.

Concept drift occurs when the relationship between features and the target changes. In the grocery example, the relationship between “day of week” and “demand” changed during the pandemic. Saturday demand used to be high (people shop on weekends); during lockdowns, demand was high every day (people shopped whenever they could).

How to monitor drift: Compare the distribution of features and predictions in production to the training distribution. Common methods include the Kolmogorov-Smirnov test (for univariate distributions), the Population Stability Index (PSI), and multivariate drift detection using domain classifiers.

How to react to drift: The response depends on severity. Minor drift: monitor closely and retrain at the scheduled time. Moderate drift: retrain early with recent data. Severe drift: investigate the cause (is it a data bug or a worked change?) and decide whether to retrain, retrain with data from a specific period, or change the model architecture.

Prediction drift monitoring compares the distribution of model outputs over time. If your model’s predictions shift (e.g., it starts predicting higher values on average), this may indicate that the input data has drifted or that the model is being exploited (adversarial users manipulating inputs to get favorable predictions).

Pairwise feature correlations: A powerful monitoring signal

One monitoring technique that deserves more attention: tracking pairwise feature correlations over time.

In training data, features have characteristic correlations. If “temperature” and “ice cream demand” are correlated at r=0.7 in training data, and this correlation drops to r=0.2 in production, something has changed. Maybe the temperature data is lagging, or maybe consumer behaviour has shifted.

Correlation monitoring catches problems that univariate monitoring misses. A feature might have the same mean and standard deviation as in training, but its relationship to other features might have changed completely.

The failure mode: Alert fatigue

The failure mode of monitoring is not too few alerts. It is too many. When every minor fluctuation triggers an alert, engineers stop looking at alerts. This is alert fatigue, and it is more dangerous than no monitoring at all, because it provides a false sense of security (“we have monitoring in place”) while being practically useless.

The defense: tiered alerting. Critical alerts (model server down, predictions violating constraints) go to a pager. Warning alerts (metric degradation beyond threshold) go to a Slack channel. Informational alerts (minor drift detected) go to a dashboard that is reviewed weekly.

Dossier check

Before leaving this chapter, record the assumption that matters most to monitor data, score, decision and outcome, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

The challenge of serving ML models

An inference pipeline is a sequence of steps that takes raw input and produces a prediction. For a computer vision system: receive image → resize → normalize → run model → postprocess → return result. For a recommendation system: receive user ID → fetch features from feature store → compute scores → apply business rules → return ranked list.

The pipeline is usually sequential, and latency is the sum of all steps. Model execution is only one part of end-to-end latency; feature access, serialisation, queues and downstream calls must be measured in the same trace.

Tradeoffs: You cannot optimize everything

Inference optimisation involves at least eight factors, and they conflict with each other.

Latency is the time between receiving a request and returning a response. Lower is better, but lower latency usually costs more.

Throughput is the number of requests handled per second. Higher is better, but higher throughput at constant latency requires more hardware.

Scalability is the ability to handle increasing load. Systems must handle traffic spikes (Black Friday, viral events) without degrading.

Target platforms constrain your options. A model serving on a cloud GPU cluster has different constraints than a model running on a smartphone.

Cost is the total expense of serving: hardware, energy, engineering time. A model that requires four A100 GPUs to serve is impractical for most companies.

Service reliability describes whether the system returns a valid response inside its accepted envelope; predictive correctness is evaluated separately. Acceptable error and drop rates follow from the decision consequences and service objectives.

Flexibility is the ability to update the model without redesigning the serving infrastructure. If deploying a new model version requires a two-week infrastructure change, iteration speed is crippled.

Security and privacy constrain where computation happens. If user data cannot leave the device, inference must run on-device.

These factors are often mutually exclusive. Lower latency requires more expensive hardware. Higher reliability requires redundancy, which increases cost. Flexibility requires abstraction layers, which add latency. The art of inference optimisation is choosing the right tradeoffs for your specific application.

Patterns: The architectural toolbox

Several serving patterns address different tradeoff profiles.

Online serving (synchronous, real-time) handles each request individually. The user sends a query, waits for the response, and the model runs on demand. This is the default pattern for user-facing applications. Latency requirements are strict (sub-second).

Batch serving (asynchronous) processes large sets of inputs at once, typically on a schedule. Demand forecasts are computed nightly for all products. Recommendations are precomputed hourly for all users. Latency is irrelevant (hours are fine); throughput and cost are the primary concerns.

Precomputation is a hybrid: compute predictions in advance for all likely inputs and store them in a cache. When a request arrives, look up the cached prediction instead of running the model. This converts an inference problem into a lookup problem (sub-millisecond latency). The limitation is that you can only precompute predictions for inputs you can anticipate. For a recommendation system with 10 million users and 100 thousand items, precomputing all user-item scores is infeasible (10^12 combinations).

Batching groups multiple incoming requests and processes them together. Neural networks are more efficient at processing batches than individual examples (GPU utilization improves materially with larger batch sizes). The tradeoff: individual requests wait for the batch to fill, increasing average latency.

Tools and frameworks: Training is not serving

A critical but often forgotten step: train your model in one framework, serve it in another.

Research frameworks (PyTorch, TensorFlow) are designed for flexibility and experimentation. Serving frameworks (TensorRT, ONNX Runtime, a model-serving runtime) are designed for performance and reliability. The gap between them is an interframework medium, typically ONNX, that converts a trained model into a format optimised for inference.

Why does this matter? Because a PyTorch model running in eager mode (interpreting operations one at a time) is typically 2-5x slower than the same model compiled to a static graph and optimised with operator fusion, constant folding, and memory layout optimisation.

The interframework conversion also decouples the training and serving pipelines. When your research team experiments with a new training framework, the serving infrastructure does not need to change. This separation is important for teams that iterate quickly on models.

Serverless inference (the selected cloud platform a managed request runtime, the selected cloud platform Functions) offers a different tradeoff: zero infrastructure management, automatic scaling, pay-per-request pricing. The limitations are cold start latency (the first request after an idle period takes seconds, not milliseconds), execution time limits, and limited hardware options (no GPUs in most serverless platforms).

Serverless is ideal for low-traffic models where maintaining a dedicated server is wasteful. It is a poor choice for latency-sensitive, high-traffic applications.

Optimizing inference pipelines

When performance matters, optimise in this order:

Step 1: Profile. Before optimising anything, measure where time is spent. Use version-compatible runtime, hardware and language profilers to identify bottlenecks. The measured bottleneck may differ from the team’s intuition. Teams spend weeks optimising the model when the actual bottleneck is a database query in the feature pipeline.

Step 2: Eliminate unnecessary work. Can you precompute any features? Can you cache frequent predictions? Can you reduce the input size (smaller images, shorter sequences)? These are the cheapest optimizations and often the most impactful.

Step 3: optimise the bottleneck. If the model is the bottleneck, consider quantization (FP32 → FP16 → INT8), pruning (removing unimportant weights), or distillation (training a smaller model to mimic the larger one). If the feature pipeline is the bottleneck, consider precomputation or caching. If I/O is the bottleneck, consider batch fetching or data locality.

Model quantization deserves special attention. Reducing numerical precision from 32-bit floating point to 16-bit or 8-bit integer can reduce model size by 2-4x and improve inference speed by 1.5-3x, with minimal accuracy loss. The analogy from earlier in this edition holds: reducing the color depth of a photo from 24-bit to 8-bit produces a nearly identical image at 3x smaller file size.

Step 4: Scale horizontally. If a single optimised server cannot handle the load, add more servers behind a load balancer. This is the most expensive optimisation and should be the last resort.

A concrete example: Serving for Merehaven Market

Merehaven Market needs demand forecasts for 10,000 SKUs across 500 stores, computed daily for a 7-day horizon. Total predictions per day: 10,000 × 500 × 7 = 35 million.

Serving pattern: Batch. The forecast is computed nightly and loaded into the procurement system before 6 AM. Latency per prediction is irrelevant; total batch time must be under 8 hours.

optimisation: With a simple gradient-boosted tree model, 35 million predictions take approximately 15 minutes on a single machine with 32 CPU cores. No GPU required. No optimisation required.

Inference optimisation matters only when measurement shows that serving constraints are binding. Workloads with generous latency and freshness windows may be better served by simple batch routes; interactive or high-volume routes require a separately tested envelope.

The failure mode: Premature optimization

Knuth’s law, “premature optimisation is the root of all evil,” applies with special force to ML inference. Teams that optimise before profiling waste effort on non-bottlenecks. Teams that optimise before validating business need build infrastructure for a latency requirement that does not exist.

Ask: “What latency does the application actually require?” If the answer is “minutes” or “hours,” spend your engineering time on model quality and monitoring, not on serving optimisation.

Dossier check

Before leaving this chapter, record the assumption that matters most to design serving around an envelope, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.

Accountability: Someone must own every component

Accountability means that for every component of your ML system, there is a named person (or team) who is responsible for its correct functioning. Not “the data science team.” Not “whoever has time.” A specific person.

Why does this matter? Because systems without clear owners degrade through inaction. When a monitoring alert fires at 2 AM, someone needs to respond. When a data source changes its schema, someone needs to update the pipeline. When a stakeholder asks “why did the model predict X?”, someone needs to answer.

Without clear accountability, maintenance work remains unowned. The ownership matrix assigns one authority and one backup to each component and operating decision.

Create an ownership matrix: a simple table listing every component of the system, its current owner, and a backup owner.

This table is embarrassingly simple. It is also one of the most powerful tools for system reliability. When something breaks, the question “who should fix this?” has an immediate, unambiguous answer.

Continuity risk: how concentrated is essential knowledge?

The continuity risk is a morbid but useful metric: how many team members would need to be suddenly unavailable (hit by a bus, quit, go on extended leave) before the project can no longer function?

A continuity risk of 3 or higher is healthy. It means at least three people understand the system well enough to maintain it independently.

But there is a tradeoff. Increasing the continuity risk requires redundancy: multiple people knowing the same things. Redundancy costs time (training people, pair programming, documentation reviews) and reduces efficiency (two people doing what one could do). Highly efficient teams with zero redundancy are extremely fragile. Highly redundant teams with maximum continuity risk are slow.

How to increase the continuity risk without killing efficiency:

1. Code reviews for all changes. Every commit is seen by at least one other person. This spreads knowledge passively.
2. Paired maintenance for critical components. Concentrate shared work on the consequential and least understood paths.
3. Documentation that records mechanism and rationale.

Documentation preserves operating continuity

The design document from Chapter 4 is the foundation, but it is not sufficient for maintenance. You also need operational documentation:

Runbooks describe how to respond to specific incidents. “If monitoring alert X fires, check Y, run Z, escalate to W if unresolved within 30 minutes.” Runbooks turn emergency response from improvisation into a checklist.

Architecture diagrams show how components connect. Not the aspirational architecture from the design document, but the actual architecture as deployed. These diagrams should be updated with every significant change.

Decision logs record why specific choices were made. “We chose LightGBM over XGBoost because LightGBM’s categorical feature handling reduced preprocessing complexity. We considered a neural network but rejected it because the training data is too small (< 100K examples) and interpretability is a requirement from the compliance team.”

Decision logs are the most valuable and most neglected form of documentation. Without them, a new engineer looks at the code and asks “why is this done this way?” The answer is either “I don’t know, the person who wrote it left” or “I think it was for performance reasons, but I’m not sure.” Neither answer helps the new engineer make informed decisions about future changes.

The deceptive appeal of complexity

The greatest enemy of maintainable ML systems is not technical debt. It is complexity that was not needed in the first place.

Complexity is seductive. A system with five interacting models, a custom training framework, a novel loss function, and a hand-tuned inference pipeline feels like an achievement. It is also a maintenance nightmare. Every component increases the surface area for bugs. Every interaction between components creates potential failure modes. Every novel technique requires specialized knowledge that might not exist on the team six months from now.

The authors offer a blunt assessment: excessive complexity either results from personal ambitions or indicates a poor understanding of the problem, surface-level research, setting irrelevant goals, or not covering obvious risks from the beginning.

The test is simple: for every component of your system, can you explain why a simpler alternative would not work? If you cannot, the simpler alternative probably would, and you should use it.

Maintenance in practice: The Merehaven Market story continues

A year after deployment, the Merehaven Market demand forecasting system faces the following maintenance challenges:

1. A new product category (prepared meals) was added. The model has no historical data for these products. Fix: use the cold-start heuristic from Chapter 6 (predict based on similar categories) until enough data accumulates for model-based predictions.

2. A data source migration changed the format of promotional event data. Fix: update the feature pipeline’s parser. This was caught by the data quality monitor (Chapter 14) before it affected predictions.

3. The original ML engineer left the company. Fix: the design document, runbooks, and decision logs allowed the replacement engineer to understand the system within two weeks. Without documentation, this would have taken months.

4. A stakeholder requested the model be extended to predict demand at the hour level (not just daily). Fix: the design document’s antigoals explicitly stated that hourly prediction was out of scope. The stakeholder’s request was redirected to a new project with its own design document.

The maintenance design makes such changes manageable through documentation, monitoring, modular contracts and explicit recovery ownership.

A final thought experiment

Consider two ML systems. Both achieve the same prediction accuracy. Both serve the same business function.

System A was built by a single brilliant engineer. It uses a custom loss function, a novel training procedure, and an intricate feature pipeline. The code is dense and uncommented. There is no design document. The engineer understands every detail. No backup owner can reconstruct it.

System B was built by a team of three. It uses standard tools, a well-known loss function, and a straightforward feature pipeline. The code is clean, commented, and tested. There is a design document, runbooks, and a decision log. Any team member can understand and modify the system.

Which system is better? The answer depends on the time horizon. System A might be faster to build. But System B is the one that will still be running three years from now, long after the brilliant engineer has moved on.

The goal of ML system design is to build System B.

Dossier check

Before leaving this chapter, record the assumption that matters most to give every component an owner, the evidence that tests it, the owner who can accept the result, and the recovery route if the assumption fails. A design is incomplete when any of those four fields is blank.