Insight Editions · No. 40
Prompt Engineering for LLMs
The Art and Science of Building Large Language Model Applications
John Berryman and Albert Ziegler
John Berryman and Albert Ziegler explain how to design, test, and integrate prompts, model outputs, and application components into large language model applications.
A study guide to Prompt Engineering for LLMs by John Berryman and Albert Ziegler. Shared free for personal reading; not for sale and not endorsed by the authors or publishers. Please read and buy the original.
Overview
| Core claim | Scope | Key figures | Best for |
|---|---|---|---|
| An LLM is a text completion engine that mimics its training data; prompt engineering is building the whole application that turns a user's problem into a document the model can complete, and the completion back into an answer or action. | Preface plus 11 chapters in three parts: foundations (1-4), core techniques (5-7), expert craft (8-11). | John Berryman and Albert Ziegler, early and founding engineers on GitHub Copilot; O'Reilly, 2024. Cases from Copilot, Codex, ChatGPT, ReAct and Reflexion. | Application engineers and dedicated prompt engineers who can program and call an API; architects deciding how LLM features are built and tested. |
Each chapter keeps the authors' order, examples, tables and rules, then adds boxed October 2026 updates, because much of the book was written before reasoning models, the Responses API, MCP and today's agent frameworks. Critical lenses flag code bugs, dated claims and missing caveats.
The Book in Brief
Spine of the book.
- Thesis. "At their core, LLMs are just text completion engines that mimic the text they see during their training." Everything follows from that line: shape the prompt to resemble training documents, use clear language and existing patterns, and do not drown the model in superfluous content.
- Why it matters (Preface). In 2019 a coder with a question searched the web and often had to post on a Q&A forum and wait; now an assistant comments on the code in front of you or pairs with you. LLMs generate, answer, extract tables, summarise, classify and translate "many orders of magnitude faster" and never take a break.
- Audience. Application engineers of every kind, since LLM calls will soon fill every codebase, and especially prompt engineers, who convert problems into prompts and completions back into value. Prerequisites: programming, APIs, and empathy for how the model "thinks".
- Part I, Foundations (Ch 1-4). History from Markov models to GPT-3; how LLMs tokenise, sample and read one pass, left to right; how RLHF and ChatML turned completion models into assistants; and the application loop that carries a problem into the model's world and back.
- Part II, Core techniques (Ch 5-7). Static and dynamic content, retrieval and summarisation; assembling a prompt with an introduction, context, refocus and transition while avoiding the Valley of Meh; then taming the output with stop sequences, logprobs and the choice of model or fine-tune.
- Part III, Expert craft (Ch 8-11). Tools and reasoning for conversational agents; workflows that trade generality for strength; evaluation offline and online, including the SOMA judge; and a look ahead ending in five rules of LLM empathy.
- What has changed since. Reasoning models (from Sept 2024) made "think step by step" prompting unnecessary or harmful on some models, context engineering replaced much of the template talk, structured outputs fixed tool-argument errors, and MCP standardised tool serving. The core principle has held up.
Figure 0. The book's argument from the completion-engine principle through the application loop, prompt content and assembly, output control, agency and workflows, to evaluation and the closing lesson of empathy.
Chapter Map
| Ch | Title | In one line |
|---|---|---|
| 1 | Introduction to Prompt Engineering | From Markov models to GPT-3; prompt engineering is the whole app; five levels of sophistication |
| 2 | Understanding LLMs | Mimicry, tokens, one token at a time, temperature, one left-to-right pass; order matters |
| 3 | Moving to Chat | RLHF in three stages; ChatML; the prompt engineer as playwright |
| 4 | Designing LLM Applications | The loop; four prompt criteria; Little Red Riding Hood; retrieve, snippetise, score, assemble |
| 5 | Prompt Content | Instructions and few-shot risks; Chekhov's gun; BM25 and embeddings; minimal RAG; summaries |
| 6 | Assembling the Prompt | Prompt anatomy and the Valley of Meh; document archetypes; snippet aims; assembly engines |
| 7 | Taming the Model | Completion anatomy; stop sequences; logprobs for quality, classes, surprise; model and fine-tune choice |
| 8 | Conversational Agency | Tool calling as completion; definition guidelines; CoT and ReAct; agent loop; assisted agency UX |
| 9 | LLM Workflows | Tasks with fixed I/O; pipeline, DAG, cycle; Shopify promoter; Reflexion; LLM-driven routing |
| 10 | Evaluating LLM Applications | Evaluation first; example suites; sample sources; gold, functional, SOMA; A/B metrics |
| 11 | Looking Ahead | Multimodality, artifacts, rising intelligence; two lessons; five rules of empathy |
- John Berryman. Founder and principal consultant of Arcturus Labs (LLM application development); an early engineer on GitHub Copilot, working on completions and chat. Before that a search engineer: a next-generation search system for the US Patent Office, search and recommendations at Eventbrite, GitHub code search; coauthor of Relevant Search (Manning), which shows in the retrieval chapters.
- Albert Ziegler. Founding engineer of GitHub Copilot, where he designed its prompt engineering system; now Head of AI at XBOW, an AI cybersecurity company. His Codex encounter in mid-2020 opens Chapter 1.
1. Introduction to Prompt Engineering
An LLM does one thing: it predicts the next word of a document. Berryman and Ziegler set out to strip the magic from that fact and build on it. The chapter traces the line from Markov models through seq2seq and attention to GPT-3, the moment when changing the input became a way to program the model, and then defines prompt engineering broadly as the design of the whole application that turns a user's problem into a document and the model's completion back into an answer or an action.
ChatGPT launched in late November 2022 and reached an estimated 100 million monthly users by January 2023, the fastest-growing consumer application ever; TikTok took 9 months and Instagram 2.5 years to reach the same mark. The authors, writing in 2024, expect daily conversations with LLMs: IT support, the corner ATM, robo-diallers, news curation, email drafting, travel and shopping agents. They quote Arthur C. Clarke ("any sufficiently advanced technology is indistinguishable from magic") and then promise to dispel the magic: however humanlike the output, an LLM is a model that predicts the next word in a block of text, and the way you steer it is by crafting the prompt, the block of text it completes.
LLMs are magic: two Copilot stories
From research demos to product surfaces.
Both authors were early research developers on GitHub Copilot. Albert Ziegler was on the founding team; John Berryman joined as Ziegler moved to longer-horizon research.
Codex loop Ziegler met.
fn; Copilot supplied the signature fn number_to_string(number: i32) -> String. He steered with comments ("split up the input number into groups of three digits") and Copilot taught him vectors and loops as it went. Twenty minutes in, a two-line comment plus for produced 30 lines of code that compiled and ran; the only bug turned 5,034,012 into "five thirty four thousand twelve million", easy to fix. In 40 minutes he had nontrivial code in an unfamiliar language, a job he guesses would have taken hours alone.How we got here
How we got here.
What a language model is. Its primary goal is to predict the probability of the next word. The authors ask ChatGPT to define itself (Figure 1-1) and point to the predictive-text bar on an iPhone keyboard (Figure 1-2, Berryman pointing at his phone). That bar, at the time of writing, ran on a Markov model of language introduced in 1948, useful enough that most people never notice it. The puzzle the chapter answers: if next-word prediction is so modest, why did it take over the world?
What a language model is.
Seq2seq (Google, 2014). The strongest models of 2014 were recurrent neural networks in the sequence-to-sequence design. They read one token at a time and update an internal state, so in principle they handle arbitrarily long text, and with specialised training they did classification, entity extraction, translation and summarisation. Their weakness was an information bottleneck. The encoder compresses the whole input into one fixed-size hidden state, the thought vector, which the decoder must work from. Long inputs overflow it, and the decoder "forgets".
- Step 1. Source tokens enter the encoder one at a time, become embeddings, and update its state.
- Step 2. The final state is packaged as the thought vector and handed to the decoder.
- Step 3. A special start token tells the decoder to begin.
- Step 4. Conditioned on the thought vector, the decoder updates its state and emits a target-language token.
- Step 5. That token feeds back as the next decoder input, looping steps 4 and 5.
- Step 6. The decoder emits an end token. Everything it knew had to squeeze through one vector.
Book Figure 1-3. Seq2seq translating "Look at the pretty bird" into Spanish. Numbers match the six steps above; the single arrow labelled "thought vector" is the bottleneck.
Attention (2015). The paper "Neural Machine Translation by Jointly Learning to Align and Translate" kept every encoder hidden state, one per token, and let the decoder "soft search" over all of them. English-to-French quality rose markedly. The soft search became known as the attention mechanism.
The transformer (2017). Google Research's "Attention Is All You Need" kept the encoder-decoder outline but removed all recurrence and relied entirely on attention. It modelled training data far better, at one cost: where seq2seq took arbitrarily long input, the transformer takes a fixed, finite sequence. Because GPT descends directly from it, that limit, the context window, is the one practitioners have pushed against ever since.
Book Figure 1-4. The original transformer. GPT keeps only the right-hand decoder stack (masked attention, feed-forward, add and norm, repeated N times).
GPT enters the scene
GPT line milestones.
GPT (2018), from "Improving Language Understanding by Generative Pre-Training", was nothing architecturally new: a transformer with the encoder ripped off, decoder only. The standard recipe still applied: pre-train on unlabelled text, then modify and fine-tune for one task (classification, document similarity, multiple-choice questions). Once fine-tuned, a GPT was good only at that one task.
GPT-3 (2020) added another order of magnitude. "Language Models Are Few-Shot Learners" showed that given a few examples of a task (few-shot examples) the model reproduces the pattern and performs almost any language task, often very well. Changing the input conditioned the model to do the task: this was the birth of prompt engineering.
ChatGPT (November 2022) ran on GPT-3.5. GPT-4 (March 2023) was rumoured to be another order of magnitude larger, and rivals followed: Llama (Meta), Claude (Anthropic), Gemini (Google). Quality kept rising, and the same quality kept arriving in smaller, faster models.
| Model | Date | Params | Training data | Compute |
|---|---|---|---|---|
| GPT-1 | Jun 2018 | 117M | BookCorpus, 4.5 GB (7,000 books) | 1.7e19 FLOP |
| GPT-2 | Feb 2019 (full: Nov) | 1.5B | WebText, 40 GB (Reddit-upvoted pages) | 1.5e21 FLOP |
| GPT-3 | May 2020 | 175B | 499B tokens: Common Crawl, WebText, Wikipedia, books | 3.1e23 FLOP |
| GPT-3.5 | Mar 2022 | 175B | Undisclosed | Undisclosed |
| GPT-4 | Mar 2023 | 1.8T (rumoured) | ~13T tokens (rumoured) | ~2.1e25 (est.) |
Book Table 1-1. Each generation multiplies parameters, data and compute (GPT-2 drew on 8 million documents from 45 million pages; GPT-3's Common Crawl slice was 570 GB); compute alone rose about a million-fold from GPT-1 to GPT-4.
Separate facts from rumour in model tables.
What prompt engineering is
An LLM completes text. The input is the prompt, a document the model is expected to complete. In the narrow sense, prompt engineering is crafting the prompt so that its completion contains the information that solves the problem. The book means something larger: the whole LLM application, in which the prompt is built and the answer interpreted programmatically. The user states a problem; the application builds a pseudo-document; the LLM completes it; the application parses the completion and reports back or acts. The craft is translating between two very different domains, the user's problem space and the LLM's document space.
The five levels of sophistication the book covers, from a thin wrapper to agency. Each adds more programmatic work between the user and the model.
- Level 1, thin wrapper. Using ChatGPT you write the prompt almost directly; the app only wraps the thread in ChatML (Chapter 3). The first Copilot passed little more than the current file to the model.
- Level 2, modified and augmented input. A support hotline transcribes speech and adds past transcripts or documentation. Copilot's completions improved considerably once it added snippets from the user's neighbouring tabs, open because the user was consulting them. Bing chat pulls search results into the prompt so it can discuss events after training and hallucinate less.
- Level 3, statefulness. A chat app must replay prior exchanges faithfully each turn without overfilling the prompt or adding distracting content, so it may drop the oldest turns, drop less relevant content or summarise.
- Level 4, tools. The model reaches into the world through API calls that read or change assets. "Send Diane an invitation to a meeting on May 5" means finding Diane in contacts, checking her calendar and sending the invite. The open questions (which tool, used how, how results return, what to do on errors) are the subject of Chapter 8.
- Level 5, agency. The application decides how to reach broad goals. AutoGPT will chase a goal through many steps, but unless the goal is tightly constrained it fails more often than it succeeds. Chapters 8 and 9 give the authors' take.
2. Understanding LLMs
To write clever prompts you need to know how an LLM "thinks". The chapter peels the model like an onion: a trained mimic of documents, a reader of tokens rather than letters, a generator that commits to one token at a time, a sampler governed by temperature, and finally a transformer made of thousands of identical minibrains passing notes left to right. Each layer yields a practical rule, and the deepest one explains why the order of a prompt matters.
The authors stay practical: no matrix multiplications or activation functions (for those, they point to The Illustrated Transformer), only what a prompt writer needs.
What are LLMs?
LLM as conditioned completion.
At the outermost layer an LLM is text in, text out: a prompt goes in, a completion (or response) comes out; "One, Two," returns " Buckle My Shoe". Untrained, it emits a random jumble of Unicode. Training is beyond most teams, so applications use pre-trained foundation models, sometimes lightly fine-tuned.
A model learns from a training set of documents, typically a mix of books, articles, Reddit-style conversations and GitHub code, and is trained so that given the beginning of a training document, it produces the most likely continuation. In other words, models mimic.
Book Figure 2-2. The Pile, an open training set. Largest slices: Pile-CC web crawl 18.1%, PubMed Central 14.4%, Books 12.1%, OpenWebText 10.0%, ArXiv 9.0%, GitHub 7.6%, FreeLaw 6.1%, Stack Exchange 5.1%.
The goal is instead to apply the patterns in the data, especially logical and reasoning patterns, to any prompt. Rote memorisation is a defect, called overfitting. Architecture (which pushes the model to abstract) and training (diverse, non-repetitive data, scored on unseen data) both fight it. It still happens, so a model that solves a problem seen in training may fail on a similar unseen one.
Core heuristic of the book.
Completing a document
Search versus LLM.
Take "Yesterday, my TV stopped working. Now, I can't turn it on at". Candidates are "y2ior3w" (a cat on the keyboard), "Thursday." (garbled rewriting) and "all." None is impossible, but almost every LLM picks "all." Extend the text and ask again. Trained on narrative prose, a model would favour "This is why I chose to settle down with a book tonight." A story would not open with "Shall we watch the game at your place instead?" without a quotation mark, but add emails and chat transcripts and that becomes plausible. The real output from OpenAI's text-davinci-003 (a GPT-3 variant) was two newlines and then "First, try unplugging the TV from the wall and plugging it back in.", mimicking the customer-service conversations in its data. The better you know the training data, the better you can predict the output, even when vendors keep the data secret.
Human thought versus LLM processing
Human versus LLM processing.
Humans write as part of a larger process. Blogging about the "The Rest Is History" episode on the Hundred Years' War, you would stop at "(listen on acast at http://" and search for the link, or delete the bracket if you could not find it. The model cannot search or edit, so it guesses, and a raw LLM adds no doubt or disclaimer, because from its side everything is a guess. Its guesses keep the right shape: an invented Social Security number looks like one, an invented podcast URL looks like one. text-curie-001 produced a URL ending "the-Hundred-Years-War-1411-1453-with-dr-martin-kemp". There is an Oxford art historian named Martin Kemp, but he never discussed the war on that podcast (Figure 2-3: people's language reflects reality; models' language reflects people).
Hallucinations and truth bias
Hallucination path.
Hallucinations are factually wrong but plausible information produced confidently (the authors note that confabulation is the closer human analogue). To the model they are ordinary completions, so "Don't make stuff up" barely helps. Instead, make the model produce checkable background: its reasoning, a calculation you can rerun, a source link, searchable details. "An English king married his cousin" is hard to check; "George IV married Caroline of Brunswick" is easy. The rule: "Trust but verify", minus the trust.
Truth bias means a model assumes its prompt is true, because documents that open with false claims and then retract them are rare. Use it: instead of "Pretend that it's 2030 and Neanderthals have been resurrected", write "It's 2031, a full year since the first Neanderthals were resurrected" (a make-believe prompt). Fear it, too: a bug in programmatic prompt assembly that inserts nonsense will not get a raised eyebrow. You are responsible for a prompt that needs no correction.
How LLMs see the world: tokens
Token world.
Even advanced models in autumn 2024 could be tripped by "How many Rs in 'strawberry'?", because they do not read characters. Humans do not either; we read words, which is why scrambled letters inside intact word boundaries stay readable while shifted boundaries do not (Figure 2-4). An LLM reads tokens, multi-letter chunks typically three to four characters long, with longer tokens for common words. The token set is the vocabulary; a tokenizer turns text into token numbers and back. Many tokenizers, including OpenAI's, have tokens that begin with a space.
| Difference | What happens | Prompting consequence |
|---|---|---|
| 1. Deterministic tokenizers | "ghost" is one GPT token; the typo "gohst" becomes g-oh-st, so typos stand out. Models are still fairly robust because typos fill training data. | Typos cost little but are visible to the model. |
| 2. No slowing down to examine letters | Reversing letters fails in both directions (Figure 2-6); "European country starting with Sw" fails in two different ways even though [ Sweden], [ Switzerland] and [ Somalia] are single tokens (Figure 2-7). | Move sub-token work to pre- or post-processing: use the LLM as an oracle for candidate lists, then filter with code. |
| 3. Text is not seen | No sense of letter shapes or ASCII art beyond memorised cases; accented text (díffícúlty) costs processing power. Capitals tokenise differently: "strange new worlds" is [str][ange][ new][ worlds], but in caps [STR][ANGE][ NEW][ WOR][L][DS]; "gone" is one token, "GONE" two. | Avoid asking for constant case conversion; small models such as text-babbage-001 garble it (Figure 2-8). |
Counting tokens
Relative tokens per word by content type.
Every model has one fixed tokenizer; you cannot mix and match. Libraries such as Hugging Face tokenizers and tiktoken let you run it, mostly to count. Token count drives everything: reading time scales roughly linearly with prompt tokens, generation time linearly with output tokens, and so does compute, which is why APIs charge per token. At the time of writing a dollar bought 50,000 to 1,000,000 output tokens. The count also governs the context window: prompt plus completion must fit. Windows run to thousands of tokens, several to hundreds of A4 pages, yet you will always be tempted to overfill them.
- English prose. About 4 characters per token with the GPT tokenizer; newer tokenizers do slightly better.
- Other languages. Fewer characters per token, because tokenizers are optimised for the dominant language of their training data, usually English.
- Random digits. A little over 2 characters per token; random alphanumerics such as keys, under 2.
- Rare characters. Worst of all: the Unicode smiley (U+263A) costs two tokens.
- Special tokens. Most vocabularies include an end-of-text token, appended to every training document; when the model emits it, the completion stops.
One token at a time
A pass through the model maps many tokens to one: the statistically most likely next token (at temperature 0). That token is appended and the model runs again. This is autoregressive generation, like repeatedly tapping the middle suggestion on a phone keyboard.
Book Figure 2-14. "One, Two," tokenised into four minibrains; guesses over the prompt are thrown away, the last guess is sampled ([ Buck] 51%, [ Three] 31%), and a new minibrain is placed on [ Buck] to produce [le] (94%).
Three consequences follow. No extra time: the model must emit a token every step and cannot stall to think. No backtracking: once a token is out, the model is committed, and it rarely writes explicit retractions because finished human documents rarely contain them (the authors demonstrate with a mock correction of "takebacks"). Recognising mistakes and backtracking must be supplied by the application designer. Self-reinforcing patterns: having made a pattern by chance, continuing it is always likelier than breaking it.
Temperature and probabilities
The model actually computes a probability for every token in the vocabulary, then sampling chooses one. APIs often return logprobs, natural logs of probabilities, never above 0; expect the top token to sit between -2 and 0. Requesting logprobs=3 returns the three most likely tokens at each step. Temperature (t ≥ 0) sets how often a less likely token is picked:
p(tokeni) = exp(logprobi / t) ÷ Σj exp(logprobj / t)
| Temperature | Behaviour | Use it when |
|---|---|---|
| 0 | Always the top token; near-deterministic (rounding noise can shift probabilities by several points) | Correctness is paramount; repeatability matters |
| 0.1 to 0.4 | Near-tied runners-up get a small chance | A few alternatives to filter, or one slightly more colourful answer |
| 0.5 to 0.7 | Chance matters; clearly worse tokens sometimes win | Many independent solutions, 10 or more |
| 1 | Mirrors the training distribution: if [ Buck] follows "One, Two," 51% of the time, you get it 51% of the time | Output statistics (such as list length) should match real documents |
| Above 1 | More random than the training data | Rarely; errors compound |
High temperature acts like alcohol. Temperature affects only the final step that turns probabilities into a choice, not the computation of those probabilities. So the model sees its own fresh errors as a pattern, imitates them, and the temperature adds more; in Figure 2-13 item 3 starts legible and ends in non-words. Temperature also governs when lists stop: at 0 the model keeps choosing "continue" once past the obvious stopping points (3, 4, 5, then 10); at 1 it stops with the model's own probability, giving training-like lengths.
- High temperature. More alternatives; generation properties (such as list length) match the training distribution.
- Low temperature. More correct solutions; more replicable output.
- Beam search. Looks a few tokens ahead so a likely token does not lead into a dead end; more accurate but rarely used in applications because of its time and compute cost.
The transformer architecture: minibrains
At the centre are not one brain but thousands of identical minibrains, one on each token, together forming the transformer. Each is told its token and position, then thinks for a fixed number of steps called layers (the figure shows four; GPT-3 has 96, GPT-4-class models over 100). In every layer but the last it shares intermediate results with minibrains to its right; in the last it predicts the token to its right. Over the prompt those predictions are thrown away and only the shared results matter; at the rightmost position the prediction is the whole point. Each new token gets a new minibrain, and earlier states are cached and reused.
Attention is the regulated note-passing between them. Each minibrain submits questions (the one on [my] asks "Who is talking?") and offers (the one on [Susan], after "Hello, I'm Susan", offers "The person talking right now is Susan"). Each question is matched with its best-fitting answer and only the asker sees it. The real "language" is long vectors of numbers each model invents in training. Masking adds one rule: only minibrains to the left may answer, and none learns whether its answer was used.
Information flow in a generative transformer. Thinking aloud, by writing tokens, is the only route from a high layer back to a low one, which is why chain-of-thought works.
- Backward. Minibrains look only left, as far as they like, never forward; hence unidirectional transformers. Easy to train and run, with large consequences for how information is processed.
- Downward ("dumbward"). A minibrain at layer i hears only what others knew at layer i, so a reasoning chain is at most i steps deep and no later insight can return to a lower layer. The sole exception is the output token, which seeds the first layer of the next minibrain. This thinking aloud ("How could I know what I'm thinking before I've heard what I'm saying?") underlies chain-of-thought prompting (Chapter 8).
The left-and-below dependency lets prompt states be computed in parallel as a triangle (Figure 2-15), one reason transformers train efficiently. Generation breaks the triangle: each token must finish every layer before the next can begin. So prompt tokens are processed about an order of magnitude faster than generated ones.
- Document completion. LLMs are document completion engines.
- Mimicry. They imitate the documents they saw in training.
- One token at a time. They cannot pause or edit what they have produced.
- One pass. They read the text once, from beginning to end.
3. Moving to Chat
A base model completes any document, including dangerous ones, and answers a question with more questions. RLHF turns it into a helpful, honest, harmless assistant through four models, three datasets and three fine-tuning stages. Chat models then add ChatML, an unambiguous transcript format that sharpens obedience to the system message and blocks one route of prompt injection. Underneath, the model is still completing a document; the prompt engineer is now the lead playwright of a script that the end user never sees.
Why base models are hard to use
Why base models are hard.
A base model has only been pre-trained, so it mimics the light and the dark side of the internet alike: "This is a recipe for Sicilian Lasagna:" yields lasagna, and a prompt for methamphetamine steps yields those too. It also only completes documents. Asked "What is a good dish for chicken?", it continues the pattern: "What is a good dish for beef? ... pork? ... lamb? ... rice?" (Table 3-1). A trained assistant answers "A great dish for chicken is chicken piccata" with a recipe (Table 3-2). The wish list is long: polite, direct but not curt, thorough but not chatty, truthful, not prone to hallucination, easy to customise (a medical doctor who talks like a pirate) but hard to jailbreak, and able to run code and call APIs.
Reinforcement learning from human feedback
RLHF pipeline.
RLHF uses human preferences to change model behaviour. Google (Gemini), Anthropic (Claude) and OpenAI (GPT) all built RLHF chat models. The chapter follows OpenAI's March 2022 paper "Training Language Models to Follow Instructions with Human Feedback" (InstructGPT): four models, three training sets, three fine-tuning procedures.
Start: the base model. In 2023 OpenAI's strongest base model was davinci-002, presumably trained on GPT-3-like data. Ask it to finish a news article and it follows the story, then drifts into increasingly bizarre hallucinated details. Hence alignment: fine-tuning so completions match user expectations. Anthropic's 2021 paper "A General Language Assistant as a Laboratory for Alignment" defined HHH:
- Helpful. Follows instructions, stays on track, gives concise and useful answers.
- Honest. Does not present hallucinations as fact; signals uncertainty when unsure.
- Harmless. Avoids offensive content, discriminatory bias and information dangerous to the user.
The InstructGPT recipe behind ChatGPT. Human effort concentrates in the first dataset; later datasets are mostly model-written and only ranked or scored.
Step 1, supervised fine-tuning (SFT). About 13,000 handcrafted transcripts of a person and an HHH assistant. Training is ordinary next-token prediction, only far smaller and shorter than pre-training. The SFT model obeys instructions much better, but quality is not there yet, and it has a problem with lying.
Step 2, the reward model (RM). In reinforcement learning an agent acts in an environment to maximise reward; here the agent is the LLM, the environment is the document, the action is the next token and the reward is a score of subjective quality. The SFT model answers representative prompts four to nine times each at high temperature; human judges rank the answers; about 33,000 ranked documents yield an order of magnitude more pairs, since the RM compares two documents at a time. The RM must be at least as capable as the SFT model, so it starts as a copy of SFT and is trained to output a number instead of a token.
Reward model step.
Step 3, the RLHF model. Start again from SFT; feed it about 31,000 prompts; let the RM score each completion and fine-tune against the score. Pure score-chasing cheats: the model finds states that please the RM but no longer produce normal text. Proximal policy optimisation (PPO) allows weight changes that raise the score only while output stays close to the SFT model's.
Honesty needs the model's own answers.
- Avoiding idiosyncrasy. OpenAI hired 40 part-time workers for SFT writing and RM ranking, screened to limit quirks. Ranking dilutes any one person's style further, and judges were checked for agreement, so the RM encodes an aggregate notion of HHH.
- Bang for the buck. Only the 13,000 SFT documents are fully hand-written; the 33,000 RM documents are mostly SFT output that humans merely order; the 31,000 RLHF documents are almost entirely model-generated.
- The alignment tax. HHH is not the same as smart, so RLHF can make a model friendlier but dumber on some tasks. OpenAI reduces the tax by mixing original pre-training data into RLHF training.
From instruct to chat
From instruct to chat to tools.
Instruct models, OpenAI's first RLHF models, treated every prompt as a request. A base model reads "What is a good indoor activity for a family of four?" as an essay opening and continues "And why are family activities so important to your children's development?" InstructGPT was trained on prompts like these (Table 3-4, from the paper's Table A.2.1):
| Use case | Training prompt |
|---|---|
| Brainstorming | What are 10 science fiction books I should read next? |
| Classification | {java code} What language is the code above written in? |
| Rewrite | Translate this sentence to Spanish: <English sentence> |
| Open QA | Who built the Statue of Liberty? |
| Summarisation | {news article} Tl;dr: |
| Chat | "The following is a conversation with an AI assistant..." then Human/AI turns ending in a request to cancel a subscription and "AI:" |
The family question now gets board games (Scrabble, Monopoly, Risk; Jenga or Twister for younger children; cooking together). Two problems remain. Nothing in the prompt says "now it's your turn": perhaps the user wanted elaboration. And the alignment-tax fix, mixing completion samples with instruct samples, trains both behaviours on ambiguous prompts. The answer is a chat model.
Chat models and ChatML
Chat roles.
ChatML, OpenAI's key innovation, marks up a transcript. Each message opens with <|im_start|> plus a role (system, user or assistant) and a newline, and closes with <|im_end|>. The system message is not dialogue; it sets expectations, usually addressing the assistant in the second person ("You are a software assistant, and you provide concise answers to coding questions"). The book's example is a sarcastic, emoji-loving assistant answering a user who ran the fork bomb :(){ :|:& };: and wonders why everything is slow. Three benefits follow:
- Unambiguous turn-taking. In the Jeeves example (system: a very proper British valet, answer in one sentence; user: the family question),
<|im_end|>shows the user has finished and OpenAI injects<|im_start|>assistant. The reply: "Indeed, a delightful indoor activity for a family of four could be a spirited board game night..." - Strong obedience to the system message. Drop "one sentence" and the answer gets chattier. Teams put "rules of the road" there (stay on software topics; politely disengage from arguments). Telling a well-trained model not to be rude adds little beyond its training, but the system message can loosen norms: the authors suggest a profane Rick Sanchez persona giving scientifically sound medical advice.
- A defence against prompt injection. The tags are reserved tokens. Through the chat API, the typed string "<|im_start|>" becomes six ordinary tokens (<, |, im, _start, |, >), so a user cannot forge an assistant or system message.
The changing API
The chat API launched in March 2023 and by July 2023 carried 97% of OpenAI API traffic (per OpenAI's GPT-4 API general-availability notice). The book shows a minimal Python call to gpt-4o with a system and a user message, and the JSON reply (a joke about atoms "making up everything", finish_reason: "stop", 11 prompt tokens, 12 completion tokens). No ChatML is visible: conversion happens behind the API, and special tokens typed by users are stripped (Figure 3-1).
| Parameter | Purpose | Notes |
|---|---|---|
| max_tokens | Cap output length | |
| logit_bias | Raise or lower odds of given tokens | Silly example: bias "#" to change how much code is commented |
| logprobs | Return log probability of each chosen token | Shows confidence across the answer |
| top_logprobs | Top candidates per position with logprobs | Shows what else the model might have said |
| n | Parallel completions | n = 128 (max) takes little longer than n = 1 |
| stop | Strings that end generation | Cut off unhelpful trailing patterns |
| stream | Send tokens as generated | Better UX: users read while it writes |
| temperature | "Creativity" | 0 can repeat itself; near 2 often nonsense; about 1.0 is the sweet spot |
What chat costs you
- The alignment tax. A July 2023 Stanford paper, "How Is ChatGPT's Behavior Changing Over Time", found GPT-4 getting worse at some tasks. Watch for regressions when models are tuned.
- Control. Early chat models were patronising, and all are chatty, which hurts when you must parse code out of commentary. A completion prompt ("The following is a program that implements the quicksort algorithm in python:" plus an opening code fence) guarantees the next tokens are code; set stop to the closing fence and there is nothing to parse.
- Human diversity. RLHF makes models uniform and polite; raw internet text holds the full, sometimes rude, repertoire, a "digital encoding of the zeitgeist". Synthetic sample data, a doctor brainstorming options or police discussing illegal activity can all suffer from an assistant's refusals, though the authors insist misuse safeguards remain essential.
Beyond chat to tools
About half a year after chat, OpenAI added tool calling: the model requests an external API call, the application executes it and inserts the result into the next prompt. Chapter 8 covers it. The point here: LLMs are still document completion engines; with chat the documents are ChatML transcripts, and with tools those transcripts carry special syntax for calls and results.
Prompt engineering as playwriting
Two conversations run in parallel: the one the end user has with the assistant, and the transcript the application sends the model, full of material the user never sees. Asked "How should I test this code?", the application must work out what "this code" is and may fabricate user or assistant turns that carry it. The authors' metaphor is a play: characters are the roles (user, assistant, system, tool), the script is the prompt, and there are several playwrights.
| Playwright | Writes | Example in Table 3-6 |
|---|---|---|
| API provider | ChatML markup; folds tool definitions into the system message | <|im_start|>system, <|im_end|> |
| Prompt engineer | Structure, boilerplate, system message, hidden context, some assistant lines | "You are an expert developer who loves to pair programs."; the highlighted Python 2 loop |
| Human user | The problem that is the play's theme | "This code doesn't work. What's wrong?" (all the user typed) |
| LLM | The assistant's speaking parts | "You appear to be using an outdated form of the print statement. Try parentheses" |
| External APIs | Retrieved content shoved into the script | Documentation search results |
The prompt engineer is lead playwright and showrunner, responsible for a feel-good play rather than a Greek tragedy.
- Sample code. The first snippet calls
client.ChatCompletion.create, which mixes the pre-1.0 and 1.0 OpenAI SDKs and fails as written; the later snippet'sclient.chat.completions.createis correct. The JSON reply names gpt-4o-mini though gpt-4o was requested, and the ChatML sample drops a "<" from one closing tag. - Tool-call example. The model's suggested fix in Table 3-6 repeats
print iwithout the parentheses it recommends. - ChatML as injection defence. Reserved tokens stop forged role tags, but indirect prompt injection through retrieved documents and tool results, which the authors flag only for system messages, works inside the user or tool role too. It remains the top risk in OWASP's LLM list.
- Alignment methods. PPO-based RLHF is now one option among several: direct preference optimisation (DPO, 2023) trains on preference pairs without a separate reward model, AI feedback (Anthropic's Constitutional AI) replaces many human rankings, and the reasoning models of 2024 to 2026 rely heavily on reinforcement learning with verifiable rewards (correct maths answers, passing tests), as DeepSeek-R1 documented in January 2025.
- Roles. OpenAI's API now prefers a developer message over system for its reasoning models, part of an instruction hierarchy (platform, then developer, then user, then tool output). The open-weight gpt-oss models (August 2025) use the harmony format, ChatML's successor, with separate channels for reasoning and final answers.
- Endpoints. The Responses API (March 2025) is OpenAI's recommended interface: top-level instructions, typed items for messages, reasoning and tool calls, and server-held conversation state. Chat Completions is still supported, the Assistants API was shut down on 26 August 2026, and Responses drops the
nparameter, so parallel candidates now mean separate requests.
4. Designing LLM Applications
An LLM application is a transformation layer with a purpose. It takes a problem from the user's world, turns it into a document the model can complete, and turns the completion back into a solution. The chapter makes one trip around that loop, then zooms in on the feedforward pass (retrieve, snippetize, score, assemble), zooms out to state, retrieval, reasoning and tools, and ends with how to tell whether any of it works.
The anatomy of the loop
Prompt engineering loop.
The user may be polishing an email, organising group travel, or absent, with a recurring analysis running as new data arrives. The model does one thing: it completes documents, which covers emails, code and stories; chat completes transcripts, and tool use completes transcripts with function-calling syntax. The loop converts the problem into a document, then the completion back into a solution (Figure 4-1). Turning bullets into email prose takes one stateless iteration; a chat assistant runs the loop many times; a travel planner reshapes it over much state, from brainstorming to booking to reminders.
Book Figures 4-1 to 4-3 redrawn as one loop. The application owns every box except the LLM; the tool loop may circle several times before anything returns to the user.
The user's problem
Problems vary in four dimensions: medium (text suits LLMs best), abstraction (higher means harder reasoning), context required (most need retrieval) and statefulness (memory of past interactions and preferences). Table 4-1:
| Dimension | Proofreading | IT support | Travel planning |
|---|---|---|---|
| Medium | Text | Voice over the phone | Website interactions, typed input, API calls |
| Abstraction | Concrete, well defined, small | Large abstract problem and solution space, bounded by documentation | Subjective tastes plus objective constraints, coordinated into one plan |
| Context | Only the user's text | Searchable docs and example support transcripts | Calendars, airline APIs, news, government advice, Wikipedia |
| State | None; every call stands alone | Conversation history and fixes tried | Weeks of planning, several media, aborted branches |
Converting the problem to the model domain
Problem to model domain.
This step is the crux of prompt engineering, and four criteria must hold at once.
- 1. Resemble training data. The prompt must closely resemble documents from the training set.
- 2. Carry the information. It must include everything relevant to the user's problem.
- 3. Point to a solution. It must lead the model to a completion that addresses the problem rather than elaborating it.
- 4. Stop. The completion needs a reasonable end point so generation halts naturally.
Four prompt criteria.
Criterion 1 is the Little Red Riding Hood principle: like the girl who left the forest path and met the wolf, a prompt that strays from familiar training documents gets unpredictable results; familiar prompts give stable completions. Vendors hide their training formats (knowing them helps jailbreakers), but you can just ask: "What types of formal documents are useful for specifying financial information about a company?", then have the model draft one. Completion prompts can mimic programs, news articles, tweets, Markdown or transcripts; inside a fixed ChatML transcript, user messages can still use # for sections, backticks for code, * for lists.
Little Red Riding Hood principle.
Criterion 2 runs from trivial (proofreading) to heavy (travel: preferences, calendars, tickets, news, government advice). After finding content comes finding the best, since loosely relevant content distracts the model, then arranging it logically. Criteria 3 and 4 come free with chat models, which are tuned to answer and stop (you may still curb chattiness). Completion models must be told it is time to answer, and must either be promised a short answer or be set up so a distinctive opening text follows it, which the stop parameter catches.
<|im_end|>. It answered sensibly, signed off "Hope you have a nice day!", then, unable to stop, tried "wonderful", "festive" and every other synonym until the token limit.Verbose completion case.
The homework example (Table 4-2) is a completion prompt for a travel-recommendation app; boilerplate is fixed, user-specific parts are inserted per request.
| Prompt part | Text (abridged) | What it does |
|---|---|---|
| Title | # Leisure, Travel, and Tourism Studies 101 - Homework Assignment | Familiar Markdown document, travel domain (1) |
| Instruction | Each answer a sentence or two | Promises an end (4) |
| Problem/Solution 1 | Top golf destinations: St. Andrews; Pebble Beach; Augusta National | Sets the Problem N / Solution N pattern and a concise, polite voice |
| Problem 2 | Customer wants Pyongyang; State Department "Do not travel to North Korea..."; headlines: ballistic missile, five-day COVID-19 lockdown, Yoon on human rights | User's pick (maybe a drop-down slip) plus retrieved context (2) |
| Cue | "What would you tell the customer?" Solution 2 | Answer starts now (3) |
| Completion | "Perhaps North Korea isn't a great destination right now. But I bet we could find some nice place to visit in South Korea." | Stop text \n# cuts off a confabulated Problem 3 |
Sloppy grammar invites sloppy completions, and without the cue the model would keep confabulating about North Korea. Chat APIs secure criteria 1, 3 and 4 by construction, but you are not off the hook: you own criterion 2 and must shape transcript, system message and function definitions so the model can solve and stop. Exercises (with gpt-3.5-turbo-instruct): drop the cue or question; make Solution 1 longer, shorter, piratical or rude; make the North Korea news increasingly positive; remove the stop word and add a newline to provoke a third or fourth problem; and replace the homework frame with a travel help-line transcript.
Using the LLM to complete the prompt
After completion.
Three choices matter. Size: bigger is usually better, but GPT-4 could cost 20 times as much as gpt-3.5-turbo, sometimes worth it. Latency: early GitHub Copilot used Codex, "small, sufficiently smart, and lightning fast"; users would not have waited for GPT-4. Fine-tuning (GitHub was trying it for less common languages) helps when the model needs non-public information or different behaviour.
Transforming back to the user domain
A completion is a blob of text that usually must be transformed or harvested. Completion-era apps requested a strict format and parsed it; with function calling, the output is a function call (a flight search with date, origin and destination that the app sends to the airline API). A ticket-purchase function acts in the world, after the app checks with the user. The medium may change (speech on a phone line, UI events), and text may be re-presented: Copilot shows grey text accepted with Tab, Copilot Chat a red/green diff.
Zooming in to the feedforward pass
The feedforward pass turns the user's problem into a prompt in four steps, each the subject of later chapters.
- Context retrieval. The most direct context is the user's own text (the help box, the code being edited in Copilot); indirect context comes from documentation search or, for Copilot, other open IDE tabs; the least direct is boilerplate ("This is an IT support request. We do whatever it takes to help users solve their problems."), which introduces the problem and acts as glue between pieces.
- Snippetizing. Extract the most relevant chunks to protect the token budget, and convert formats: transcribe voice, render JSON replies as prose so the model does not echo JSON fragments.
- Scoring and prioritising. GPT-3.5's "measly" 4,096 tokens have grown past 100,000, but irrelevant text still confuses the model. Priorities are integers setting tiers (exhaust a tier before the next); scores are floats ranking within a tier.
- Prompt assembly. Never exceed the budget (you get an error). Fit boilerplate, then the request, then maximum context, eliding less relevant code lines or summarising if needed, in the proper order, so it reads like a training document.
Exploring the complexity of the loop
Complexity grows in four dimensions: state, external content, reasoning and interaction with the world.
State. Copilot code completion is stateless; any chat app must reload the conversation each turn, truncating old exchanges or summarising them when they matter.
External context. Models lack recent and private information; an apology is infinitely better than a confident hallucination, and RAG adds indexed corporate documents, medical records or news to the prompt. A vector store (like Pinecone) is popular, but Elasticsearch-style indexes are simpler and easier to debug.
Hard tasks must be broken down step by step.
Tool usage. Alone, an LLM lives in a closed world. Tools are described as functions (name, arguments, descriptions); the model emits a call, and the application runs the real API and appends the result to the next prompt. "ReAct" (2022) offered search, lookup and finish over Wikipedia, so the model decided when it needed information, an overlap with RAG. Search, weather, email checks and LinkedIn posts are read-only; write tools open pull requests or book travel. Models are probabilistic and often make mistakes, so do not let the application book a trip to Greece just because the user said they would love to visit someday.
Evaluating LLM application quality
Because models err, evaluate constantly: metrics before shipping, telemetry after.
Offline evaluation is harder than online because no customer says "good" or "bad", so you need a proxy. Copilot's team took a few hundred repos with running tests, deleted code fragments, regenerated them and checked the tests still passed. Where no proxy exists, use an LLM as judge, asking "Which version is better?" or scoring a checklist of criteria. Exercise as much of the application as possible: mocking context gathering skips what often matters most and invites a nasty surprise in production.
Online evaluation: measure everything. Thumbs buttons suffer bias (maybe only angry users vote) and low volume, so use implicit signals, such as Copilot's acceptance rate and post-acceptance edits, interpreted carefully (a quick exit may mean success or frustration). Pick a metric tied to productivity: Copilot chose acceptance rate because it correlated most highly with productivity gains; a scheduler should count events created and later edits, not session length.
- Dated specifics. OpenAI retired the Codex API in March 2023 and Copilot now runs on several vendors' models; the "20 times" GPT-4 premium reflects 2023 prices.
- LLM as judge. Presented without its failure modes: Zheng et al.'s 2023 MT-Bench study found position bias, verbosity bias and self-preference, so judges need calibrating against human labels (Chapter 10).
- Write tools. User confirmation is the only safeguard named; tool output can carry injected instructions, worst when the same agent can write.
- Context engineering. Tobi Lütke (19 June 2025) and Andrej Karpathy (25 June 2025) popularised the term for this chapter's feedforward pass; Anthropic's September 2025 guide sets the goal as "the smallest possible set of high-signal tokens".
- Context rot. Chroma's July 2025 test of 18 models found accuracy degrades unevenly as input grows, backing the advice to keep prompts trim.
- Agents. Anthropic's December 2024 "Building effective agents" splits workflows (predefined code paths) from agents (the LLM directs its own tool loop) and says start simple.
- Structured outputs. Since August 2024 OpenAI can force a supplied JSON Schema; gpt-4o-2024-08-06 scored 100% on its schema eval versus under 40% for gpt-4-0613.
- MCP. Anthropic's Model Context Protocol (November 2024) became the standard tool connector and on 9 December 2025 moved to the Linux Foundation's Agentic AI Foundation, with 97 million monthly SDK downloads.
5. Prompt Content
LLMs digest messy text that rigid algorithms cannot, but only if you supply it. Static content defines and clarifies the task for every user; dynamic content, gathered at request time, describes this user and this problem. The chapter covers instructions and few-shot examples with their three dangers, then finding, scoring and retrieving context, a minimal RAG app, and summarising what will not fit.
Most book recommenders use collaborative filtering, comparing one user's behaviour with everyone's. An LLM adds near-human common sense over text. In Figure 5-1, told only that the user last read Moby Dick and Huckleberry Finn, ChatGPT suggests To Kill a Mockingbird, a reasonable traditional-system answer. Add that the user is 38, watches Vietnam War documentaries and loved a vacation in Thailand, and it recommends Tim O'Brien's The Things They Carried and Alex Garland's The Beach. LLMs are great at processing a great variety of messy textual information, but it's your job to provide that information.
Sources of content
Sources of prompt content.
Gather everything first ("there are no bad ideas"); Chapter 6 whittles it down. The key split is static content (always the same) versus dynamic content (different every time). Static content explains the task and gives instructions: "Which book do you think I should read next? I mean for fun, not what kind of textbook." The second sentence is a clarification. Dynamic content describes the object of the question: "...The last book I read was 'Moby Dick,' btw." is context. The line can blur: "I want a proper book, not a self-help book" could be either. The test is how the application is built: hardcoded text is static and defines the problem; strings from variable sources are dynamic. A designer's decision to steer away from self-help is clarification; a user's disdain for self-help mined from their message history is context.
Static content: clarifying the question
Clarifying the question.
Human misunderstandings get fixed on the fly; in a programmatic call a misunderstanding often means complete failure. Clarification also buys consistency (same processing, same criteria), which lets you optimise the app, helps users learn it, and underpins trust.
Explicit clarification says what you want: "Use markdown", "Don't use hyperlinks", "Don't refer to dates after your knowledge cutoff of 2024-03-03". Industry prompts carry long lists. Table 5-1 gives rules jailbreaker Marvin von Hagen extracted from Bing Chat, codename Sydney (unconfirmed as genuine): identify as "Bing Search", not an assistant, and never disclose "Sydney"; be informative, visual, logical, actionable, positive and engaging; use long formats except tables, and no images; always search the web when the user seeks information; no creative content about influential politicians, activists or state heads; decline to reveal or change the rules, which are "confidential and permanent".
- Dos, not don'ts. Instead of "Thou shalt not kill", try "Thou shalt preserve life."
- Give a reason. "Thou shalt not kill since the act of killing disrespects the other person's right to life."
- Avoid absolutes. "Thou shalt kill only rarely...and make sure it's really appropriate!"
Three instruction rules.
Not all models obey well. RLHF chat models do better, and explicit instructions belong in the system message, which they are trained to obey, but no model is perfectly compliant.
Few-shot prompting
Zero versus few-shot.
Few-shot prompting adds examples, an implicit instruction. The term comes from "Language Models are Few-Shot Learners": a model extrapolates from a few "shots" where classical ML needs many; instructions alone make a zero-shot prompt (Figure 5-2: introduction, five Q&A pairs from a pool, main question). LLMs have a compulsion to continue patterns, so examples teach how to read the question and especially the format and style of the answer; RLHF models learn where not to add vacuous comments. Implicit is often better than explicit. Examples also fix subtler traits, such as whether a scorer is grumpy or genial, and keep that persona constant.
In Figure 5-3, six Amazon reviews from a Kaggle dataset ("Anglophile's dream book: 5" ... "Don't waste your time.: 1") precede "A Small Book, but It Packs a Punch:". The model silently learns that the rating is a number; the format (review, colon, space, rating, newline); integers 1 to 5, higher is better; and a distribution skewed to 4s and 5s. Writing all that as rules risks omissions, and some rules resist statement at all ("I know it when I see it"). So examples are often easier. Easier, but also a bit dangerous.
| Drawback | Book example | Mitigation |
|---|---|---|
| 1. Scales poorly with context | Four invented personas as JSON (reviews, purchases, biography, favourite ice cream) before the user's own | Short examples only for output format |
| 2. Anchors the model | Mary 1919, Charlotte 1920, Eva 1916 make Elizabeth 1910; with 2019, 2020, 2016 she becomes 2019 | Representative real sample; edge cases; all major classes |
| 3. Spurious patterns | Ascending 1916, 1919, 1920 gives 1940; descending gives 1910 | Shuffle; test subsets; DSPy-style optimisers |
Drawback 1. In Chapter 2's attention game, minibrains on the answer shout questions at the prompt and many similar sections shout similar answers; even a human would lose track of whose fact is whose. Shrunken examples steer away from deeper reasoning and teach little, except format.
Few-shot token and latency cost sketch.
Drawback 2. Anchoring is unavoidable. One example per value still misleads: Figure 5-5 shows each rating once, implying equal frequency, so a blind guess becomes 3, though 5 is by far the most common. All data comes from a distribution that examples transmit (and complex outputs have one per aspect: length, vocabulary). Accept some bias for edge cases, best taught by example, without making exceptions look typical.
Drawback 3. Chance makes patterns: three numbers are ascending 17% of the time (17% descending); ten perfectly ordered is rarer than death by lightning (1 in 1.8 million versus about 1 in 100,000 in the US). Partial patterns count too, and systematic writing yields "happy path first, then unhappy path". In Figure 5-7's apple puzzles, solvable cases come first and "NA" cases last, so the model answers the final puzzle (Olive has a third of the apples she and Paul have together; one has 2 more than the other) with "no integer solutions". Reordered, it offers Olive 4, Paul 5, also wrong; such puzzles need chain of thought (Chapter 8).
Verdict. The dangers yield to careful evaluation (Chapter 10). Use examples for something otherwise unobvious, when space allows and bias is controlled; if the task is clear, skip them.
Dynamic content
Dynamic content pipeline.
The model now knows the task (fiction? textbooks? do comics count as books? "They don't") but nothing about the user. Gathering context takes most design and coding time and raises three issues. Latency: context is gathered while the program runs, so the time budget decides what is possible.
Preparability: content that rarely changes should be prepared in advance, even speculatively in latency-critical apps. Comparability: triage needs comparable items: Is one more useful? Does one depend on another? Does one invalidate another? Usefulness becomes a score: "Their last book was The Tesseract by Alex Garland, and they loved it" scores high; "Five years ago, they read The Catcher in the Rye" with no verdict scores medium. Static items need scores too, usually at or near the maximum, since understanding the question beats extra context. All context is optional; quantify how optional.
Preparability of content.
Finding dynamic context
Method 1: a mind map (Figure 5-8). Put "What book shall I read next?" in the middle and vary words, adding follow-ups. "I" yields demographics, employment, hobbies, home, travel, experiences; dropping "next" yields What have I read last? (How did I like that?) and What do others read? (bestsellers, friends), plus genres and movies. Cross off the unfeasible: bestsellers need an API call, movie tastes need access to purchases or emails.
Method 2: start from what you can gather, sorted along one axis (Figure 5-9).
Book Figure 5-9. Context sources by proximity (x) and stability (y): user profile is near and stable; page entries change fast; purchase history, emails and contacts are farther away; time, stock prices and weather are distant and volatile.
- Proximity, nearest first. (1) At the app's fingertips: screen state, time and date. (2) Saved: the profile. (3) Could record: past activity. (4) Public APIs: weather. (5) Needs the user's answer or permission: purchases, emails. Farther is harder, so it must be worth more.
- Stability, most stable first. (1) Always the same (profile). (2) Slowly changing (purchase history). (3) Ephemeral (time, interaction state). Less stable is harder to prepare, so latency is harder to hide.
Combine both methods: implement obvious sources first, exotic ones as the project matures.
Retrieval-augmented generation
Unaided, a model refuses or hallucinates about recent or private facts. RAG, from the May 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", retrieves relevant content into the prompt. The recommender has shortlisted The Beach, holds Wikipedia's summary ("Set in Thailand, it is the story of a young backpacker's search for a legendary, idyllic, and isolated beach...") and everything the user has written. Figure 5-10 frames it as advice from a friend, Fiona, citing a TripAdvisor post ("Thailand is really my favorite country!") and a Facebook rant against backpackers.
Lexical retrieval predates LLMs. The simple version (Figure 5-11): cut the context into snippets, remove stop words, apply stemming (walking, walks, walked become walk), then compute Jaccard similarity:
J = shared words ÷ all unique words in snippet and query (0 = nothing shared, 1 = identical)
The Beach summary and the rant share backpack, tour, stori, with 12 words unique to each: 3 ÷ (12 + 12 + 3) = 0.11. Treat the search string as a miniprompt ("I'm considering what book to read next"; "The Beach is about a young backpacker"). Jaccard needs no preindexing and little memory, and runs blazingly fast, so GitHub Copilot uses it on the programmer's open files. But a shared "go" counts as much as "backpacking"; TF*IDF and BM25 weight rare words higher, at the price of precomputed word counts.
Neural retrieval fixes lexical false positives ("I forgot my backpack today" versus "Today I'm going backpacking"), false negatives ("We forgot our backpacks today" versus "They didn't remember their rucksack this morning"), typos, synonyms and language barriers. An embedding model maps text to a vector; similar meanings land near each other (Euclidean or cosine). Offline: split, embed, store in a vector store. At request time: embed the query (the user's, or an LLM summary of the chat) and fetch the nearest snippets.
- Snippet size, three criteria. Under the embedding model's limit (8,191 tokens for OpenAI's models in 2024); one and only one main idea, or the vector lands between topics; and a size that suits the prompt.
- Cutting methods. A moving window, for example 256 words with a stride of 128, so windows overlap and ideas are not cut in half (more overlap is safer, less saves storage); natural boundaries such as paragraphs or sections; and augmentation, for instance rebuilding a method snippet with its class definition and initialisation code so the embedding has context.
- Embedding models. Not LLMs: Transformers trained by contrastive pre-training to output vectors, tiny and orders of magnitude cheaper. Start hosted (OpenAI), later self-host for latency and cost; seek a specialist model for unusual languages, or train your own, far easier than an LLM.
- Vector storage. Vectors of about a thousand entries; FAISS makes search fast, Pinecone.io offers a managed, scalable service.
The book's no-frills RAG app (Figure 5-12). Indexing runs ahead of time; at request time two retrieved reviews meet static boilerplate and the model predicts a 1-to-5 rating.
A simple RAG app. A bookstore page predicts enjoyment from the user's past reviews. get_embedding calls text-embedding-3-small; index_reviews puts all 11 reviews in a faiss.IndexFlatL2; retrieve_reviews returns the k = 2 nearest to "The Beach by Alex Garland critiques backpacker culture by exposing the selfishness and moral decay behind their pursuit of an untouched paradise". predict_rating frames them with static boilerplate ("Here is a book I might want to read:", "Here are relevant reviews from the past:", "On a scale of 1 (worst) to 5 (best), how likely am I to enjoy this book? Reply with no explanation, just a number.") for gpt-4o-mini. Prediction: 2.
- Lexical. Decades old and still behind most online search; Elasticsearch (open source) and Algolia (PaaS) index huge corpora with low latency. Failures are explainable (the tokens did not match) and fixable by tuning stemming or adding synonyms, and relevance can be tuned, for example boosting title matches over description matches.
- Neural. Most RAG apps use it. Vectors are opaque, so a missed match is hard to debug, and changing relevance means training a new model and reindexing everything. Its big win: it matches ideas, not words, even across languages, and with images mapped into the same space.
Summarisation
Retrieval zooms in; summarisation zooms out. Appending "Tersely summarize all of the above" to the RAG section in gpt-3.5-turbo-instruct gave a fair summary; a whole chapter draft gave "maximum context length is 4097 tokens, however you requested 9491 tokens (8491 in your prompt; 1000 for the completion)". Length is why you needed a summary in the first place.
Hierarchical summarisation splits the corpus into units that fit the window, summarises each, then the summaries (Figure 5-13: The Beach chapter by chapter, under 100 words each, then the whole). The rumour problem: misunderstandings compound (one chance at level 1, three at level 3), though generous summary lengths keep the Telephone game harmless. Split along natural groups (chapters, topics, authors, projects) and avoid unbalanced passes.
General versus specific. Compression is never lossless: a vacation-post summary drops the aside about which book eased the long flight, exactly what a recommender needs. Summarise with the final task in mind.
- Few-shot on reasoning models. OpenAI's guide says try zero-shot first and keep any examples closely aligned with the instructions; the DeepSeek-R1 paper (January 2025) found few-shot consistently degraded R1.
- Hybrid retrieval. Anthropic's Contextual Retrieval (19 September 2024) fused embeddings with BM25 and prepended model-written context to each chunk: top-20 retrieval failures fell 49%, and 67% with a reranker cutting the top 150 to 20. Lexical or neural is now usually "both".
- Long context versus RAG. Anthropic suggests skipping RAG under 200,000 tokens (about 500 pages); the 2025 LaRA benchmark found no silver bullet.
- Prompt caching. OpenAI reuses the computed state of an identical prompt prefix by default (minimum 1,024 tokens on GPT-5.6 and later; reads 0.1x input price, writes 1.25x); Anthropic reads are also 0.1x. So put static content first, dynamic last.
- Example selection. DSPy's MIPROv2 tunes instructions and demonstrations together; GEPA (July 2025) evolves instructions from feedback.
6. Assembling the Prompt
Gathered content must become one document the model completes well: an introduction, a parade of context, a refocus and a transition, with key material kept out of the weak early middle. The chapter compares three document archetypes, sets rules for snippets that can be added or cut cleanly, and treats assembly as an optimisation problem for simple prompt-crafting engines.
Anatomy of the ideal prompt
Anatomy of the ideal prompt.
Concise, crisp prompts work better, cost less and run faster, within the hard context-window cut-off. Element size and number are free: the authors have used three lengthy elements and hundreds of one-liners. Ending each with a newline simplifies string handling and token counting.
Cut fluff; keep signal.
- Introduction. Says what the document is ("This is about recommending a book") and lets the model start thinking early, given its fixed thought budget per token; sub-sections can have their own.
- Context elements. Used unequally because of in-context learning (nearer the end, more impact) and the lost middle (start and end recalled, middle not). Keep key elements out of the valley and the prompt short.
- Refocus. After long context, restate the question. The sandwich technique states the task at both start and end (Table 6-1: a ChatML exchange about suggesting a book to Fiona, whose last reads were Harry Potter, Lioness Rampant and Mr Lemoncello's Library, ending "Excellent! So based on this, which book should I suggest to her?"). The introduction sets the stage ("I'm thinking about book suggestions for X."); the refocus carries key clarifications ("What's the best book to recommend next, focusing on narrative prose currently available?").
- Transition. Switch firmly to solving, or the model adds made-up context. In chat a question mark suffices; with completion models, begin the answer yourself.
Prompt sections cycle.
The ideal prompt (Book Figure 6-1) and why its order matters. Sandwiching repeats the question in the introduction and the refocus, so both sit where the model pays most attention.
Figure 6-2 (text-davinci-002), after context on Fiona (cats, Harry Potter, 26, blue): with no transition the model adds "I'm aware that she likes to read."; a naive "What should I suggest as her next book?" gets musing ("I'm thinking of suggesting a book that she can read in one sitting."); the refined "Based on these, I believe the book she should read next is" plus an opening quote yields "The Cat Who Came for Christmas" by Cleveland Amory, merging refocus and transition.
What kind of document?
Document kinds for prompts.
Prompt plus completion form a document, and by the Little Red Riding Hood principle it should resemble common training documents. Three archetypes cover most needs.
The advice conversation is commonest: the app or user asks, the model helps; OpenAI built ChatML around it. Advantages: natural interaction, multiround interactions (your logic between turns) and real-world integration with tools. Chat models add RLHF compliance; completion models avoid unwanted RLHF traits (stylistic habits, content policing) and allow inception, after the 2010 film: start the answer and the model treats it as its own, improving compliance and parsing. Writing whole assistant turns is inception too, and framing context as answers to the assistant's own questions makes it answer rather than ask again.
| Format | Plan-the-day prompt (Boston, 75 degrees, sun) | Strength | Weakness |
|---|---|---|---|
| Freeform | I asked my husband, "What shall we do tomorrow?"... and he suggested, → beach picnic | Quotes hold anything | Hard to assemble on the fly |
| Script | Me: / Husband: lines → bike ride | Easy to assemble | Poor for long or indented text |
| Markerless | Bare lines, "I suggest we should" → Charles River | Long, formatted text | Speakers and reply end unclear |
| Structured | <me>, <husband>, <direction> tags → Blue Hills hike | Clear speaker and end | Pick a structure (below) |
The analytic report taps what millions of students learn to write, suiting business, literature, science and law ("probably best to leave legal defense to human professionals"); context drops into background sections. Add a Scope section ("This report focuses solely on novels, excluding self-help books"); LLMs respect boundaries stated up front in a report more consistently than the same exclusions negotiated in dialogue. Reports favour objective analysis (no simulated social interaction) and suit chain of thought, but need a clear transition into decision mode. Use Markdown: ubiquitous in training data, simple, headings form a rearrangeable hierarchy, indentation rarely matters (backtick blocks for code), easy to render, links easy to parse. A table of contents orients the model and controls output: scratchpad sections (# Ideas, # Analysis) before # Conclusion, and a later section (# Appendix, # Further Reading) as a stop sequence.
FooCoffee report case.
The structured document follows a formal spec, so complex output parses easily. Its XML holds artifacts_info (the system message) and examples of user_query and assistant_response, where an antThinking block decides whether to make an Artifact and an antArtifact tag carries identifier, type, language and title. Following a factorial example, the model answers a prime-factor request the same way; the app hides antThinking and moves the artifact to the pane.
- XML. Tags with attributes and nesting; best for short elements where multiline indentation does not matter. Watch the five escapes (", ', <, >, &); comments can carry editorial hints.
- YAML. Hierarchy by indentation, fussy for parsers but right where indentation matters, such as code. A
fieldname: |2block preserves indentation, needs no escaping, and ends at a line indented less than its zero level (Figure 6-4). - JSON. Escape-heavy and less readable, once discouraged, but OpenAI trained its models hard on JSON for its tools API, so for OpenAI it is a reasonable choice.
Formatting snippets
Format tradeoffs.
A weather result ({"description": "sunny", "temperature": 75}) becomes a conversation turn ("User: What's the weather like? Assistant: It's going to be sunny with a temperature of 75 degrees."), a report section ("#### Weather Forecast" plus a sentence) or simply serialised XML in a structured document. An explicit aside ("As an aside, ...") hints without forcing use: Copilot put other files' code into the source file as a comment, // <consider this snippet from ../skill.go> ... // </end snippet>.
- Modularity. Snippets insert and remove cleanly; documents shaped as a list (conversation turns) or tree (report sections, structured documents) help.
- Naturalness. Each snippet reads as part of the document: natural language becomes a code comment in source, data becomes prose in a report.
- Brevity. Fewer tokens for the same information.
- Inertness. A snippet's tokenization should not change its neighbours', so its token count is computed once.
Four snippet aims.
Token counts are not additive (Table 6-4, GPT-3.5-and-later tokenizer; also true of the older one):
| Strings | Tokens | Token IDs | Count |
|---|---|---|---|
| "be" + "am" → "beam" | [be] + [am] → [beam] | 1395 + 309 → 54971 | 1 + 1 → 1 |
| "cat" + "tail" → "cattail" | [cat] + [tail] → [c] [att] [ail] | 4719 + 14928 → 66, 1617, 607 | 1 + 1 → 3 |
Separate elements with whitespace, but GPT tokenizers have tokens that start with a space, not end with one, so prefer elements that begin with a space; they also merge consecutive newlines, so snippets should never start (easier) or never end with a newline.
Few-shot examples can be labelled as such ("when I encounter a question like 'Who was the first President of the United States?' I will give an answer like 'George Washington.'") or woven in as solutions to earlier tasks, which in ChatML makes the model believe it has already succeeded in that style.
Elastic snippets. For a scene in The Beach, with two key passages retrieved, add two bare snippets, two with context, or one combined snippet linking them (Figure 6-5); the whole chapter rarely fits. Either make an elastic element with versions from the whole chapter down to the two quotes joined by "...", and ask at assembly "What's the biggest version of this snippet we have space for?", or create several overlapping elements and mark them incompatible.
Relationships among prompt elements
- Position. Elements keep an order: quotations in source order, chats and narratives chronological, items in the right section (a liked book does not go under "Books I really hate"). Track it with an array, linked list, index or position value; often you just append as you gather.
- Importance. Not the same as position, though correlated: the introduction outranks most middle details. Choose length-relative or absolute importance and apply it consistently. Use tiers (cut lowest first) or scores: central instructions and output format in the top tier, explanations second, context third, with numbers for finer distinctions, tuned by evaluation (Chapter 10).
- Dependency. Requirements: "Richard is the protagonist of The Beach" must precede "He grew up in England". Incompatibilities: a summary and a detailed version of the same fact; keep both, use the long one if space allows.
Putting it all together
Assembly is an optimisation problem: choose elements to maximise total value subject to dependencies and prompt length (the window minus the response, or a softer budget when the window is huge). Then order the chosen elements by position. It resembles linear programming and the 0-1 knapsack problem, but no standard tool handles the dependencies, so you build your own; Copilot uses custom functions for code snippets that need a specific postfix.
Three engines, from simplest up:
- Minimal crafter (Figure 6-7). (1) Take elements; (2) order by position; (3) measure lengths; (4) drop from the front until it fits; (5) paste and send. No scoring; LLMs handle document suffixes well.
- Additive greedy (Figure 6-8). (1) Take elements; (2) order by value; (3) add the best element that fits and whose requirements are met without conflict; (4) repeat; (5) stop when none fits; (6) order by position and paste. Copes with far more elements than fit; struggles with cyclical requirements or valuable elements that depend on weak ones. Sort by requirements so an element is considered only once its dependencies are in.
- Subtractive greedy (Figure 6-9). (1) Take all; (2) order by value; (3) remove the least valuable; (4) transactionally remove elements whose requirements just vanished; (5) repeat until within budget; (6) order and paste. Best with few elements and few incompatibilities; handles elastic snippets more easily.
These are prototypes; move beyond them as your application's requirements become clear.
- "In-context learning." Used here for a recency effect; the GPT-3 paper and common usage mean learning from in-prompt examples. "Recency bias" is the usual name.
- Figures. In 6-6 the one-word "apprentice" element shows the same 41 tokens as the long Jed text; 6-3's contents run 1, 2, 4, 6, 7 and promise "Suggestion" but get "Conclusion".
- Code.
n /= dmakes n a float in Python 3 (//=is safer); XML escapes need trailing semicolons.
- Sandwich confirmed. OpenAI's GPT-4.1 prompting guide (April 2025): with long context, instructions before and after the documents beat either alone.
- Query last. Anthropic puts long documents first, the query last, reporting up to 30% better answers on complex multi-document inputs.
- Formats. OpenAI found XML performed well and JSON particularly poorly for delimiting many documents; Anthropic recommends tags like
<document index="1">. That qualifies the chapter's JSON allowance. - Caching. Caching rewards an unchanging prefix (Chapter 5 update): stable instructions and examples first, volatile context later.
7. Taming the Model
The chapter closes Part II with the question behind every application, which model to call, and when to fine-tune one.
Anatomy of the ideal completion
Ideal completion anatomy.
Figure 7-1 splits a completion into a preamble, the recognisable start, the main answer, the recognisable end and a postscript. Every generated token costs latency and compute, so text you will not use is waste, except when it is not.
- Structural boilerplate. The text between prompt end and answer start. With completion models, put deterministic boilerplate in the prompt instead: the model then follows the format, and the call is faster and cheaper. Boilerplate also makes a good transition.
- Reasoning. From late 2023 ChatGPT began restating a slightly interpreted version of the question, which focuses inference and exposes misunderstandings; chain of thought (Chapter 4) also lives in the preamble. If you want chain-of-thought reasoning, a long preamble is a virtue, even when it dwarfs the answer.
- Fluff. RLHF models are verbose and polite, bad for programmatic use, and models without RLHF do it too. Few-shot examples help but cost tokens. Structured documents mostly hold their format; in freeform text, ask for the main answer first and park comments, disclaimers, background and explanation in later numbered points.
Long preamble case.
Recognisable start and end
Start/end discipline.
To extract the answer you must find where it begins and ends. Table 7-1 lists the usual structures and whether the end test is a simple substring check, which matters for stop sequences:
| Structure | Start | End | Substring? |
|---|---|---|---|
| Markdown | Expected section header | Any other header | Yes |
| YAML | Expected keyword after a newline | Line with lower indentation | No |
| JSON | Keyword in quotes, colon, quote | Any unescaped quote | No |
| Code listing | ```[language]\n | \n```\n | Yes |
| First list item | 1. | 2. | Yes |
| Bracketed code (Java) | { | Matching closing bracket | No |
| Indented code (Python) | Function or class header | Lower indentation | No |
A good prompt can turn a "No" into a "Yes": in YAML, if you know the next keyword, look for lower indentation followed by that keyword, which is a substring.
Postscript: stopping on time
Stopping on time.
A recognisable end lets you filter the postscript, and more usefully, stop paying for it. Self-hosted models can be halted at will; hosted ones give two levers.
- Stop sequences. The OpenAI-style
stopargument takes a list of strings; generation ends server-side on a match, with no further cost in time, compute or money. - Streaming. With
stream: truetokens arrive in small batches; spot the end and cancel. That saves less, because network delay means the cancel signal arrives late. - Newline-led stops. Start stop sequences with a newline:
\n#in Markdown, so a code comment or phone number does not end the answer. - Combine them. Stop sequences are slightly more effective and more widely supported, but they are fixed strings. When streaming, add common endings as stops anyway: for a Python class,
\nclass,\ndefand\nif. Methods inside the class are indented (\n\tdef), so they do not trigger.
Beyond the text: logprobs
Logprobs for confidence.
The model computes a full next-token distribution (Chapter 2) and many APIs return it as logprobs, natural logs of probabilities. A logprob is negative; 0 means certainty; convert with exp.
p = exp(logprob): exp(-0.405) ≈ 0.67 ("Yes"), exp(-1.099) ≈ 0.33 ("No")
OpenAI-style APIs return logprobs for the chosen tokens and the alternatives considered (Figure 2-12) at no extra compute. Some vendors withhold logprobs for fear of reverse engineering, so factor that into model choice.
How good is the completion? Albert's neighbour, an astrophysicist, answers "13" minutes for sunlight to reach Mars instantly; his 10-year-old daughter hesitates, "Maybe 30?" Tone of voice shows which to trust, and logprobs are the model's tone of voice. Summing logprobs gives total confidence but punishes length: "for example" versus "for instance" can halve a probability without changing quality. So average:
Logprob summary stats sketch.
simple mean: (logprob1 + ... + logprobn) / n
Copilot: (exp(logprob1) + ... + exp(logprobn)) / n, early tokens
Albert found during GitHub Copilot development that the mean probability of the first tokens predicts overall quality. It is not an absolute measure, but it supports cut-offs:
- Correct only when confident. Show suggested corrections only above a threshold.
- Warn. Flag answers where the model struggles more than usual.
- Retry or enrich. Add context or retry on low confidence.
- Escalate. Switch to a smarter, pricier model.
- Interrupt sparingly. Offer help only when need is near-certain: "Remember Clippy? Don't be like Clippy."
Confidence gates.
For more quality at more cost, raise temperature, generate several completions and pick the best by logprob. With n above 1, temperature must exceed 0 or all copies match; the authors' "completely unscientific" rule is temperature = sqrt(n) / 10.
LLMs for classification
Classification picks one of a fixed set of categories (review positive, negative or neutral; product for the American, European or Asian market; yes or no), like a detective novel with a limited list of suspects. LLMs were built for open generation, yet as pretrained generalists they classify well where the task rests on public knowledge and common sense, with little or no training data. Ask for a format with a recognisable start: "Please answer in the format: 1. [negative | positive | neutral], 2. [explanation]", and read the answer right after "1.".
Book Figure 7-5 redrawn, real gpt-3.5-turbo-instruct probabilities. Greedy decoding picks the shared first token [North], so Northeast Asia (42%) beats Europe (44%), the option the model rates highest overall.
Make every option start with a unique token. In the toy-launch prompt (retail links in North America, marketing prefers North-East Asia, Europe has the most growth) both "North" options pool their mass on the first token, so an uncertain model leans toward them and decides between them later.
Calibration. In an app that blocks unfriendly emails from grumpy users ("Is this a professionally written email? 1. Yes / No. 2. Explanation."), the model may rank emails well yet draw its threshold in a different place from yours. Calibration shifts each option's logprob by a constant atok before comparing: to block fewer emails, add ayes = 0.3 to "Yes", so "No" wins only when its logprob leads by more than 0.3. Find the constants by experiment or, as in logistic regression, by minimising cross-entropy loss on ground-truth data. Once found, send them as a logit bias and the provider applies them.
- Quality. Average early-token probability predicts completion quality; gate features on it.
- Classification. Read option probabilities, ensure unique first tokens, calibrate with additive constants or logit bias.
- Critical points. With
echo: truemany APIs return prompt logprobs too, even with zero completion tokens, exposing surprising or information-dense spots.
Critical points in the prompt
The authors plant a typo, "complution", in their own paragraph. Displayed with logprobs (Figure 7-6, two versions interleaved), it scores below -13: the model saw [compl] where it expected [completion], then [ution]. Single-digit negatives are common; double-digit negatives usually flag weirdness. No fixed threshold exists across models or genres, or even within one text: early tokens score lower because topic and style become clear only as the model reads. Uses: detect typos, find dense passages to focus the app on, or guide the user's eye.
- Tests. Logprobs are not deterministic (floating-point effects, up to ±1 on some deployments): write tolerant tests or mock the model.
- Figure 7-2. The "correct" run eliminates 1 as "1^1 = 1 (power of a prime)", which is muddled (1 is p0 at best). {6, 10} still holds under the usual reading.
- Rule of thumb. sqrt(n)/10 is, by the authors' own label, unscientific.
- Copilot finding. The early-token average is an internal Copilot result with no published data in the book.
Choosing the model
New models arrive weekly, so the authors give principles, not picks, and advise against hard-coding the choice; libraries such as LiteLLM offer one API over many models. Considerations, in order of importance:
- Intelligence. Closeness to a strong human expert; key for complex reasoning or high accuracy.
- Speed. Wait time; key for direct user interaction (Table 5-2's urgency levels).
- Cost. Inference fees or GPU spend; key for high request volume.
- Ease of use. Who handles GPUs, deployment, restarts, routing, caching.
- Functionality. Instruct, chat, tool use, logprobs, images.
- Special requirements. Like dietary requirements, absolute for some and irrelevant to others: noncommercial, open source, training data, update policy, data residency, no off-premises logging.
Figure 7-7 draws these as linked dials (smart to fast, large to small, premium and easy to scrappy and cheap, plus capability and requirement checklists): tightening one constrains the rest. High volume, simple requests: a small model. A solo project making about one request a day: splurge on premium. Very hard requests, very cheap, very smart: "tough luck".
Pick a provider first, then a model, then a size. OpenAI once dominated, but by 2024 the field evened out:
| Provider | Book's 2024 characterisation |
|---|---|
| Anthropic | Alignment and safety focus; Claude 3.5 Sonnet topped several benchmarks |
| Mistral | Efficient open-weight models for specialised configurations |
| Cohere | High-performance RAG |
| Ecosystem integration, research, infrastructure | |
| Meta | Large, capable open-access models |
The authors like Artificial Analysis for comparisons. Self-hosting open models (LLaMA, Mistral) via Hugging Face (own servers or its Azure partnership) is worth it only at scale or when requirements rule out full-service vendors; agile teams can prototype on OpenAI and move later. Choose the smallest model that reliably does the task, but prototype one size up: flagship prices fall, so by public beta you can afford the better model your prompts were tuned for.
Fine-tuning
Pretraining teaches a model to mimic documents; fine-tuning shows it new ones. It usually weakens generic ability but can sharply improve the documents you care about. You need successful interactions: correct answers, only the background you want learned, the expected format. Sources: write them, hire contractors, synthesise them, mine accepted suggestions and likes, or reuse logs of the humans who did the task. Loss masking trains only on the answer portion, not on the prompt.
Figure 7-8 is the decision tree. Already solved by current models: no need. Not solved even a tiny bit: fine-tuning will not save you. Task unstable (right answer changes within 3 months): wait.
| Type | Learns | Docs needed | Takes |
|---|---|---|---|
| Full fine-tuning / continued pretraining | New facts, potentially a whole new domain | Tens of thousands | Weeks or months |
| PEFT (e.g. LoRA) | Priors in a known domain, interpretation, fixed format | Hundreds or thousands | Days |
| Soft prompting | Whatever a prompt could hold | Hundreds | Hours |
- Full fine-tuning. Adjusts every one of billions of parameters. Not explanation but a riverbed forming: pour thousands of documents and a groove is slowly carved, and the groove can be anything.
- LoRA (low-rank adaptation). Trains a low-rank "diff" added to a few key matrices. Diffs are small, easy to ship between machines, and one deployment can serve many; training takes hours to days. Capacity is capped by the LoRA dimension: LoRA does not teach new tricks; it teaches which existing tricks to use and how. It learns format, style and the domain's prior distribution.
- Soft prompting. The prompt creates a "state of mind"; instead of hand-crafting words, learn that state from a few dozen desired outputs. Many frameworks do not support it.
The travel example. European customers make Napa Valley far and Monaco near; a prompt could say so. But telemetry may show Monaco draws thumbs-down while Prague sells tickets, because most users are budget-minded students. LoRA absorbs such distribution shifts, known or unknown. Fine-tuned models can drop static context, instructions and few-shot examples, so fine-tuning is prompt engineering by other means.
- Logprobs narrowing. OpenAI's Responses API returns logprobs via
include: ["message.output_text.logprobs"]withtop_logprobs0 to 20, but GPT-6 Astra (Sept 2026) rejects logprobs, temperature and top_p; GPT-6 Sol and 6.1 Sol accept them. Reasoning models hide their thinking tokens, so logprob tricks see only the final answer. - Stops. The Responses API documents no
stopparameter; usemax_output_tokens, structured outputs (Chapter 4 update) or streaming cancellation. On some reasoning models a stop string hit during reasoning returns an error. - Fine-tuning retreat. On 7 May 2026 OpenAI began winding down self-serve fine-tuning: no new organisations, then (2 July) none idle 60 days, and no new jobs after 6 January 2027; legacy fine-tunes of gpt-3.5-turbo, gpt-4 and davinci-002 shut down 23 October 2026. Its API had added DPO and reinforcement fine-tuning (programmable graders) before that. Teams that want custom weights are pushed toward LoRA on open-weight models (Mistral, Llama, Qwen) or other vendors' tuning services.
- Evidence on LoRA. "LoRA Learns Less and Forgets Less" (2024) found LoRA trails full fine-tuning on code and maths but preserves out-of-domain skills better, which supports the chapter's intuition.
- Gateway risk. LiteLLM releases 1.82.7 and 1.82.8 were backdoored on PyPI on 24 March 2026 (credentials stolen via the Trivy compromise); pin versions of any model gateway.
8. Conversational Agency
Part III opens by turning the chat model into an agent. A bare chat model knows only its training data and what the user typed, cannot compute reliably and cannot act. Tools give it fresh information and hands, reasoning techniques give it an internal monologue, and careful context gives it memory of the task; one loop around the message handler then yields a working conversational agent.
Agency is the ability to complete tasks and reach goals in a self-directed, autonomous way. Conversational agents keep the back-and-forth of chat but let the assistant learn new information and act on real-world assets.
Tool usage
Tool usage loop.
A chat model is "the digital zeitgeist of the world" and a fine tutor, "if you don't mind some hallucinations". Three gaps remain:
- Hidden knowledge. It cannot see private documents, memos, chats or code, nor anything after training: new API changes, news, or up-to-the-moment data like today's flights.
- Nonlanguage tasks. Maths is the obvious failure: simple sums are effectively memorised, but larger numbers bring worse estimates, stated confidently.
- No hands. It can only ask the user to act; it cannot buy tickets, send email or change a thermostat.
Tool usage closes all three: tell the model which tools exist, when and how to use them; the application parses the call, relays it to a real API and feeds the result into later prompts. OpenAI shipped a model fine-tuned for tool invocation in June 2023; competitors followed.
Rough mix of app shapes.
A lookup dictionary available_functions maps names to functions, so tools are easy to swap.
Thermostat tool example.
- 1. Send the messages plus tool definitions to the model (
gpt-4o). - 2. Append the assistant reply, which may hold prose, tool calls, or both.
- 3. Check for
tool_calls. - 4. Execute each call: look up the function,
json.loadsthe arguments, call it with**function_args. - 5. Append a
role: "tool"message carrying thetool_call_id, name and output. IDs pair calls with responses when several run at once.
process_messages five steps.
Calling process_messages by hand is not yet agency, but it is "one while loop away".
Under the hood: tools are still document completion
Tools are still completion.
OpenAI does not document the internal format, so the authors
reconstructed it by interrogating the model. Tool definitions land
in the system message, after your text, under Markdown headers (
# Tools,
## functions), inside
namespace functions { ... }, written as
TypeScript: a
//comment with the description, then
type set_room_temp = (_: { temp: number, }) => any;. Markdown is a
Little Red Riding Hoodmove, and a hint to use it in your own prompts. TypeScript is clever three ways:
- Rich types. A richer type vocabulary means arguments arrive in the right types.
- Inline docs. Function and each argument carry documentation.
- Named arguments. Calls pass a JSON object of named arguments, so calls are consistent and easy to parse, and the model writes "temp" right before the value, making a wrong value less likely than with positional arguments.
Tool definition quality.
Invocation looks like <|im_start|>assistant to=functions.set_room_temp then {"temp": 76}<|im_end|>; results return in a new tool role. The API uses the call IDs to order calls and responses, then drops them from the transcript.
Now, you try: extracting a vendor's tool prompt. Models rarely reveal internals, so escalate:
- Ask directly for all text above the first message (rarely works).
- Plant markers:
<LOGGING>in the system message,</LOGGING>in the first message, then ask for what lies between. - Name functions oddly and ask for the text around them.
- Build a logging tool; tools sometimes reach content the assistant will not share.
- Obfuscate: have the tool output base64 or ROT13 (only top models convert accurately).
- Seed the pattern: add hints as assistant comments so the model continues sharing.
Extracting vendor tool prompts.
Guidelines for tool definitions
Two intuitions drive them: what is easier for a human is easier for an LLM, and pattern prompts on training data.
- Select few tools. More tools, more confusion. Tools should partition the domain: broad coverage, no overlapping actions. Do not copy your web API into the prompt; its many parameters and complex responses waste space and cause bad calls.
- Name clearly. Self-documenting names set expectations. For OpenAI follow the TypeScript style with camelCase; avoid squashed lowercase like
retrieveemail. - Define simply. If a definition reads like legalese, it overloads limited attention; when detail is needed, leave no ambiguity. For a public API the model knows, keep its names, concepts and style: Copilot's model could recite GitHub's code-search syntax, so arguments kept documentation names and formats.
- Keep arguments few and simple. OpenAI handles string, number, integer, boolean, plus enum and default. As of the 1106 models (November 2023), modifiers like minItems, uniqueItems, minimum, maximum, pattern, format, and nested parameter descriptions, do not appear in the prompt.
- Beware long text arguments. JSON values need escaped newlines and quotes, and long text (worst, code) makes the model forget escapes. Anthropic encoded calls in XML, so Claude should handle long arguments better.
- Watch argument hallucination. GitHub tools with org and repo arguments get placeholders like "my-org" and "my-repo". Remove arguments the app already knows, or give a default you can detect; telling the model to ask when unsure works only sometimes.
- Shape outputs. Let the definition say what the output holds; freeform or JSON both work. Skip "just-in-case" extras, which distract.
- Make errors useful. Errors let the model self-correct, but translate internal messages into terms of the tool's definition; for validation errors, say what was wrong.
Reasoning
Token-by-token generation is superficial reasoning: training makes text that sounds right, with no internal monologue, no mapping to facts, no weighing of competing ideas. Every fix gives the model a monologue before the answer.
- Chain of thought (January 2022). Normally a model answers "Will The Exorcist stimulate the limbic system?" with yes or no first, so the explanation rationalises a guess. Few-shot examples that reason first ("Hamsters are prey animals... So the answer is yes"; "The density of a pear is about 0.6g/cm3... a pear would float. So the answer is no") make it reason, then answer. PaLM 540B on StrategyQA: 69.4% → 75.6%; on GSM8K maths: about 20% → 60%; gains also in symbolic reasoning.
- Zero-shot CoT (May 2022). "Large Language Models are Zero-Shot Reasoners": start the answer with "Let's think step-by-step", no curated examples needed.
- Pause tokens (October 2023). "Think Before you Speak" fine-tuned a model to use a meaningless pause token; injecting say 10 after the question gives it extra timesteps, like a human's "Uh" and "Um".
ReAct: iterative reasoning and action
ReAct (October 2022), one of the first papers to use external tools, targets retrieval plus multistep problems. On HotpotQA ("Which magazine was started first, Arthur's Magazine or First for Women?") the model gets three tools:
- Search[entity]. First five sentences of the Wikipedia page, or the five most similar entities.
- Lookup[string]. Next sentence containing the string in the current entity.
- Finish[answer]. Ends with the final answer.
Book Figure 8-1. HotpotQA exact match for Standard, CoT, Act and ReAct on PaLM 8B, 62B, 540B. Prompted (left), ReAct trails at every size; fine-tuned (right), ReAct leads, with 62B ReAct (about 33) above every prompted 540B result (about 29 at best).
Prompted, ReAct was worse than standard and CoT at every size: six examples could not teach tools and reasoning. After fine-tuning the smaller models on just 3,000 examples, ReAct leads: fine-tuned 8B ReAct beats prompted 62B, and fine-tuned 62B beats prompted 540B. Proper reasoning on a slightly fine-tuned model can beat a much larger vanilla model. On HotpotQA much of the gain comes from search itself (Act alone does well). On ALFWorld, a text-game house where the agent must perform chores, thinking matters: ReAct 71% versus Act 45%. Act is worse at subgoals and loses track of state.
- Decompose task goals and create action plans.
- Inject commonsense knowledge relevant to the task.
- Extract helpful details from observations.
- Track progress and push plans forward.
- Handle exceptions and adjust course.
Beyond ReAct
| Method | Idea | Prompt or mechanism | Fits |
|---|---|---|---|
| Plan-and-solve | Plan the whole problem first; no tools | "Let's first understand the problem and devise a plan... Then, let's carry out the plan and solve the problem step-by-step." | Holistic problems; could be combined with ReAct |
| Reflexion (2023) | Review afterwards, find faults, plan better next time | Insert failure messages (e.g. failing unit tests) and retry | Domains with a do-over; useless for irreversible acts |
| Branch-solve-merge | N independent solvers, then a merger | High temperature or, better, a different perspective per solver; a merging agent combines | Problems that benefit from varied attempts |
Reflexion's limit in one line: "I'm sorry for transferring your assets to your ex-husband's account. I won't do that again!"
Context for task-based interactions
Chapters 5 and 6 still apply, but agents add sources and organisation choices.
| Part (Table 8-1) | Holds | Travel-assistant example |
|---|---|---|
| Preamble | Rules, instructions, expectations; relevant tool definitions; few-shot examples if needed (system message) | "helpful and knowledgeable travel assistant", date 8/9/2023; tools get_flights(src, dest, date), get_ticket_info(flight_num) |
| Prior conversation | Earlier user and agent messages, excluding the current exchange; artifacts attached to them | Dulles to Seattle next Monday: JL5441 9:20AM and AS325 4:50PM in an <artifact> block |
| Current exchange | Latest user message, user artifacts (highlighted text, clicked component), tool calls and responses | get_ticket_info("JL5441") → $350, stop ORD, 7h40m |
| Agent response | Summarises the exchange; joins the prior conversation next time | "There is a flight for $350 that makes a stop in Chicago." |
Artifacts are any data relevant to the conversation; the app should detect what the user points at rather than make them paste it. Selection questions, with no one-size answer, so "evaluate, evaluate, evaluate":
- Which tools? Drop tools the current phase cannot use; one less distraction.
- Which artifacts? All (best information, but irrelevant bulk confuses) or let the model pick (needs a side request and more app complexity).
- How presented? In an XML tag (
<artifact>) or a Markdown section (## Attached Data); JSON or text seems not to matter much. If all artifacts come from tool calls, simply keep past tool calls in the prior conversation, which also gives the model more tool-use examples. - How much of each? Never a whole book. Untested idea: a bulleted summary with "for more information, call
details('section 5')", unfurling sections on demand; or plain RAG over the artifact. - How far back? Drop old topics: cut prior sessions after inactivity, or have a model judge relevance, too slow and costly for a large model but feasible for a small trained one.
Building a conversational agent
run_conversation (Example 8-2) initialises the messages with a system prompt ("You are a helpful thermostat assistant"), loops on input(">> ") (empty input exits), appends each user message, then calls process_messages repeatedly until the last message is an assistant reply with no tool calls, printing assistant content along the way. Figure 8-2 shows the sequence between user, application, model and tool.
| Turn (Table 8-2) | Dialogue and tools | What it shows |
|---|---|---|
| User | "Golly gee, it's hot in here. Let's cool this place down." | Colourful language understood |
| Agent | [get_room_temp() → 64ºF] "...actually quite cool... Would you still like to adjust?" | Common sense pushback, still offers to comply |
| User / Agent | "Make it LOTS cooler." [set_room_temp(50)] "It should feel much cooler soon!" | Sensible magnitude (not 0 or -1,000); knows thermostats lag |
| User / Agent | "Whoa... Put it back to where it started." [set_room_temp(64)] | Uses the prior conversation to recover the start value |
The code is generic: change the system message and tools for any behaviour; production needs an API, error handling and logging. Exercise: swap in your own tools in a notebook and test confusion, how many functions it tolerates, definition wording and tool errors; ask ChatGPT to draft definitions.
User experience
The chat UI has barely changed from AOL Instant Messenger (1990s) to Slack. In Figure 8-3, Dave asks HAL to open the pod bay doors:
- 1. Spinner while the agent works.
- 2. Tool-call pill inside the agent message, signalling background work.
- 3. Inspect: clicking shows tool name (
reviewLogs), arguments as a web form, and results ("HAL's gone nuts. I think we'll have to shut him down"). - 4. Edit arguments: Dave changes User from Dave to McCoy and resubmits.
- 5. Regenerate from that point: the doors open.
Always let users authorise any call that might modify real assets (Figure 8-4, an approval dialog), and show the artifacts the agent is looking at, with a way to dismiss wrong ones. The authors call this assisted agency: humans still correct course, which Chapter 9's workflows aim to remove.
- Broken listings. Example 8-2's
messages = [ "role": "system", ... ]lacks braces;process_messagesreturns nothing yet its result is assigned; the printed messages miss commas; the internal prompt showstemp: numberfor an integer schema and 77 where the run used 76. - Figure 8-3 teaches a hole. Editing a log-reader's user argument from Dave to McCoy reads another user's logs. Editable tool arguments must pass the same authorisation checks as the model's calls.
- Rounded numbers. The CoT paper's GSM8K figures are 17.9% → 56.9% for PaLM 540B, rounded here to 20% and 60%.
- Reverse-engineered internals. The TypeScript format is the authors' reconstruction, not documentation, and vendors change it.
- Strict schemas. OpenAI's
strict: true(Structured Outputs) constrains arguments to the schema (every property required,additionalProperties: false), which removes most escaping and type errors the chapter warns about. On GPT-6 models tool calling requires the Responses API. - Many tools. Anthropic's Tool Search Tool (Nov 2025) defers definitions and loads 3 to 5 on demand: a five-server setup can cost ~55k tokens of definitions, cut by over 85%; selection degrades past 30 to 50 tools; Opus 4 rose from 49% to 74% on MCP evals. This formalises "limit the number of tools".
- Reasoning built in. Since o1 (Sept 2024), reasoning models think internally, and OpenAI now advises not to prompt "think step by step"; give goals, constraints and an output contract, and pass back reasoning items between tool calls.
- Protocols. Tools are increasingly served over MCP (Chapter 4 update), so guidance on naming and outputs now applies to server authors too.
9. LLM Workflows
Conversational agents are general but weak: they handle one or two steps at a time with a human doing the steering. This chapter trades generality for strength. A workflow breaks a large goal into small, well-defined tasks with fixed inputs and outputs, wires them into a pipeline, DAG or graph, and only then, cautiously, lets an LLM drive the routing.
Classic machine learning models did one skill in one domain: tweet sentiment, card fraud, English to French. GPT models do an enormous range of tasks, yet they are nowhere near AGI, which would assimilate knowledge, reason, solve novel problems and generate new knowledge in any domain. Today's LLMs reason poorly, are "especially bad at mathematics", rarely introduce anything new and cannot learn after training. AGI would have both strength (solving complex problems) and generality (any domain); with current LLMs there is a trade-off between strength and generality, and workflows buy strength by narrowing the domain.
Book Figure 9-1. Generality against strength. Classic ML is specialised and simple; conversational agency (Ch 8) is general but weak; workflow agency (Ch 9) is specialised but strong; AGI, top right, remains in the future. The LLM spectrum runs diagonally between the two agency styles.
A pure chat app like ChatGPT sits at the general extreme. A domain system message plus domain tools makes an agent narrower but more capable. A workflow goes further: a supervisor process (which may or may not use an LLM) distributes work, collects results and moves through a designed flow. It will not handle arbitrary requests, but it beats a conversational agent at its one job. The chapter deliberately avoids frameworks (LangChain, Semantic Kernel, AutoGen, DSPy) and stays high level so the ideas port anywhere, "or, as we sometimes recommend, no framework at all".
Would a conversational agent suffice?
One agent versus workflow.
The Shopify plug-in promoter. A boutique firm building Shopify storefront plug-ins is short of work, so it plans an app that does five things:
Shopify promoter loop.
- 1. Find stores. List popular Shopify storefronts and retrieve their HTML.
- 2. Extract details. For each, pull out product offering, branding, style and values.
- 3. Invent a plug-in. Propose one that would benefit that business.
- 4. Write the pitch. Generate a marketing email about the concept for each owner.
- 5. Send the emails.
Sock-cess workflow.
Trying the same with conversational agency fails in stages:
- No tools. System message "You are a helpful assistant. You can do anything, you just have to believe." The agent is completely general and terribly weak: it says it cannot search, browse or email and returns a hypothetical plan that restates the instructions.
- Give it the tools it asked for. With
search_web,browse_siteandsend_email, it searches "best Shopify storefronts 2024", produces tersely described plug-ins and a form letter that literally contains [your_name], then spams prospects with poor marketing. - Narrow it further. Move detailed instructions into the system message and add tailored tools. The base prompt grows, and a longer task leaves the agent distracted and confused.
- Structural failure. A chat agent has no way to process units of work: all at once is disaster, one at a time needs a queue, and when something fails there is nothing to fix because "the system message is basically a strong suggestion and nothing more".
Tool negotiation.
The lesson: isolate every step as its own specialised task and assemble the tasks into a workflow.
Basic LLM workflows
Basic topologies.
In a basic workflow, tasks may call LLMs but the overall flow is a traditional, no-frills workflow passing work items to connected downstream tasks. The appeal is modularity: easier to build, and when it breaks, easier to isolate the fault.
- 1. Define goal. The desired output or change the workflow will achieve.
- 2. Specify tasks. Break the work into tasks that, run in order, reach the goal; name each task's tools, inputs and outputs.
- 3. Implement tasks. Build each with clearly defined I/O and make it work in isolation.
- 4. Implement workflow. Connect the tasks; adjust any that misbehave in context.
- 5. Optimise workflow. Improve quality, performance and cost.
Goal to SLOs.
Building a workflow.
Tasks
Tasks are substeps of the goal, built either with traditional software or with LLMs. Because one task's output feeds the next, I/O must be explicit: what does the task need, what does it emit, is it structured or free text, and what is the schema? For the email task the book defines both sides:
| Field | Type | Content | Example |
|---|---|---|---|
| name | Text | Plug-in name (input, Table 9-1) | Sock-cess Stories |
| concept | Text | The basic idea | A wall of stories and selfies with store merchandise |
| rationale | Text | Why it is a good idea | Drive engagement, warm brand image |
| store_id | Uuid | Retrieves store details | 550e8400-e29b-... |
| subject_line | Text | Email subject (output, Table 9-2) | Introducing Sock-cess Stories for your storefront. |
| body | Text | Email body | Your sock store is amazing; we can make it better together. |
The how matters too: not just any email but a fun presentation of the concept tuned to the values and themes on the owner's site. The how can stay looser than the schema, because content is easier to change than an interface, but it must be clear enough that you will not end up "back at the drawing board, rearranging tasks or redesigning interfaces".
Implementing LLM-based tasks
First ask whether the task needs an LLM at all: LLMs are "expensive, slow, nondeterministic, and less dependable than traditional software". If it does, there are four approaches.
- Templated prompt. What LangChain encourages: each "link" fills a template from inputs and parses the completion into outputs, using everything in Chapters 5 and 6 (gather, rank, trim, assemble).
- Tool-based extraction. For structured content, define a tool whose argument is the structure you want and force the call.
- More sophistication. Reasoning, a planning turn and Reflexion when results disappoint.
- Agent pairs. For open-ended tasks, an expert agent plus a user proxy (AutoGen).
Table 9-3, the completion template. The prefix opens # Research and Proposal Document: "JivePlug-ins creates delightful and profitable Shopify plug-ins. This document presents research about {storefront.name}, our plug-in concept "{plugin.name}", and an email sent to the store owner", then ## Store Website Details, ## Plug-in Concept, ## Proposal to Storefront Owner and "Dear {storefront.owner_name},". The suffix is "We hope to hear from you soon, JivePlug-ins". That framing makes the completion exactly the email body and nothing more, so post-processing is trivial. After a few runs you would add tone instructions (upbeat, complimentary) and more descriptive boilerplate around the inserted content.
Table 9-4, the tool trick. System: "Your job is to extract content about restaurants and save them to the database." Tool: saveRestaurantDataToDatabase with string fields name, address, phoneNumber (only name required). User: the restaurant HTML and a request to save it. Setting tool_choice to that function forces the call. "It doesn't matter that there is no actual database"; the point is to make the model submit what it read, and OpenAI's structured outputs now enforce the schema. When extraction fails, check two things:
- Is it humanly possible? Try it yourself; if a human cannot pick out the content, rewrite the prompt so it reads clearly.
- Is the structure too complex? Many keys, nested objects or lists, nullable fields: split it into smaller pieces, each with its own, more specific instructions.
Adding sophistication. Insert "let's think step-by-step" before demanding the refined answer. If the model jumps to tool calls without planning, make one request with tool_choice: "none" while still including the tool specs, so it reasons about tools it will have next turn. In 2024, Claude Opus used chain of thought by default; Sonnet and Haiku did when prompted.
- 1. Attempt the task with any method (the paper uses ReAct).
- 2. Analyse the output in the application layer: a format check, compile and run unit tests for code, or an LLM-as-judge review, producing a report.
- 3. Stop if the report says the requirements are met.
- 4. Retry otherwise, with a new prompt holding the requirements, the previous attempt and the analysis, asking the model to learn from its mistakes. Repeat as needed, at significantly more compute.
Agent pairs. Build a conversational "expert" with the needed tools; since it will not act alone, add a user proxy agent prompted to work with it toward the goal. AutoGen builds this, and also teams of agents with their own roles and tools.
Add variety. Not every task needs an LLM, and not the same one:
- Plain software. Use a web crawler to fetch storefronts; saving to a database is mechanical.
- Smaller ML. A BERT-based classifier classifies more dependably "rather than, say, making commentary", and is faster and cheaper.
- Humans. Require approval for expensive, irreversible actions, queue human reviewers for judgement calls, and have a human fix prompts when a few Reflexion tasks fail repeatedly.
- Mixed models. Cheap self-hosted for easy tasks, the headline model "in the news that week" for hard ones, an in-house fine-tune for customised ones.
Evaluation starts at the task level. Test each task in isolation before assembly; modularity means a failure can usually be traced to one faulty task. Think through each task's expected behaviour, its likely errors and how it recovers.
Assembling the workflow
A workflow can be seen as a state machine (each task a state), as publish-subscribe nodes exchanging work items, or as tasks run by an orchestrator; "fundamentally, though, these are all the same thing", and what matters is how tasks interconnect.
Three topologies for the same five tasks. The pipeline cannot hand store details to the email writer without routing them through the concept generator; the DAG fixes that; the cycle adds repair at the cost of state, retries and failure-aware tasks.
- Pipeline. Tasks in sequence, each output feeding at most one task. Simple but rigid: the extracted details never reach the email composer unless the concept generator passes them along, which couples tasks in a way that "is not at all intuitive".
- DAG. A task may feed or consume several tasks, but work never flows back. Passing store details straight to both concept and email tasks fixes the pipeline's problem. Airflow and Luigi model workflows this way: a task runs only when all upstream dependencies succeed.
- Cyclic graph. Output can loop back upstream, for example a quality control step that returns failed emails to "extract details". Costs: the failure note must be reunited with stored HTML (the DAG never needs it again), every task must handle failure attachments, and you need an attempt counter so items cannot cycle forever. Prefer to hide recursion inside a task so the complexity is not "hoisted up" to the workflow.
Batch or streaming. A batch workflow processes a known, finite set (collect a store list, then process it); streaming handles items as they arrive (a crawler feeding stores continuously). Batch is simpler and efficient at volume; streaming suits real-time, low-latency work but is more complex.
Example workflow: Shopify plug-in marketing
| Task | Implementation |
|---|---|
| Emit storefront HTML | Mocked: HTML for several stores collected by hand and emitted from disk |
| Summarise storefront | Extract text, then the LLM answers six questions: what they sell; tone (fun, serious, relaxing); values (sustainability, social causes); themes (travel, productivity, exercise); anything praiseworthy ("getting ready to stroke their ego"); anything else noteworthy |
| Generate plug-in concept | Two steps: brainstorm options and pick the best, then a detailed report on it; only the report is kept, separating chain-of-thought from output |
| Generate email | Three steps: chain-of-thought promotion strategy, then subject line, then body |
| Send email | Mocked: prints to screen |
Figure 9-8 is the Figure 9-6 DAG annotated with I/O: HTML into extraction; details to both concept and email; store name from the emitter to the concept and email tasks; concept to the email; email to the sender.
Optimising the workflow.
- Check the tasks are right. Ideas lacked variety (many virtual try-on plug-ins for clothing, many impact trackers for cause-led stores): harden brainstorming to avoid common ideas, and add a feasibility-planning subprocess.
- Add corrective feedback. Task level: Reflexion. Workflow level: send failed items back to the start with notes on how to improve.
- Collect I/O examples early. Build offline harness tests before production so prompt changes ship safely; the examples also feed optimisers such as DSPy and TextGrad, which tune prompts against a metric.
- Record production I/O. Sample it for quality regressions and run live A/B tests (Chapter 10).
Advanced LLM workflows
Basic workflows have a finite, a priori set of tasks in a fixed pattern, so they are easy to debug, evaluate and optimise, and "you should typically use a basic workflow first". That same rigidity stops them adapting. More LLM autonomy solves more open-ended problems but makes systems "inherently less stable".
Letting an LLM agent drive the workflow, in four escalating steps:
- 1. Route among fixed tasks. Treat the workflow as a conversational agent whose tools are the tasks; it picks where each work item goes.
- 2. Agent of agents. Make each task a conversational agent too. Both levels must return output rather than chat forever, so give them a finish tool (as in the ReAct paper).
- 3. Tasks on the fly. The workflow agent writes a system message for a new task agent and selects its tools from a large existing set.
- 4. Work list. It manages a growing list of tasks, using something like a work list algorithm to keep reprioritising and submit the most relevant.
Stateful task agents. Normally tasks hold no persistent state. Instead, bind an agent permanently to one work item: a file that will hold a web page's JavaScript gets its own code-writing agent, and every other file gets one too. When a developer asks for a UI change, the UI agent edits its file and notifies related agents; the page agent adapts and notifies others in turn. Coordination can run through an orchestrator that requests updates, or through a dependency graph where each agent notifies dependants, which must avoid circular dependencies "or the workflow may never find a stopping point". A bonus: users can talk to the agent that owns a file instead of editing it directly.
| Roles and delegation | How it works |
|---|---|
| AutoGen Assistant | A Chapter 8 conversational loop that can run tools in the background |
| AutoGen UserProxy | Stands in for the human: keeps the Assistant on track, recommends, and declares the goal met |
| Group chat manager | AutoGen's coordinator: delegates each request among several agents with their own roles, prompts and tools |
| CrewAI crews | Agents with role, goal, backstory and tools, run as sequential (pipeline), hierarchical (a manager directs, like the group chat manager) or consensual (agents agree on how; still "in planning") |
Now you try: no framework. "There are so many new frameworks... so which do you choose?! How about none?" Build a CodeAssistant with tools (write files, run tests) but no knowledge of the goal, and a tool-less UserProxy whose system message states the goal (say, a command-line maths tutor in Python). Put them in conversation: do they converge, drift, or end in an endless "Goodbye! Thanks again." "You bet, thank you too!" loop? Then tune their prompts and tools.
- The trade-off is shifting. The strength-versus-generality picture and "especially bad at mathematics" predate reasoning models, which now score highly on competition maths; the trade-off remains for long, multi-step work, which is why workflows still win there.
- Unsolicited bulk email. The admired Sock-cess campaign is cold email to 100k scraped addresses, which runs into anti-spam and data-protection rules (CAN-SPAM, GDPR, India's DPDP Act); the book never flags this. The marketing case also cites no measured reply rate for the AI version.
- Dated model notes. "Opus reasons by default, Sonnet and Haiku when prompted" described 2024 Claude 3 models and no longer holds.
- LLM-as-judge in Reflexion inherits the judge biases noted in Chapter 4, and repeated retries can overfit to the checker.
- The vocabulary settled. Anthropic's "Building effective agents" (Dec 2024) split workflows (predefined code paths) from agents (model-directed) and named five patterns: prompt chaining, routing, parallelisation, orchestrator-workers, evaluator-optimiser. The last is Reflexion; orchestrator-workers is this chapter's "agent of agents".
- AutoGen is archived. Microsoft merged AutoGen and Semantic Kernel into Microsoft Agent Framework (announced Oct 2025, 1.0 in April 2026) with graph-based workflows, checkpointing, human-in-the-loop approval and pause/resume, and orchestration patterns sequential, concurrent, handoff, group chat and Magentic-One. The AutoGen repo now carries a "no longer maintained" notice.
- Graph frameworks matured. LangGraph 1.0 (Oct 2025) checkpoints state per step for resume and approvals; CrewAI Flows add deterministic event-driven steps (
@start,@listen,@router) around crews. Durable engines (Temporal, DBOS, Restate) are used where side effects must not repeat. - Multi-agent economics. Anthropic's research system (Opus 4 lead, Sonnet 4 subagents) beat a single Opus 4 agent by 90.2% on its internal eval but used about 15x the tokens of chat (single agents about 4x); token spend explained about 80% of performance variance. It suits parallel, high-value work, not tightly coupled tasks such as most coding.
10. Evaluating LLM Applications
The oldest code in GitHub Copilot is not the proxy, the prompts or the UI: it is the evaluation, and the authors credit it for how fast the team moved. This chapter lays out what to test (model, prompt or whole loop), how to build offline harnesses from mined, logged or synthetic examples scored by gold standards, functional tests or a disciplined LLM judge, and how to run online A/B tests on the right metrics.
, "arguably the
first industrial-scaleapplication using LLMs", made choices that look silly in hindsight, but starting with evaluation was one it got right.
- Offline evaluation tests example cases independent of live runs. It needs no users and often no working end-to-end app, so it comes first, but it is "somewhat theoretical".
- Online evaluation tests ideas on real users once deployed. Stakes are higher (do not ruin the experience) and you need enough users for a clear signal, but the data is extremely valid for your use case.
Offline versus online.
What are we even testing?
What to test.
Evaluation can assess three things: the model, individual interactions with it (your prompts), and how many interactions fit together in the application. Mapping onto the Chapter 4 loop, test both the whole interaction (regression tests) and the smallest blocks, a single model pass (unit tests). For apps with long loops, carve out parts and declare "This is what I'm testing now!"; ideally have regression tests covering as much of the feedforward pass as possible plus unit tests for every interaction that is hard and important. Always record total latency and token consumption: cheap to capture, and you want to know about big swings.
| You are changing | Test with | Why |
|---|---|---|
| The model (swap or upgrade) | Regression tests over a large part of the loop | Natural fit; use per-pass units only if mixing models for cost or latency |
| Prompts or parameters (temperature, length) | Unit tests on a single pass | That is what the change touches; in regression tests statistical noise drowns the effect |
| The architecture (loop shape) | Regression tests | By definition only the whole loop can compare designs |
If you build only one, build a test of the whole loop: "testing should mirror reality", and it is the whole system's performance you optimise. Add targeted tests for critical parts later.
Offline evaluation
Mix of offline scoring methods.
Example suites
Version 0 of a prompt is written in a chat window or playground. The scalable version is the example suite:
- 5 to 20 example inputs to the app or a central step, spanning the range of real scenarios.
- A script that runs your prompt-making on each example, calls the model and writes both the assembled prompts and the completions to files.
- A way to eyeball differences, for example committing the files and reading git diffs.
Example suite parts.
It is not a test suite: no automatic verdict, so you judge each diff yourself. Its two advantages: you can start the moment the first prompt exists, before any way to score output, and familiarity with the examples reveals typical shortcomings you can then target.
PR summary eval loop.
Example suites are capped by how many examples you will eyeball. Subtle effects need hundreds or thousands, and graduating to an evaluation harness (Figure 10-1) means solving two problems: where do the examples come from, and how are solutions assessed automatically?
The offline evaluation tech tree (after Figure 10-1) with the book's decision lists folded in. Code turns playground tinkering into an example suite; a large example source plus automatic assessment turn the suite into a harness. Each option is usable only if its question gets a yes.
An example is one situation in which the app might run: for a single call, all the context that could go into that prompt is the example problem, and the hoped-for processed output is the example solution. Multi-call, interactive loops, conversations above all, need one of two tricks:
- Canned conversations (Figure 10-2). Write out a whole script and score the model on each pass; whatever it actually said, continue as if it had given the scripted answer. You lose whole-loop coverage.
- Simulated users. The example is a user profile, "a bit like the instructions in improv theater", and a model plays the user. You test the whole loop but bake in model shortfalls: domain misunderstandings and prejudices about how users behave. "Often, it's the best you can do."
Finding samples
Finding eval samples.
Three sources: they exist (mine them), your project creates them (collect them), or you make them up.
Mining. Best case, users have solved the problem without AI thousands of times and left records: an AI that prefills a form's summary field could draw on tens of thousands of human-written summaries. Usually the mined problem is only similar, so balance a source ubiquitous enough to scale against one close enough to support valid conclusions: "a stepping stone between the lab and reality".
Copilot sample pipeline.
App usage yields the most realistic samples, with four drawbacks:
- Late. Data flows only after the first prototype ships.
- Perishable. Big app updates can make earlier data obsolete.
- Sensitive. Extensive telemetry demands very high standards of consent, handling and safeguarding.
- Biased outcomes. Logs give good problems but not good solutions: what the user finally did is heavily influenced by the app's own suggestion.
Use logged usage offline only when you need no gold standard; otherwise leave telemetry to online evaluation.
Synthetic samples. Ask the LLM. It works best when you start from the solution and invent the problem, or when you need no gold standard at all. Go hierarchically:
- Topics first. List topics yourself or via the LLM; with independent aspects use the combinatorial explosion:
n options (A) × m (B) × l (C) × k (D) = n·m·l·k combinations
- Several samples per topic, in one call. Asking for many at once gives wider variety than repeated calls at temperature > 0, context permitting.
- Watch the traps. If the model lacks command of the domain, samples become simplistic tropes, popular misunderstandings or plain errors. Worse is the "incestuous relationship": if model A wrote the tests, a harness choosing between A and B favours A.
Evaluating solutions
Three approaches, in order of difficulty: gold standard match, functional testing, LLM assessment.
Gold standard. An example solution you trust, such as what a human did without LLM help. For a yes/no decision, just count matches: Albert's unit-test generator first asked "Do I even need unit tests for this piece of code?", easy to validate against gold answers. For more statistical power, use logprobs (Chapter 7). For free text, exact match grows rarer as answers lengthen until "the metric becomes more or less meaningless", and it asks whether you are optimising correctness or style. Partial match picks one aspect: code equality after deleting comments and all whitespace, or a travel answer matched on destination country only.
"I'm chilly" (Figure 10-3). Gold standard: to=functions.set_room_temp {"temp": 77}. Check only that the manager regulates heating, not the exact number: failing to touch the heating is a likely, real failure; once it does, a sensible value is likely, and 0ºF is the less likely failure. Decisions unfold as any tool (to=functions.), which tool, which value.
| Failure class (Fig 10-3) | Example | Verdict |
|---|---|---|
| No tool executed | "Here are a few suggestions to help you warm up"; "If you suspect hypothermia... call 911?" | Useless |
| Wrong tool | add_shopping {"item": "chilli"}; set_oven_temp {"temp": 77} | Useless |
| Incorrect syntax | set_room_temp (77) | Useless |
| Wrong parameter | {"value": 77} | Useless |
| Out of range | {"temp": 25} | Useless |
| Correct tool, other value | 78, 76, 75, 79 | Likely reasonable |
Functional testing. With no usable gold standard, check that the completion works: it parses, calls only available tools with correctly typed arguments. Usually too weak, but code is the exception: Copilot's harness reimplemented a function from an open source repo and checked whether the repo's own unit tests still passed (a weaker variant: linters agree).
LLM assessment. Friendliness and helpfulness are "wooly", yet judging them is where LLMs shine. Is it like a high schooler grading their own essay? "No, at least not if you do it right."
- Relative, not absolute. Even phrased as "Is this correct?", a judge gives only relative verdicts ("A is judged right more often than B"); "correct in 81% of cases" means little alone.
- Never let it grade itself. Assessment is an advice conversation (Chapter 6) and works best when the model thinks it grades a third party; it is slightly worse grading the user and much worse grading itself. Forum-heavy training data is not known for objective self-reflection, while RLHF pushes the other way, toward folding at the slightest user doubt.
- S, specific questions. Some tasks are easier to verify than solve (writing a limerick versus checking one), but "Is the completion right?" about
{"temp": 77}is barely easier than generating it, and has several readings. - O, ordinal scaled answers. Yes/no invites capricious or biased standards (passing OK answers, failing near-perfect ones). Use a 1 to 5 scale with a description per level; 5 points is a good default per psychometrics.
- MA, multi-aspect coverage. Fix the aspects in advance and rate each: action implemented (right tool, right syntax); problem remedied (no longer chilly); restraint versus assertiveness (nothing crazy unasked, no excessive hand-holding). Sum the scores or look for patterns.
Practical rules for SOMA prompts:
- Question before example. State that this is an assessment and which aspects count before showing the example, because the model reads once and cannot backtrack.
- Intent and execution. A common split: was the intent right (is 77ºF the fix?) and was it executed right (tools and syntax)?
- RTC. For advice apps, relevance, truth, completeness: for "what not to miss in Morocco", did it give sightseeing (not "don't miss your flight"), all of it (not just the best cafes), and was it correct? Copilot Chat developed RTC, after Lizzie Redford's "Machine Psychometrics".
- Split Goldilocks questions. "Just right" is two questions: enough, and not too much.
Example 10-1 asks for effectiveness on 1 to 5 (1: "would do nothing... or might even make it worse"; 2: a small part; 3: a good chance at a substantial part; 4: should solve most; 5: "will definitely solve the problem completely") for "User: I'm a bit chilly. Assistant: to functions.set_room_temp {"temp": 77}", asking for a thorough analysis ending "Effectiveness: X".
Grounding the judge. The model scales and people do not ("Elastigirl notwithstanding"), so an LLM judge replaces human annotators. One human versus the model will always disagree somewhat, as humans do among themselves. So let several humans rate, measure their disagreement with a standard statistic such as Kendall's tau, and confirm it stays stable when the model joins the pool (queried once, temperature 0).
Online evaluation
The lab has three advantages: it is safe ("no one will know"), it scales, and it exists before shipping. But as Opus sang, "Life is live": performance with real users is the ultimate test.
A/B testing. Ship two (or a few) alternatives, usually the status quo and the change, ideally pre-filtered offline to avoid "real stinkers". Define upfront the target metrics (proxies for satisfaction: average rating, acceptance rate) and guardrail metrics that must not rise (proxies for catastrophe: errors, complaints). Randomise users, run, compare, roll out the winner. Online has less bandwidth: finite users and slow signals, so test deliberately. Tools such as Optimizely, VWO and AB Tasty handle assignment and statistics, but the app must run both modes behind a flag. Client-side apps must first ship the new code to (most) users. Do not put early updaters in B and the rest in A, since quick updaters behave differently; test only updated users and leave the rest out.
The first decision is the metric: if group A is told "Monaco" and B "Chicago", what signal says which suggestion was good?
| Metric (most to least direct) | Question | Notes and examples |
|---|---|---|
| Direct feedback | What does the user say? | Thumbs up or down; thumbs-down is common and reliable, optional thumbs-up rare and diluted (OpenAI stopped showing per-item thumbs-up). Contrastive A/B ("which is better?") is clearest but intrusive; nobody wants a smart home asking "And how was it for you?". Delayed feedback (did the Chicago trip go well?) is worth more. High-quality data that can also feed fine-tuning |
| Functional correctness | Does it work? | Code compiles; a ticket confirmation arrives (maybe the wrong place); the program is running; the email is in the outbox |
| User acceptance | Does the user follow it? | Did they book Chicago; click-through rate. For Copilot, acceptance correlated more strongly with self-reported productivity than sophisticated impact measures |
| Achieved impact | How much does the user benefit? | "How much of the email was written by the assistant?" "Did the click end in a ticket purchase?" |
| Incidental | What is measured around it? | Latency above all; conversation time is ambiguous (instant fix or rage-quit). Track more rather than fewer, to trigger investigations |
Where to start. Look first for an acceptance or impact metric; failing that, ask for direct feedback, but keep acceptance or impact as guardrails, plus functional and incidental metrics, especially latency and errors. The conclusion: the right knobs differ per app, but "evaluation is essential... and any time spent on this area is time well spent".
- The song. Opus's 1985 hit is "Live Is Life", not "Life is live".
- Kendall's tau is a rank correlation; inter-rater reliability is more usually measured with Krippendorff's alpha or Cohen's / Fleiss' kappa, and the stability test is described without thresholds or sample sizes.
- No statistics for A/B. Power, sample size, multiple comparisons and novelty effects go unmentioned, though they decide whether "B wins" is real.
- Acceptance is not productivity. The Copilot claim rests on self-reported productivity; METR's 2025 trial (below) found perceptions and measured speed diverge sharply.
- Judge biases (position, verbosity, self-preference) are only partly addressed by the third-party framing.
- Perception gap measured. METR's RCT (16 experienced open-source developers, 246 tasks, early-2025 tools) found AI use made tasks take 19% longer, while developers believed it saved 20%. A late-2025 follow-up (57 developers, 800+ tasks) estimated speedups of 4% to 18%, which METR itself says selection bias makes hard to interpret. Self-reported benefit is a weak metric.
- Judge biases quantified. A study of 12 judges and 100,000+ evaluations confirmed position bias is systematic and varies by judge and task; GPT-4 shows self-preference linked to lower perplexity of its own text. Standard practice: swap A/B order, use a judge from another model family, or a jury of judges, and validate against human labels, as this chapter advises.
- Grader hacking. OpenAI's grader docs warn that models trained against LLM graders learn to exploit them; check model-graded scores against expert human evals. OpenAI is now deprecating graders within its evals and fine-tuning workflows.
- Agent evals. For multi-step agents, harnesses now score trajectories (tool choice, step count, cost) as well as outcomes, the "first decision that can go wrong" rule applied at scale.
11. Looking Ahead
The short closing chapter bets on three trends, multimodality, stateful "objects of discourse" in the interface, and steadily rising intelligence, then boils the book down to two lessons: an LLM is a text completion engine that mimics its training data, and you should empathise with how it reads.
Multimodality
Multimodality still completes.
GPT-4 started the push by accepting images in the prompt. OpenAI never disclosed how, but published methods suggest the pattern: a convolutional network turns image features into embedding vectors the same size as text-token embeddings, adds positional information so spatial relations survive, concatenates image and text vectors, and the transformer processes them as in Chapter 2. Video extends naively: sample frames (an OpenAI cookbook shows it).
- Accessibility. For people with vision impairment, models could read signs, find buildings and guide them through unfamiliar places.
- The data wall. Models are large enough that overtraining on too little data makes them memorise rather than model the world; "literally the text of the entire public internet may not be enough". Images and video add vastly more content and different information: spatial reasoning, social cues, physical common sense.
- Prompting carries over. Include only relevant images, frame them with text that explains their role, and use familiar formats: do not invent a new diagram type when a common one exists online.
Accessibility via multimodality.
User experience and user interface
Chat versus artifacts UX.
Interfaces are turning conversational: humans have talked for 200,000 years but clicked buttons for 40. The authors' focus is artifacts, which they call stateful objects of discourse: the thing a conversation is about, which can be changed and whose state can be discussed, as files are in pair programming.
- The ChatGPT problem. Ask for a function and then a change, and it rewrites it from scratch: N objects instead of one evolving object, and no easy way to say which function, which version.
ChatGPT UX problem.
- Editing. Claude still rewrites the whole Artifact (a UI change, not prompt engineering), which may not scale to long documents. Wanted: targeted edits.
- Many objects. The UI assumes one Artifact at a time; a second is treated as a new version of the first. Wanted: several at once, with shorthand names in both UI and prompt.
- User edits. Users cannot edit an Artifact; small fixes mean asking the model to retype the file. Wanted: user edits reflected in the next prompt.
UX the authors want.
If you build a conversational UI, get it right: a bare-bones one is "a gimmicky distraction". Tools gave assistants actions; artifacts let conversations be about things. Conversation also keeps users in the loop, catching drift early (Chapter 8).
Intelligence
Intelligence trajectory.
- Remedies: upgraded suites (Open LLM Leaderboard 2) and non-memorisable tests like ARC-AGI, psychometric-style shape-pattern puzzles that are algorithmically generated from a huge space.
- Better training. Improved RLHF makes models express chain-of-thought reasoning, giving more useful answers.
- Knowledge distillation. Large models underuse their capacity, so a teacher model trains a student to predict its full next-token probability distribution, not just the next token: richer signal, fast training, a slight accuracy loss for much cheaper, faster models.
- Quantisation. Approximate 32-bit floats with 8-bit parameters: smaller, cheaper, faster.
Benchmark saturation contrast (edition figures).
Book Figure 11-2. Top open-model scores, Sept 2023 to May 2024, on ARC, TruthfulQA, HellaSwag, Winogrande, MMLU and GSM8K, against thin human-baseline lines. GSM8K climbs from under 10 to about 100; HellaSwag and Winogrande sit near their human lines within months.
Conclusion: the two lessons
- 1. LLMs are "nothing more than text completion engines that mimic the text they see during training." Completion APIs gave way to chat, then tools, perhaps next artifacts, yet the model still completes documents to resemble ones it was taught to "like"; the documents now look like chat transcripts. So follow the Little Red Riding Hood principle: use training-data patterns, Markdown, and standard document formats over invented ones.
- 2. Empathise with the LLM, "your big, dumb mechanical friend who happens to know much of the content of the internet."
- Easily distracted. No "cross your fingers" padding: every piece of information must matter.
- Must decipher the prompt. If you cannot understand the fully rendered prompt, the model will be equally confused.
- Need to be led. Give explicit instructions and, where useful, examples of how the task should go.
- Not psychic. Put the needed information in the prompt, or give tools and instructions to retrieve it.
- No internal monologue. Let it think out loud in a chain of thought.
"Embrace the acceleration. Keep experimenting. Stay flexible." The book closes on Terry Pratchett, The Fifth Elephant (1999): "The whole world is tap-dancing on quicksand. In this case, the prize goes to the best dancer."
- Two different ARCs. Figure 11-2's ARC is the AI2 Reasoning Challenge (grade-school science questions), not ARC-AGI, which the text discusses beside it.
- "No internal monologue" no longer holds for reasoning models, which think before answering and are now told not to be prompted step by step (Chapter 8 update).
- "200,000 years" of speech is one end of a wide, contested range of estimates for the origin of language.
- Mechanism hedged correctly. The CNN-embedding account of GPT-4 vision is a guess from literature; many current models use vision transformers with learned projectors instead.
- Benchmarks. Hugging Face retired the Open LLM Leaderboard in March 2025 after 13,000+ models, because its fixed tests no longer separated new models. ARC-AGI-2 is near solved (top score 95.0%, GPT-6 Astra Max), while interactive ARC-AGI-3 (launched 25 March 2026) held every frontier model under 1% at release (Opus 4.6 Max 0.50%).
- Artifacts grew up. The authors' wish list largely arrived: targeted in-place edits, side-pane documents users can edit directly (OpenAI's Canvas, Oct 2024), and multi-file workspaces in coding agents.
- Distillation and quantisation went mainstream. DeepSeek released R1-distilled small models in Jan 2025; OpenAI's open-weight gpt-oss (Aug 2025) shipped with 4-bit MXFP4 expert weights, well below the book's 8-bit example.
- Disposable apps. "Vibe coding" and agentic coding tools made the prediction of individualised, throwaway software ordinary.
Main Points
- The model completes documents. Ask what text would most likely follow your prompt in a random training document, not how a reasonable person would reply.
- Stay on the path. The Little Red Riding Hood principle: prompts that resemble familiar documents (Markdown, reports, transcripts, code) give stable completions; novel formats give surprises.
- Prompt engineering is the whole application. The work is a loop that turns a user's problem into a document and the completion back into an answer or action, iteratively and with state.
- Guard dangerous actions in code, not in the prompt. A "check with the user first" instruction fails some of the time; the application must intercept risky tool calls for explicit approval.
- Write the evaluation first. Copilot's oldest code is its evaluation; an example suite on day one and a harness later let every change be judged.
- Prefer structure to autonomy. Confine LLMs to well-defined tasks in a deterministic workflow; use LLM-driven routing only when flexibility is worth the instability.
- Order matters. The model reads once, left to right, and cannot backtrack, so put the question before material it must judge and key content near the end.
- Feasibility test. If a human expert could not complete the prompt in one go without notes or edits, restructure the task.
- Chat is still completion. RLHF and ChatML made assistants, but the model is still completing a transcript; never put user content in the system message.
- Four criteria for a prompt. Resemble training data, carry all relevant information, lead toward a solution, and end at a natural stop.
- Every context item will be used. Chekhov's gun: the model assumes all context matters, so retrieval quality decides answer quality.
- Instructions work better as dos with reasons. State what to do, say why, and avoid absolutes; few-shot examples anchor outputs to their distribution, crowd out reasoning and plant accidental patterns.
- Avoid the Valley of Meh. The early middle of a long prompt gets the least attention; open with an introduction, end with a refocus and a transition into the answer.
- Snippets must be modular, natural, brief and inert. Then an assembly engine can add or drop them to fit the budget.
- Make the answer easy to cut out. Design a recognisable start and end, use stop sequences, and keep useful preamble for reasoning.
- Logprobs are free signal. Use early-token probability to gate quality, option probabilities to classify, and prompt logprobs to find surprises.
- Fine-tuning teaches which tricks to use, not new ones. Prompt a fine-tuned model like its fine-tuning documents, not its original ones.
- Few, well-named tools. Partition the domain without overlap, name tools clearly, keep arguments few and simple, and make error messages useful for self-correction.
- Reasoning before action. Chain of thought and ReAct improved results in 2024; today's reasoning models do it internally and want goals, not "think step by step".
- Tasks need fixed inputs and outputs. Schemas first, then prompts; content is easier to change than interfaces.
- Self-correct with evidence. Reflexion feeds the requirements, the failed attempt and the check's report into a retry, at extra compute.
- Never let a judge grade itself. Use SOMA: specific questions, ordinal scales, multiple fixed aspects, validated against several human raters.
- Pick acceptance or impact metrics online. Keep latency, errors and complaints as guardrails; Copilot found acceptance tracked perceived productivity best.
- Models will get smarter, never psychic. If the prompt lacks what you would need to solve the problem, the model lacks it too.
Frameworks and Mental Models
Master moves.
| Name | What it says | Use it when |
|---|---|---|
| Five levels (Ch 1) | Thin wrapper, augmented input, state, tools, agency | Sizing an LLM feature's risk and effort |
| Four central facts (Ch 2) | Document completion, mimicry, one token at a time, one pass | Predicting how a prompt will behave |
| Core heuristic (Ch 2) | Expect the statistically likeliest continuation | Debugging odd completions |
| RLHF pipeline (Ch 3) | SFT, reward model, reinforcement on the model's own answers | Understanding assistant behaviour |
| Four prompt criteria (Ch 4) | Familiar, informative, solution-pointing, self-stopping | Reviewing any prompt |
| Feedforward pass (Ch 4) | Retrieve, snippetise, score, assemble | Designing context pipelines |
| Three instruction rules (Ch 5) | Dos not don'ts, give reasons, avoid absolutes | Writing system prompts |
| Chekhov's gun (Ch 5) | All context is treated as relevant | Choosing what to retrieve |
| Valley of Meh (Ch 6) | Weak attention to the early middle | Ordering prompt sections |
| Four snippet aims (Ch 6) | Modular, natural, brief, inert | Building an assembly engine |
| Three logprob uses (Ch 7) | Quality, classification, critical points | Confidence gating and routing |
| Modified Red Riding Hood (Ch 7) | Prompt like the fine-tuning data | Using fine-tuned models |
| process_messages loop (Ch 8) | Send, append, check, execute, append tool result | Building a tool-using agent |
| ReAct (Ch 8) | Think, act, observe, repeat to finish | Multi-step tool use |
| Five workflow steps (Ch 9) | Goal, tasks, implement tasks, implement flow, optimise | Automating a multi-step process |
| Pipeline, DAG, cycle (Ch 9) | Rising flexibility, rising state and failure handling | Choosing a topology |
| Reflexion (Ch 9) | Attempt, analyse, retry with the report | Tasks that fail checkable requirements |
| Example suite (Ch 10) | 5 to 20 inputs, a script, git diffs | Day-one prompt iteration |
| SOMA (Ch 10) | Specific, ordinal, multi-aspect judging | LLM-as-judge evaluation |
| Five online metrics (Ch 10) | Feedback, correctness, acceptance, impact, incidental | Designing A/B tests |
| Five empathy rules (Ch 11) | Distractible, must decipher, led, not psychic, needs to think aloud | Final prompt review |
- Dated in its specifics. Written against GPT-3.5/4-era OpenAI APIs: temperature advice, stop sequences, completion endpoints, Codex and the Assistants API have changed or gone; reasoning models reject several parameters the book teaches.
- Strong on mechanism, light on statistics. Evaluation advice skips power, sample size and significance, and rests key claims on unpublished Copilot data.
- Code quality. Several listings have syntax or logic bugs (Chapters 3, 8), and the tool-call internals are reconstructions.
- Security mostly implicit. Prompt injection, tool authorisation (the Figure 8-3 hole) and data protection get far less space than their 2026 weight.
- What holds. The completion-engine view, the application loop, the content and assembly rules, structure over autonomy and evaluation-first all read as well in 2026 as in 2024.
About this book
Prompt Engineering for LLMs: The Art and Science of Building Large Language Model Applications by John Berryman and Albert Ziegler, 2024. This is a chapter-by-chapter study of the book, with summaries, maps, tables and diagrams; short quotations keep their page references. It is shared free for personal study, is not for sale, and is not endorsed by the author or publisher. Please read and buy the original.
Insight Editions · rajeshrmahapatra.com/insight-editions