Insight Editions · No. 36
Hands-On Large Language Models
Language Understanding and Generation
Jay Alammar and Maarten Grootendorst
Jay Alammar and Maarten Grootendorst explain how large language models work and how to use them for classification, search, retrieval-augmented generation, and fine-tuning.
A study guide to Hands-On Large Language Models by Jay Alammar and Maarten Grootendorst. Shared free for personal reading; not for sale and not endorsed by the authors or publishers. Please read and buy the original.
Overview
| Core claim | Scope | Key figures | Best for |
|---|---|---|---|
| Language models are best understood visually and by hand: two families, representation models that encode and generative models that write, built from the same Transformer parts, and both worth using, combining and fine-tuning. | Preface, 12 chapters in three parts (understand, use, train), Afterword. Runs on a free 16 GB T4 in Google Colab. | Jay Alammar (Cohere, The Illustrated Transformer) and Maarten Grootendorst (IKNL, BERTopic); O'Reilly, 2024. | Python programmers with basic machine learning who want working intuition and code for classification, search, RAG, topic models and fine-tuning; architects choosing between encoders and LLMs. |
Critical lenses flag slips in the text and code.
The Book in Brief
- Intuition first. The authors skip derivations and teach through illustrations and runnable code, because the field moves too fast to chase; the bet is that fundamentals such as tokens, embeddings, attention and fine-tuning stay stable while model names churn.
- Two families, one architecture. Representation models (BERT-like, encoder-only) do not generate text but excel at classification, clustering and search; generative models (GPT-like, decoder-only) write. The authors use "large" loosely, since size says little about capability, and insist encoders still matter.
- Part I, Understanding (Ch 1-3). A history from bag-of-words to the Transformer; tokenizers and embeddings compared on one test string; and an updated Illustrated Transformer: the forward pass, the KV cache, attention and the 2024 refinements.
- Part II, Using pretrained models (Ch 4-9). Classification five ways, BERTopic clustering, prompt engineering, LangChain chains, memory and agents, semantic search and RAG, and multimodal models such as CLIP and BLIP-2, all without training.
- Part III, Training (Ch 10-12). Building embedding models with contrastive learning, fine-tuning BERT for classification and NER (including SetFit from 32 examples), and fine-tuning generative models with QLoRA and DPO.
- The working stance. Open models first, proprietary APIs (OpenAI, Cohere) where they teach a contrast, and local alternatives wherever a free API key would hit rate limits.
- Afterword. The book is "only just the beginning": the authors promise to keep adding resources to the repository and urge readers to keep following the field.
- What has changed since. Reasoning models (from September 2024), mixture-of-experts and hybrid architectures, Transformers v5, LangChain 1.0, 128K to 1M contexts and RLVR post-training have arrived; the book's mental models still explain all of them.
Figure 0. The book's arc: turn text into vectors, see how the Transformer uses them, apply encoders and decoders separately and together, then train embedding models, encoders and finally generative models.
Chapter Map
| Ch | Title | In one line |
|---|---|---|
| 1 | An Introduction to Large Language Models | Bag-of-words to Transformer; encoder vs decoder; the moving definition of "LLM"; responsible use; Phi-3 on a T4 |
| 2 | Tokens and Embeddings | Four tokenisation schemes; eight tokenizers on one string; token, text and word embeddings; word2vec song recommender |
| 3 | Looking Inside Large Language Models | Forward pass and sampling; KV cache 4.5s vs 21.8s; attention; MHA, MQA, GQA; RoPE |
| 4 | Text Classification | Five methods on Rotten Tomatoes, F1 0.78 to 0.91; zero-shot by embedding labels; GPT-3.5 by prompt |
| 5 | Text Clustering and Topic Modeling | Embed, reduce, cluster; BERTopic with c-TF-IDF; representation blocks; LLM topic labels |
| 6 | Prompt Engineering | Sampling knobs; seven prompt components; in-context learning; chains; CoT, self-consistency, tree of thought; grammar-constrained output |
| 7 | Advanced Text Generation Techniques and Tools | Quantized models; LangChain chains; three memory types; ReAct agents |
| 8 | Semantic Search and RAG | Dense retrieval and chunking; reranking; MAP; grounded generation; advanced RAG and evaluation |
| 9 | Multimodal Large Language Models | ViT patches; CLIP contrastive embeddings; BLIP-2 Q-Former; captioning and visual chat |
| 10 | Creating Text Embedding Models | Contrastive learning; SBERT bi-encoders; cosine vs MNR loss; hard negatives; Augmented SBERT; TSDAE |
| 11 | Fine-Tuning Representation Models for Classification | Full fine-tune F1 0.85; freezing; SetFit 0.84 from 32 labels; continued MLM; NER with BIO alignment |
| 12 | Fine-Tuning Generation Models | Three training steps; adapters, LoRA, QLoRA; TinyLlama instruction tuning; evaluation; reward models, PPO, DPO, ORPO |
- Jay Alammar. Director and Engineering Fellow at Cohere, advising enterprises and developers on practical language-model use. His blog's visual explainers (The Illustrated Transformer, BERT, GPT-3, Stable Diffusion) have reached millions and ended up in the documentation of NumPy and pandas; co-creator of courses on DeepLearning.AI and Udacity. Chapter 3 is an expanded Illustrated Transformer.
- Maarten Grootendorst. Senior Clinical Data Scientist at IKNL, the Netherlands Comprehensive Cancer Organisation, with master's degrees in organisational psychology, clinical psychology and data science. Author of BERTopic, KeyBERT and PolyFuzz, downloaded millions of times; Chapter 5 is BERTopic from its maker.
- Prerequisites. Python and machine-learning basics; no PyTorch or TensorFlow and no generative-modelling background assumed.
- Hardware. A free NVIDIA T4 with 16 GB VRAM on Google Colab is the minimum; some chapters need less, and the repository lists each chapter's GPU needs. Local setup: a conda Python 3.10 environment named
thellmbookandpip install -r requirements.txt. - Accounts. Free OpenAI and Cohere API keys (rate-limited; local alternatives provided) and a Hugging Face token for gated models such as Llama 2 in Chapter 2.
- Reading order. Each chapter stands alone, so skim what you know.
1. An Introduction to Large Language Models
Alammar and Grootendorst open with a compressed history of Language AI, from counting words to the Transformer, and use it to set the vocabulary of the whole book. The chapter's working claim is that "large language model" is a moving label: small encoder models that never write a sentence count too, and most useful systems combine several kinds of model. It ends with the practical stance of the book: open models, a free 16 GB T4 GPU, and a first joke from Phi-3-mini.
ChatGPT reached 1 million active users in five days and 100 million in two months. Within a year open and proprietary rivals were "closing in on, and eventually catching up to" it, and the authors call 2023 the year that changed their field. Their field is Language AI: systems that understand, process and generate human language, a term they use almost interchangeably with natural language processing (NLP) but stretch to cover non-LLM technology such as retrieval systems, which "give LLMs superpowers" (Chapter 8). They quote John McCarthy (2007): AI is "the science and engineering of making intelligent machines", and need not confine itself to biologically observable methods. The label is loose: game characters called "AI" are often nothing more than if-else statements.
The chapter sets out to answer four questions: what Language AI is, what LLMs are, what they are used for, and how to use them yourself.
Book Figure 1-1. Language AI from bag-of-words (~2000) to ChatGPT (2022), coloured by architecture: decoder-only (GPT, GPT-2, GPT-3), encoder-only (BERT, DistilBERT, RoBERTa), encoder-decoder (T5, Switch, Flan-T5) and non-transformer methods.
A recent history of Language AI
Text is unstructured: written as zeros and ones, characters lose their meaning. So the history of Language AI is largely a history of representing language in a structured, numeric form a computer can use for tasks such as classification, clustering, search and generation.
Bag-of-words. First mentioned in the 1950s, popular from the 2000s. Step one is tokenization, splitting sentences into words or subwords (tokens), most simply on whitespace, which fails for languages such as Mandarin that do not space words. Step two builds a vocabulary of every unique word across the sentences. Step three counts how often each vocabulary word appears in each sentence. The result is a vector representation, and models that output such vectors the book calls representation models. The method is old but not obsolete: Chapter 5 uses it to complement modern models.
Bag-of-words in three steps. The vector keeps counts and throws away word order and meaning.
Dense embeddings: word2vec (2013). Bag-of-words treats text as a literal bag and ignores meaning. word2vec (Mikolov et al.) learned embeddings, vector representations that try to capture meaning, by training a neural network on huge text such as all of Wikipedia. A neural network is layers of interconnected nodes; the weights on the connections are its parameters. Each vocabulary word starts with a random vector (say 50 values); at every training step the model takes pairs of words and predicts whether they are likely neighbours in a sentence, adjusting the embeddings toward the truth. Words that tend to share neighbours end up with embeddings close to one another, and that closeness is what "meaning" becomes for a machine.
word2vec's training loop. Words that share neighbours drift together, and closeness becomes meaning.
- Properties, loosely. Picture dimensions as properties: "baby" scores high on "newborn" and "human", "apple" low. The authors stress this is an oversimplification; real dimensions are obscure and seldom map to one human concept, but together they work.
- Similarity. Distance metrics measure how close two words are; compressed to two dimensions, similar words cluster (Chapter 5 shows how to compress).
- Levels. Embeddings exist for words, sentences and whole documents. Bag-of-words is document-level; word2vec is word-level only. Embeddings recur in classification (Ch 4), clustering (Ch 5) and semantic search and RAG (Ch 8).
Context: RNNs and attention. word2vec embeddings are static: "bank" gets one vector whether it means a financial bank or a riverbank. Recurrent neural networks (RNNs) model sequences and were used in pairs: an encoder represents the input sentence, a decoder generates the output, as in translating "I love llamas" into Dutch "Ik hou van lama's". Each step is autoregressive: generating the next word consumes every word generated so far. The encoder squeezes the whole input into one context embedding, which struggles with long sentences. Attention (2014, Bahdanau, Cho and Bengio) let the model focus on the parts of the input relevant to each output and amplify their signal: "lama's" attends strongly to "llamas" and weakly to "I". Instead of one context embedding, the decoder now receives the hidden states of all input words. The cost: the sequential design precludes parallel training.
What attention added to recurrent translation, before the Transformer removed recurrence in 2017.
"Attention Is All You Need" (2017). Vaswani et al. proposed the Transformer, built solely on attention with the recurrence removed, so it trains in parallel, which "tremendously sped up training". Encoder and decoder blocks are stacked, and generation stays autoregressive.
- Encoder block. Self-attention plus a feedforward neural network. Self-attention attends to every position in a single sequence, forward and back, in one go rather than token by token.
- Decoder block. An extra attention layer that attends to the encoder's output, much like the RNN attention decoder.
- Masking. The decoder's self-attention masks future positions, attending only to earlier tokens so it cannot "look into the future" while generating.
- Still to come. Multi-head attention, positional embeddings and layer normalisation are deferred to Chapters 2 and 3.
From counting words to chat models. The Transformer splits into three families; the book's colour code is teal for representation (encoder) models and pink for generative (decoder) models.
Two families of Transformer models
The original Transformer is an encoder-decoder that suits translation but not, say, text classification. In 2018 two simplifications appeared.
BERT (Bidirectional Encoder Representations from Transformers, Devlin et al.) keeps only the encoder; BERT base stacks 12 encoders. Its input carries an extra [CLS] (classification) token whose embedding stands for the whole input and is the usual starting point for fine-tuning. It is trained by masked language modelling: part of the input is hidden and the model predicts it, a hard task that forces accurate intermediate representations. BERT-style models are the workhorse of transfer learning: pretrain on something like all of Wikipedia, then fine-tune for a task such as classification, which needs far less data and compute. Because they emit embeddings at almost every layer, they are also feature-extraction machines with no fine-tuning at all.
GPT (Generative Pre-trained Transformer, Radford et al., 2018), now called GPT-1, stacks decoder blocks and drops the encoder-attention layer. It was trained on 7,000 books and Common Crawl and had 117 million parameters. With "everything else the same", more parameters should mean more capability, and models grew fast: GPT-2 with 1.5 billion, then GPT-3 with 175 billion. These decoder-only models, especially the big ones, are what people usually mean by LLMs.
| Family | Book's name | Trained by | Typical output | Book chapters |
|---|---|---|---|---|
| Encoder-only (BERT) | Representation model (teal) | Masked language modelling | Embeddings; [CLS] for classification | 4, 5, 8, 10, 11 |
| Decoder-only (GPT) | Generative model (pink) | Next-token prediction, then instruction tuning | Text completions, answers | 3, 6, 7, 9, 12 |
| Encoder-decoder (T5) | Not singled out | Sequence-to-sequence objectives | Translated or rewritten text | Ch 1 history |
The authors are careful that the split is about purpose, not mechanics: representation models mainly create embeddings and typically do not generate text; generative models mainly generate text and are typically not trained to produce embeddings.
From completion to instruction. A generative LLM is a sequence-to-sequence machine that autocompletes text. Its "true power shone" once it was fine-tuned to follow instructions, producing instruct or chat models that answer a user's prompt with the most likely response. That is why generative models are often called completion models. Their key limit is the context length (context window), the maximum number of tokens they can process; because generation is autoregressive, the current context grows with every token generated.
The Year of Generative AI and the moving definition
ChatGPT is the product, first powered by GPT-3.5 and later by stronger variants such as GPT-4. In 2023 it had company: open foundation models (base models anyone can fine-tune) and proprietary ones arrived at a steady clip. New architectures also challenged the Transformer: Mamba (Gu and Dao) and RWKV (Peng et al.) aim for Transformer-level quality with longer context or faster inference.
Book Figure 1-28. The Year of Generative AI. Proprietary releases above the line (ChatGPT, GPT-4, Bard, PaLM 2, Claude 2, Grok, Gemini); open models below with sizes (Llama 7B to 65B, MPT, Falcon 40B, Llama 2 70B, Mistral 7B, Qwen, Yi 34B, Mixtral 8x7B, Phi-2 2.7B, DeciLM, Command R 35B).
What counts as "large"? A model with GPT-3's abilities but 10 times smaller, or a GPT-4-sized model that classifies brilliantly but cannot generate, would both fall outside the narrow definition. The book's answer: what we call a model does not change how it behaves, so "large language models" here include models under 1 billion parameters that never generate text and run on consumer hardware.
The training paradigm
Traditional machine learning is one step: train a model for a target task such as classification or regression. LLMs take at least two.
- Language modelling (pretraining). Most of the compute and time. The model learns grammar, context and patterns from a vast internet corpus by predicting the next word. The output is a foundation or base model that generally does not follow instructions. Llama 2 saw 2 trillion tokens.
- Fine-tuning (post-training). Further training of the pretrained model on a narrower task or behaviour, such as classification or instruction following. It saves "massive amounts of resources", since pretraining is out of reach for most organisations (Chapter 12).
- Optional alignment. Extra fine-tuning steps can align the model with user preferences (Chapter 12).
Any model that has been pretrained, fine-tuned ones included, the book calls a pretrained model.
From corpus to useful model. Most compute goes into the first box; most organisations start at the third.
What LLMs are useful for
The authors map five common jobs to techniques and chapters:
- Review sentiment. Supervised classification, with encoder or decoder models, pretrained (Ch 4) or fine-tuned (Ch 11).
- Topics in support tickets. Unsupervised classification: encoder models do the clustering, decoder models label the topics (Ch 5).
- Finding and inspecting documents. Semantic search gives an LLM external information (Ch 8), improved by a custom or fine-tuned embedding model (the book says Ch 12; embedding training is actually Ch 10).
- A chatbot with tools and documents. Prompt engineering (Ch 6), RAG (Ch 8) and fine-tuning (Ch 12) are "all pieces of the LLM puzzle".
- Recipes from a photo of your fridge. A multimodal task: the model reasons over an image (Ch 9).
The chapter's map of use cases to techniques and chapters.
Responsible development
- Bias and fairness. Training data may carry biases the model reproduces and amplifies; since data is seldom shared, you only find them by testing.
- Transparency and accountability. Users may not know they are talking to a machine; with no human in the loop, a medical LLM app might be regulated as a medical device.
- Harmful content. Models can be confidently wrong and can mass-produce fake news.
- Intellectual property. Who owns output that echoes a phrase in the training data? Without the data, nobody can tell when copyrighted material is used.
- Regulation. The EU AI Act regulates foundation models, LLMs included.
The list is sound but brief, and treats these as considerations rather than engineering controls; later chapters do not return to most of them.
Limited resources are all you need
Interfacing with LLMs
| Proprietary (closed) | Open models | |
|---|---|---|
| Examples | OpenAI GPT-4, Anthropic Claude | Cohere Command R, Mistral, Microsoft Phi, Meta Llama |
| Access | Through an API; weights and code secret | Download weights; run locally |
| Strengths | No GPU or hosting expertise needed; usually more performant | Full control: fine-tune, run sensitive data, no API dependence, transparency, Hugging Face community |
| Weaknesses | Paid; cannot fine-tune it yourself; data goes to the provider | Needs strong hardware, more for training; setup know-how |
The authors prefer open models "wherever we can" for the freedom to experiment and inspect. A note flags the argument over what "open source" means: some public models carry licences that forbid commercial use, and training data and code are seldom released.
Frameworks. Rather than survey hundreds of 2023 frameworks, the book builds intuition with backend packages (no GUI) that load and run models efficiently: llama.cpp, LangChain and, at the core of many others, Hugging Face Transformers. For a ChatGPT-like local interface they point to text-generation-webui, KoboldCpp and LM Studio.
Generating your first text
Models live on the Hugging Face Hub (800,000+ models then). The book's main generative model is Phi-3-mini: 3.8 billion parameters, runs in under 8 GB of VRAM (under 6 GB with quantization, Chapters 7 and 12), and is MIT-licensed for unrestricted commercial use. Using an LLM loads two models: the generative model and its tokenizer, which splits text into tokens first. Both load by ID, microsoft/Phi-3-mini-4k-instruct, through AutoModelForCausalLM and AutoTokenizer with device_map="cuda".
return_full_text=False. Return only the model's output, not the prompt.max_new_tokens=500. Cap the length; some models otherwise keep going until the context window is full.do_sample=False. Always pick the most probable next token; sampling parameters come in Chapter 6.
The prompt is a list of dictionaries with a role ("user") and content ("Create a funny joke about chickens.").
2. Tokens and Embeddings
Tokens are how a model sees text and embeddings are how it computes with it. The chapter takes a real prompt apart into token IDs, tours eight trained tokenizers on the same awkward test string, then follows embeddings from the static input matrix to contextualized token embeddings, whole-sentence text embeddings, and finally out of language altogether into a word2vec song recommender.
A chat model writes its answer one token at a time, and tokens are equally how it reads its input: a prompt is split into tokens before the model sees anything. A language model never receives text; it receives a list of integer token IDs, and every capability it has is learned over the numeric embeddings of those IDs.
How the tokenizer prepares the input
Loading Phi-3-mini again, the authors send "Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.<|assistant|>" and cap output at max_new_tokens=20. The model returns "Subject: My Sincere Apologies for the Gardening Mishap / Dear". The input actually passed was input_ids, a tensor beginning 1, 14350, 385, 4876, 27746, 5281 and ending 32001. Decoding each ID shows how the prompt was cut:
- Special start token. ID 1 is <s>, marking the beginning of the text.
- Whole words. "Write", "an", "email" are single tokens.
- Word pieces. "apolog" + "izing", "trag" + "ic", "garden" + "ing", "m" + "ish" + "ap".
- Punctuation. Each mark is its own token.
- Spaces. No space tokens: continuation pieces such as "izing" carry a hidden marker joining them to the previous token, and tokens without it are assumed to follow a space.
A round trip through the tokenizer. The model only ever sees and emits integers.
On the way out the tokenizer works in reverse: the model emitted 3323 ("Sub"), 622 ("ject") and 29901 (":"), and tokenizer.decode([3323, 622]) returns "Subject".
- Method, chosen at model design time. Byte pair encoding (BPE), used by GPT models, or WordPiece, used by BERT. Both optimise an efficient token set for a dataset, by different routes.
- Design parameters. Vocabulary size, which special tokens to include, how to treat capitalization.
- Training data. The same method and parameters trained on English, code or multilingual text yield different vocabularies.
Why two tokenizers split the same sentence differently.
| Scheme | Unit | Strength | Weakness |
|---|---|---|---|
| Word | Whole words | Simple; still used in recommenders | Cannot handle words unseen in training; near-duplicate tokens (apology, apologize, apologetic, apologist) |
| Subword | Words and pieces | Expressive (apolog + -y, -ize, -etic, -ist); builds new words from pieces; the default | Arbitrary splits |
| Character | Letters | Never meets an unknown word | Model must also learn to spell "p-l-a-y"; about 3x less text fits a 1,024-token context |
| Byte | Unicode bytes | "Tokenization-free" (CANINE, ByT5); strong for multilingual text | Long sequences |
A subtlety: GPT-2 and RoBERTa keep bytes in their vocabularies only as a fallback for characters they cannot otherwise represent, which does not make them byte-level tokenizers.
Eight tokenizers, one test string
The test string mixes "English and CAPITALIZATION", an emoji and a Chinese character (鸟, "bird"), Python-like code (show_tokens False None elif == >= else:, two tabs, four spaces) and the arithmetic "12.0*50=600", chosen to probe capitalization, other languages, emoji, code whitespace, digits and special tokens.
| Tokenizer (year) | Method, vocab | What the test reveals |
|---|---|---|
| BERT uncased (2018) | WordPiece, 30,522 | Newlines vanish; all lowercase; "capital ##ization" (## marks a continuation); emoji and Chinese become [UNK] |
| BERT cased (2018) | WordPiece, 28,996 | "CAPITALIZATION" costs 8 tokens (CA ##PI ##TA ##L ##I ##Z ##AT ##ION) |
| GPT-2 (2019) | BPE, 50,257 | Keeps newlines and case (4 tokens); emoji split into byte tokens (8582, 236, 113) that decode back; tabs as token 197, spaces as 220 |
| Flan-T5 (2022) | SentencePiece, 32,100 | No newline or whitespace tokens, so poor for code; emoji and Chinese become <unk>, fully invisible |
| GPT-4 (2023) | BPE, ~100,000 | Four spaces = one token (whitespace runs up to 83); elif one token; "CAPITALIZATION" 2 tokens, "tokens" 1; fill-in-the-middle tokens |
| StarCoder2 (2024) | BPE, 49,152 | 15B code model; each digit its own token (6 0 0); <filename>, <reponame>, <gh_stars> tokens for cross-file context |
| Galactica | BPE, 50,000 | Science model; tokens for citations ([START_REF], [END_REF]), reasoning (<work>), maths, amino acids, DNA; the only one with a single token for two tabs |
| Phi-3 / Llama 2 | BPE, 32,000 | Llama 2 tokenizer plus chat tokens <|user|>, <|assistant|>, <|system|>; digits split |
BERT's utility tokens. [UNK] for unknown input; [SEP] separates two texts for a cross-encoder (reranking, Chapter 8); [PAD] fills unused positions up to the expected length; [CLS] is the classification token (Chapter 4); [MASK] hides tokens in training. Fill-in-the-middle tokens (<|fim_prefix|>, <|fim_middle|>, <|fim_suffix|>) let a model complete text using what comes after the gap as well as before, essential for code completion inside a file.
Design choices recap. Vocabulary sizes of 30K and 50K are common, 100K increasingly so. Special tokens can be added freely for a domain, as Galactica did. Capitalization is a trade-off: name case carries information, but all-caps variants consume vocabulary. And the domain of the training data changes everything: a text tokenizer chops a Python function's indentation into many pieces where a code tokenizer uses one.
Token embeddings
With text reduced to tokens, the next problem is the numeric representation the model computes on. A model trained on enough English becomes able to generate English; trained on Wikipedia, it can produce some facts.
A model holds one embedding vector per vocabulary token, stored as an embedding matrix that ships with the downloaded model. The vectors start random like all other weights, and training gives them useful values.
From ID to meaning. The model's output vectors differ by context, so "bank" can mean river or money.
From text to vectors. Token IDs index a static embedding matrix; the Transformer turns those into context-dependent vectors, which can be pooled into one text embedding for Part II's applications.
Text embeddings for sentences and documents
Many applications need one vector for a whole sentence, paragraph or document. The simplest route averages the token embeddings, but high-quality text embedding models are trained specifically for the task. With sentence-transformers (from Sentence-BERT, Reimers and Gurevych, 2019) and all-mpnet-base-v2, "Best movie ever!" becomes a single vector of 768 values. These vectors drive Part II: categorisation, semantic search and RAG. Chapter 4 covers choosing a model; Chapter 10 covers training one.
Word embeddings beyond LLMs
Before LLMs, word2vec, GloVe and fastText dominated; in language work they have largely been replaced by contextual embeddings, but the idea travels to recommender engines and robotics. With Gensim, glove-wiki-gigaword-50 (66 MB, Wikipedia, 50 dimensions) gives the nearest neighbours of "king": prince 0.82, queen 0.78, "ii" 0.77, emperor 0.77, son, uncle, kingdom, throne, brother and ruler 0.74.
- Sliding window. Over running text, such as Dune's "Thou shalt not make a machine in the likeness of a human mind", a window of two neighbours each side turns each central word into training pairs.
- Skip-gram. Each pair (centre word, neighbour) is a positive example labelled 1.
- Negative sampling. Positives alone let a model cheat by always answering 1, so random words are added as negatives labelled 0. Randomness suffices, an idea from noise-contrastive estimation.
- Embeddings as the trained weights. A vocab_size x embedding_dimensions matrix starts random; a network takes two vectors, predicts "neighbours or not", and each error nudges the vectors.
word2vec as contrastive learning: a related-or-not classifier whose weights become the embeddings.
Embeddings for recommendation systems
Treat each song as a word and each playlist as a sentence, and word2vec learns song embeddings from co-occurrence. The dataset, collected by Shuo Chen (Cornell University), holds playlists from hundreds of US radio stations; playlists with only one song are dropped.
- Training call.
Word2Vec(playlists, vector_size=32, window=20, negative=50, min_count=1, workers=4), about a minute or two to train. - Lookup.
model.wv.most_similar(positive=str(song_id))returns neighbours with similarities near 0.997.
The song recommender. Co-occurrence in playlists stands in for meaning.
3. Looking Inside Large Language Models
This is the book's engine-room chapter. It follows one forward pass through a generative Transformer: tokenizer, a stack of Transformer blocks, and a language-modelling head that scores every token in the vocabulary, run in a loop one token at a time. It then opens a block to show feedforward memory and attention step by step, and closes with the tweaks that separate a 2024 model such as Llama 3 from the 2017 original: grouped-query attention, Flash Attention, pre-normalisation with RMSNorm, SwiGLU and rotary embeddings.
Inputs, outputs and the generation loop
From outside, a Transformer LLM is text in, text out. Inside, it generates one token per forward pass (the inputs flowing once through the network's computation graph). After each pass the output token is appended to the prompt and the longer text goes back in. Software around the network runs this loop until completion. Models that consume their own earlier predictions are autoregressive, which is what separates generation models from representation models such as BERT. With max_new_tokens=50, Phi-3 begins "Solution 1: Subject: My Sincere Apologies for the Gardening Mishap / Dear Sarah, I hope this message finds you well. I am writing to express my deep" and stops mid-sentence at the limit.
One forward pass per token.
The components of the forward pass
Three parts: the tokenizer, a stack of Transformer blocks that does all the processing, and the LM head, which turns the stack's output into a probability score for every token in the vocabulary. The LM head is a single neural-network layer and only one of several possible heads; others include sequence classification and token classification heads. Printing the Phi-3 model shows the layout:
Phi-3-mini's forward pass, as printed in the chapter. One pass yields one token.
- Embedding matrix.
embed_tokens: 32,064 tokens x 3,072 dimensions. - The stack. 32 x Phi3DecoderLayer, each with self-attention (
qkv_proj3,072 to 9,216, three 3,072-wide projections fused; rotary embeddings) and an MLP (gate_up_proj3,072 to 16,384,down_proj8,192 to 3,072, SiLU activation), plus RMSNorm layers. - LM head.
Linear(3072, 32064): one score per vocabulary token.
Decoding. Choosing a token from the distribution is the decoding strategy. Greedy decoding always takes the top token, which is what temperature 0 does (Chapter 6), and it "doesn't tend to lead to the best outputs". Sampling picks each token in proportion to its probability: if "Dear" has 40%, it is chosen 40% of the time. In code, "The capital of France is" passes through model.model and then model.lm_head; the output has shape [1, 6, 32064], and lm_head_output[0,-1].argmax(-1) decodes to "Paris".
Two decoding strategies over the same probability distribution.
Parallel streams, context size and the KV cache
Each input token flows through its own computation stream, which is why Transformers parallelise so well. A 4K context model has at most 4K streams. Each stream starts as an embedding plus positional information and ends as a vector of the same size, the model dimension (3,072 for Phi-3, hence the stack output [1, 6, 3072]). Only the last stream's output feeds the LM head, yet every earlier stream must still be computed, because attention in each block reads their intermediate results.
That redundancy is what the keys and values (KV) cache removes: store the earlier streams' attention vectors, and the next pass computes only the newest stream. Hugging Face enables it by default (use_cache).
The cache test: 100 tokens on a free T4. Reusing earlier keys and values makes generation nearly 5x faster.
Inside the Transformer block
Models stack from six blocks (the original paper) to over a hundred in large LLMs. Each block has two parts: the attention layer, which pulls in information from other tokens and positions, and the feedforward layer, which "houses the majority of the model's processing capacity".
Feedforward: memory and interpolation. Given "The Shawshank", a language model predicts "Redemption" because the feedforward networks across all layers stored that pattern from many mentions in training. But the model is not a database: the same machinery interpolates between data points to generalise to unseen inputs. A chat model such as GPT-4 instead replies with a paragraph about the 1994 film by Frank Darabont, based on Stephen King's novella, because raw language models like GPT-3 are hard to use and are therefore instruction-tuned and preference-tuned.
Attention: context. N-gram models, the pre-neural approach (Jurafsky and Martin's Speech and Language Processing), show how far "previous token" memory gets you: not far. In "The dog chased the squirrel because it", predicting the next word requires knowing what "it" refers to; attention adds information from the relevant context into the representation of "it", guided by training patterns and earlier clues (calling the dog "she" would point "it" at the squirrel).
- Starting point. The layer processes one position (in "Sarah fed the cat because it", the token "it"), given its vector and the previous tokens' vectors.
- Project. Training has produced three projection matrices. Multiplying the inputs by them yields the queries, keys and values matrices; the bottom row of each belongs to the current position.
- Step 1, relevance scoring. Multiply the current query by the keys matrix to score each previous token, then apply softmax so the scores sum to 1.
- Step 2, combining. Multiply each token's value vector by its score and sum: the output for "it" now carries "cat information".
- Multi-head. The mechanism runs in parallel in several attention heads, each with its own projections, after splitting the information; the heads' outputs are combined. More heads let the model attend to several kinds of pattern at once.
One attention head. The output for "it" ends up carrying "cat" information.
scores = softmax(q · KT) ; output = scores · V
o_proj in its own printout). "Multilevel perceptron" should read multilayer perceptron. The claim that the feedforward layers do "the majority of the model's memorization" is a reasonable research hypothesis (the "key-value memories" view of MLPs), not settled fact, as the authors' own "likely" in the figure caption concedes.Recent improvements to the Transformer
Much of the original design survives, but several ideas proved their worth in models such as Llama 2. Most research targets attention, the most computationally expensive part.
Local and sparse attention. Sparse Transformers and Longformer's sliding window let a token attend only to a few previous positions. Used everywhere, quality would "vastly degrade", so GPT-3 alternates full-attention and sparse-attention blocks (for example blocks 1 and 3 full, 2 and 4 sparse). Attention-pattern diagrams also make plain that decoder blocks look only backward, while BERT attends both ways, the "B" in its name.
| Variant | Keys and values | Trade-off | Used by |
|---|---|---|---|
| Multi-head (2017) | Each head has its own Q, K, V | Best quality, largest KV cache | Original Transformer, GPT-2/3 |
| Multi-query | One K and V shared by all heads; only queries differ | Smallest cache; "too punishing" for quality at large scale | "One write-head is all you need" |
| Grouped-query | Several K/V groups, each shared by a set of heads | A little more memory for a large quality gain | Llama 2 and 3 |
Flash Attention. A method and implementation that speeds up training and inference on GPUs by optimising what is loaded and moved between fast on-chip SRAM and high-bandwidth memory (HBM). It computes exact attention, just with less memory traffic (FlashAttention, then FlashAttention-2).
- 2017. Attention, then add and LayerNorm; feedforward with ReLU, then add and norm; absolute positional embeddings added at the start; multi-head attention.
- 2024. Pre-normalisation (norm before attention and feedforward), reported to cut training time; RMSNorm, simpler and cheaper than LayerNorm; SwiGLU instead of ReLU; grouped-query attention; rotary embeddings applied inside attention. Residual connections remain in both.
Positional embeddings (RoPE)
Order is "an indispensable source of information in language". The original Transformer used absolute positions (token 1, token 2...), either static (geometric functions) or learned. These strain at scale. Training contexts are filled by packing several short documents into one window (a 10-word sentence should not waste a 4K context), and telling the model that a document's first token is "position 50" misleads it into assuming prior context that belongs to an unrelated document.
Why RoPE replaced absolute positions in models such as Llama.
Beyond LLMs. Transformer variants keep spreading: the authors point to surveys of Transformer tweaks, of vision Transformers, of time-series Transformers, and to robotics (Open X-Embodiment, RT-X).
4. Text Classification
Part II opens with the most common NLP task: assigning a label to a piece of text. On one dataset, the Rotten Tomatoes movie reviews, the chapter runs five methods with frozen pretrained models, two built on representation models and three on generative ones, and scores each with the same weighted F1. The results run from 0.78 with no labelled data at all to 0.91 for GPT-3.5, and together they make a practical menu of what to try before fine-tuning anything.
The task and the data
Classification trains or uses a model to assign a class to input text. It powers sentiment analysis, intent detection, entity extraction and language detection, and both representation and generative models now do it, by different routes. The chapter keeps every model frozen and only uses its outputs; fine-tuning BERT for classification waits for Chapter 11 and building embedding models for Chapter 10. A tip opens the chapter: always compare against a classic but strong baseline, TF-IDF features with logistic regression.
- Dataset.
rotten_tomatoeson the Hugging Face Hub (Pang and Lee, 2005): 5,331 positive and 5,331 negative reviews, split 8,530 train, 1,066 validation, 1,066 test. Labels are 1 positive, 0 negative, so the task is binary sentiment. - Splits. Train on train, evaluate on test; the validation split is for checking generalisation after hyperparameter tuning on the other two.
- Examples. "the rock is destined to be the 21st century's new 'conan'..." (positive) and "the movie is all portent and no content" (negative).
Representation models: two flavours
A task-specific model is a representation model such as BERT fine-tuned for one task, sentiment for instance. An embedding model is fine-tuned to produce general-purpose embeddings that serve classification, search (Chapter 8) and more. Both start from a foundation model such as BERT.
The two kinds of representation model the chapter uses. Both start from a foundation model such as BERT.
Model selection. The Hub held over 60,000 text-classification models and more than 8,000 embedding models at writing. Choose by language, architecture, size and performance. Encoder-only models "excel in task-specific use cases" and are far smaller than generative ones. BERT's descendants include RoBERTa, DistilBERT, ALBERT and DeBERTa. The suggested baselines: BERT base (uncased), RoBERTa base, DistilBERT base (uncased), DeBERTa base, bert-tiny and ALBERT base v2. For embeddings, start at the MTEB leaderboard but weigh inference speed, which "should not be underestimated in real-life solutions".
Generative models: classification by instruction
Generative models are sequence-to-sequence: tokens in, tokens out, where a task-specific model outputs a class. Trained on many tasks, they do not know your task until told, so you guide them with an instruction (prompt) and refine it iteratively, which is prompt engineering (Chapter 6).
Method 4: Flan-T5. T5, the Text-to-Text Transfer Transformer (Raffel et al.), keeps the original encoder-decoder design with 12 encoders and 12 decoders. Training runs in two steps:
- Pretraining. Masked language modelling that masks token spans rather than single tokens, so each mask can hide several tokens.
- Multi-task fine-tuning. Every task is recast as text-to-text and trained simultaneously. "Scaling instruction-finetuned language models" (Chung et al., 2022) pushed this to more than a thousand tasks, producing the Flan-T5 family (small, base, large, xl, xxl).
Loaded through the text2text-generation pipeline as google/flan-t5-small, the model receives each review prefixed with "Is the following sentence positive or negative?". Its text output is mapped back to numbers ("negative" to 0, anything else to 1). F1 0.84 from the smallest Flan-T5, nearly matching the trained embedding classifier.
Method 5: ChatGPT. The architecture of the original ChatGPT (GPT-3.5) is undisclosed but presumably decoder-only. OpenAI did describe its training: humans wrote desired outputs for prompts (instruction data) to fine-tune a first model, then ranked several outputs of that model from best to worst (preference data) to train the final one. Preference data carries nuance that instruction data lacks: it teaches the difference between a good and a better answer (Chapter 12).
chatgpt_generation helper sends a system message ("You are a helpful assistant.") and a user message built from a template with a [DOCUMENT] slot, at temperature 0, to gpt-3.5-turbo-0125. The template asks whether the document is a positive or negative movie review and to "return 1" or "return 0" and "not give any other answers". Scoring all 1,066 test reviews cost about 3 cents, inside the free tier. Outputs are cast with int(). Result: F1 0.91 (negative recall 0.97, positive precision 0.96), the best in the chapter.Practicalities. Track usage, because external APIs "can quickly become costly". Expect rate-limit errors, and retry with exponential backoff: sleep after each failure, lengthen the sleep each time, stop at success or a maximum retry count.
Exponential backoff, the chapter's answer to rate limits on paid APIs.
| Method | Model | Labels needed | Weighted F1 | Runs on |
|---|---|---|---|---|
| Task-specific | Twitter-RoBERTa (sentiment) | None (someone else's) | 0.80 | GPU helpful |
| Embeddings + classifier | all-mpnet-base-v2 + logistic regression | Yes, 8,530 | 0.85 | Classifier on CPU |
| Zero-shot embeddings | all-mpnet-base-v2 + label text | None | 0.78 | GPU helpful |
| Prompted encoder-decoder | Flan-T5 small | None | 0.84 | GPU |
| Prompted API model | gpt-3.5-turbo-0125 | None | 0.91 | OpenAI servers, ~3 cents |
return_all_scores (use top_k=None) and, in January 2026, deleted the text2text-generation pipeline the Flan-T5 example uses; Hugging Face now points users to a chat model through text-generation. The API model is going away. OpenAI deprecated gpt-3.5-turbo-0125 on 22 April 2026 with shutdown on 23 October 2026, naming gpt-5.6-terra as the substitute. Current practice for LLM labelling also uses structured outputs (an enum schema) rather than "return 1 or 0" and string casting.5. Text Clustering and Topic Modeling
Unsupervised methods get less attention than classification, yet clustering finds structure nobody labelled. The chapter builds the now-standard pipeline of embed, reduce, cluster on 44,949 arXiv abstracts, then turns it into BERTopic, Grootendorst's modular topic model, which describes each cluster with c-TF-IDF keywords and then refines them with "Lego blocks" ranging from embedding rerankers to LLM-written labels. The theme is combination: encoders, decoders and a 1990s bag-of-words working in one pipeline.
Why cluster text
Text clustering groups documents by semantic content, meaning and relationships. It gives fast exploratory analysis and categorisation of large unstructured collections, and supports creative uses: finding outliers, speeding up labelling and catching mislabelled data. Because modern embeddings capture context ("language is more than a bag of words"), clustering has become far more effective. Topic modelling extends it: discover the abstract topics in a corpus and describe each with keywords or keyphrases, ideally with one overarching label.
The data. maartengr/arxiv_nlp: 44,949 abstracts from arXiv's cs.CL (Computation and Language) section, 1991 to 2024, with titles kept for display.
The three-step clustering pipeline
Methods range from graph-based neural networks to centroid-based clustering, but a popular pipeline uses three steps.
- Step 1, embed. Pick a model optimised for semantic similarity with a decent MTEB clustering score and fast inference. The chapter switches from all-mpnet-base-v2 to thenlper/gte-small, newer, better at clustering and smaller. Output: (44949, 384), one 384-value vector per abstract.
- Step 2, reduce. High dimensions hurt clustering because the number of possible values grows exponentially with each dimension, making meaningful clusters hard to find. Dimensionality reduction compresses (it does not just drop dimensions) while trying to keep global structure. Options are PCA (Hotelling, 1933) and UMAP (McInnes, Healy and Melville, 2018); UMAP is chosen because it handles nonlinear structure better.
- Step 3, cluster. k-means needs the number of clusters up front, which nobody knows. A density-based algorithm finds the count itself and does not force every point into a cluster. HDBSCAN (McInnes, Healy and Astels, 2017), the hierarchical version of DBSCAN (Ester et al., KDD 1996), finds dense micro-clusters and marks outliers, useful for an arXiv set full of niche papers.
- UMAP n_components=5. Values of 5 to 10 usually capture high-dimensional global structure.
- UMAP min_dist=0.0. The minimum distance between embedded points; 0 gives tighter clusters.
- UMAP metric='cosine'. Euclidean methods struggle in high dimensions.
- UMAP random_state=42. Reproducible across sessions, but it disables parallelism and slows training.
- HDBSCAN min_cluster_size=50. The smallest allowed cluster; lower it for more clusters. With
metric="euclidean"on the reduced vectors andcluster_selection_method="eom", it yields 156 clusters.
Information is always lost. Reduction never captures the original space perfectly; there is a balance between fewer dimensions and kept information. Visualising in 2D (a second UMAP to two components, plotted with matplotlib, outliers in faint grey) loses more and can push clusters together or apart misleadingly, so human inspection of what each cluster actually contains remains a key part of cluster analysis. Reading three abstracts from cluster 0 (statistical machine translation from English to American Sign Language via the Moses tool and a 3D avatar, sign-language phonology, sign-language animation) shows it is about sign-language translation.
From clusters to topics
Classic methods such as latent Dirichlet allocation (LDA) (Blei, Ng and Jordan, 2003) instead treat each topic as a probability distribution over the vocabulary and describe it with top words ("sign", "language", "translation"), leaving the reader to infer meaning. They rest on bag-of-words features that ignore context and meaning, which Transformer embeddings capture.
Classic and embedding-based topic modelling side by side.
BERTopic: cluster, then represent
BERTopic (Grootendorst, 2022) runs in two largely independent stages. First, the same embed, reduce, cluster pipeline. Second, a bag-of-words topic representation with two fixes:
- Cluster level, not document level. Word counts are summed per cluster (c-TF) using scikit-learn's CountVectorizer, so each cluster gets one frequency vector.
- Down-weight common words. Stop words such as "the" and "I" dominate raw counts. Class-based TF-IDF (c-TF-IDF) multiplies each cluster term frequency by an IDF weight: the log of the average word frequency per cluster divided by the word's total frequency across clusters.
Wx,c = ‖tfx,c‖ × log(1 + A / cfx)
Here tfx,c is the frequency of term x in cluster c, cfx its frequency across all clusters, and A the average number of words per cluster. Each cluster becomes a topic with its own ranking of the vocabulary; the higher a word's weight, the more representative it is.
BERTopic's two independent stages plus optional representation blocks, which rerank or relabel each topic and can be stacked.
Modularity. Because the stages are independent, c-TF-IDF does not care which models did the clustering: swap sentence-transformers for any embedder, or HDBSCAN for k-means if you want no outliers. Grootendorst calls it Lego blocks, and the design lets newly released models slot in. The same base supports a long list of variants.
The Lego-block design: each stage can be swapped without touching the others.
Exploring the arXiv topics
Fitting BERTopic(embedding_model=..., umap_model=..., hdbscan_model=...) on the abstracts and precomputed embeddings and calling get_topic_info() lists each topic's count and a name built from its top four keywords.
find_topics("topic modeling") returns topic 22 at 0.95 similarity (topic, topics, LDA, latent, Dirichlet, allocation), and the BERTopic paper's own abstract lands in topic 22.Topic sizes on 44,949 arXiv abstracts. A third of the corpus went unassigned.
Outliers can be handled with a non-outlier algorithm (k-means) or BERTopic's reduce_outliers(), which reassigns them. get_topic(0) shows weights (speech 0.028, asr 0.019, recognition 0.013), with "the" still sneaking in at 0.006. Visual tools include visualize_documents (an interactive map, hovering shows titles), visualize_barchart, visualize_heatmap(n_clusters=30) and visualize_hierarchy.
Representation blocks: reranking keywords
The bag-of-words representation is fast but blind to semantics. The fix borrows reranking from neural search (Chapter 8): keep c-TF-IDF's cheap first ranking, then refine it with slower, stronger models. These representation models run once per topic, not once per document: a million documents in a hundred topics need only a hundred refinements. Blocks can be stacked. The chapter saves the originals with deepcopy(topic_model.topic_representations_) and compares with a topic_differences helper.
- KeyBERTInspired. Modelled on Grootendorst's KeyBERT (2020), which matches word and document embeddings by cosine similarity. It picks each topic's most representative documents by c-TF-IDF similarity, averages their embeddings, and reranks candidate keywords against that average. Topics read better and stop words nearly vanish, but domain abbreviations disappear: "nmt" drops out of topic 3, a loss for experts who find such terms highly informative.
- Maximal marginal relevance (MMR). Cuts redundancy ("summaries" and "summary" together) by iteratively choosing keywords that relate to the documents yet differ from those already chosen, controlled by a diversity parameter (0.2 here), going from about 30 candidates to 10. Topic 4 becomes "summarization, document, extractive, rouge".
Stacked representation blocks. Expensive models run once per topic, not once per document.
The text-generation block
Instead of asking a generative model to classify millions of documents, BERTopic asks it once per topic to write a label from the topic's keywords and a handful of representative documents.The prompt has two slots:
[DOCUMENTS], typically the
fourdocuments whose c-TF-IDF vectors best match the topic, and
[KEYWORDS], from c-TF-IDF or any earlier block.
| Topic (c-TF-IDF) | KeyBERTInspired | MMR | Flan-T5 small | GPT-3.5 |
|---|---|---|---|---|
| speech, asr, recognition | speech, encoder, phonetic | speech, asr, error | Speech-to-description | Leveraging External Data for Improving Low-Resource Speech Recognition |
| medical, clinical, biomedical | nlp, ehr, clinical | clinical, biomedical, patient | Science/Tech | Improved Representation Learning for Biomedical Terms |
| sentiment, aspect, analysis | aspect, sentiment, aspects | sentiment, analysis, absa | Review | Advancements in Aspect-Based Sentiment Analysis |
| translation, nmt, machine | translation, translating, translate | translation, nmt, bleu | Attention-based neural machine translation | Neural Machine Translation Enhancements |
| summarization, summaries | summarization, summarizers | summarization, document, extractive | Summarization | Document Summarization Techniques |
Book Figure 5-24. The top 20 arXiv cs.CL topics with GPT-3.5 labels, drawn with datamapplot: document summarisation, aspect-based sentiment, NER, multi-agent LLM systems, legal-domain evaluation, gender bias and more, with outliers as grey points.
to_plot and outliers, so the plotting snippet fails as printed. The GPT-3.5 labels are long and generic ("Advancements in...", "Techniques"), which the authors call impressive without testing whether they help a reader more than the keywords do.6. Prompt Engineering
The chapter moves from using generative models to steering them. It covers choosing and loading a model (Phi-3-mini), the chat template hidden behind every call, the sampling knobs temperature, top_p and top_k, the components of a prompt, in-context learning and prompt chaining, the reasoning family of chain-of-thought, self-consistency and tree-of-thought, and finally output verification through examples and grammar-constrained sampling. Its thesis: prompting is an iterative, modular, experimental craft with no perfect design.
Choosing and loading a model
Start by choosing proprietary or open. Proprietary models are "generally more performant", but the book favours open models for flexibility and zero cost. Foundation models usually come in several sizes, and thousands of fine-tunes descend from each. The advice: start small, because "scaling up to larger models tends to be a nicer experience than scaling down". The chapter keeps Phi-3-mini (3.8 billion parameters, runs in 8 GB of VRAM), loaded with AutoModelForCausalLM (microsoft/Phi-3-mini-4k-instruct) and wrapped in a text-generation pipeline with return_full_text=False, max_new_tokens=500 and do_sample=False.
The chat template. The pipeline silently converts a list of messages into the prompt template the model was trained on. apply_chat_template reveals it for Phi-3: <s><|user|> Create a funny joke about chickens.<|end|> <|assistant|>. The special tokens mark who said what and, through <|end|>, when to stop generating. The whole template goes to the LLM in one pass. The joke: "Why don't chickens like to go to the gym? Because they can't crack the egg-sistence of it!"
What the pipeline silently builds from a list of messages for Phi-3.
Controlling output: temperature, top_p, top_k
At each step the model assigns a likelihood to every token: after "I am driving a...", "car" and "truck" are likely and "elephant" is possible but rare. With do_sample=False only the most probable token is chosen; sampling parameters need do_sample=True.
- Temperature. Controls how likely less probable tokens are. 0 always picks the top token and gives the same answer every time; high (0.8) gives diverse output, low (0.2) near-deterministic output. At
temperature=1the joke became a rollercoaster and "chicken-soup", and changes on every run. - top_p (nucleus sampling). Considers tokens until their cumulative probability reaches p: 0.1 keeps only the head of the distribution, 1 keeps all tokens. Lower values mean fewer candidates and less "creative" output.
- top_k. Considers exactly the k most probable tokens (k=100 keeps the top 100).
Two ways to control randomness. Both require do_sample=True.
| Use case | Temperature | top_p | Effect |
|---|---|---|---|
| Brainstorming | High | High | High randomness, large token pool: diverse, unexpected results |
| Email generation | Low | Low | Deterministic, high-probability tokens: predictable, conservative |
| Creative writing | High | Low | Randomness within a small pool: creative but coherent |
| Translation | Low | High | Deterministic yet wider vocabulary: coherent with linguistic variety |
The ingredients of a prompt
An LLM is a prediction machine: a few words ("The sky is") already get a completion, but not a task. Prompt engineering designs inputs to elicit useful responses, and it doubles as a tool for evaluating models and building safeguards. "There is not and unlikely will ever be a perfect prompt design."
- Instruction plus data. The basic task prompt: "Classify the text into negative or positive" plus the text.
- Output indicators. Prefix "Text:" and end with "Sentiment:" so the model answers with a label instead of a sentence; models generalise to such structures from the many instructions they saw in training.
- Instruction-based prompting. Use cases include supervised classification, search, summarisation, code generation, named-entity recognition and more, each needing its own question.
- Specificity. Say exactly what you want: not "Write a description for a product" but "in less than two sentences and use a formal tone". The authors rank this most important; without "two to three sentences" you may get whole paragraphs.
- Hallucination guard. LLMs state wrong facts confidently. Ask the model to answer only if it knows and otherwise say "I don't know".
- Order. Put the instruction at the start or end. In long prompts the middle is often forgotten ("Lost in the middle", Liu et al., 2023): models favour the beginning (primacy effect) and end (recency effect).
The chapter's three general-purpose fixes. Order matters because long prompts lose their middle.
Advanced prompts: components, examples, chains
The modular prompt. A complex prompt is built from parts you add, remove and reorder, testing each change. The chapter's summarisation prompt concatenates seven strings.
| Component | Purpose | Chapter's example |
|---|---|---|
| Persona | Role to adopt | Expert in LLMs who breaks complex papers into digestible summaries |
| Instruction | The task, as specific as possible | Summarise the key findings of the paper |
| Context | Why the task exists | Help researchers quickly grasp the main points |
| Format | Output shape; without it automated systems break | Bullet summary of the method, then a results paragraph |
| Audience | Target and level (ELI5 for teaching) | Busy researchers following LLM trends |
| Tone | Voice | Professional and clear |
| Data | The material itself | "Text to summarize: ..." |
Creative extras such as emotional stimuli ("This is very important for my career", the EmotionPrompt paper) also work. Prompting is partly reverse-engineering what a model learned, so a prompt that works on one model may not transfer to another trained differently.
In-context learning. Instead of describing the task, show it. Zero-shot uses no examples, one-shot one, few-shot two or more ("an example is worth a thousand words"). From the GPT-3 paper (Brown et al., 2020): a user turn defines "Gigamuru" (a Japanese musical instrument) with an assistant turn giving a sample sentence, then asks for a sentence with "screeg" (to swing a sword at something). Phi-3 answers "the knight skillfully screeged his opponent's shield". Roles matter; without them "it would seem as if we were talking to ourselves". Sampling can still lead the model to ignore the examples.
In-context learning: show the task instead of describing it, as with "Gigamuru" and "screeg".
Chain prompting. Split a problem between prompts, feeding each output into the next: product features to name, then name plus features to slogan, then all three to sales pitch. The demo produces "MindMeld Messenger", "Unleashing Intelligent Conversations, One Response at a Time", then a pitch. Each call can get its own parameters (few tokens for a name, many for a pitch). Uses: response validation (have the LLM check earlier output), parallel prompts merged in a final pass (several recipes into one shopping list), and story writing (summary, characters and beats before dialogue).
A prompt chain. Each call gets its own parameters, such as few tokens for a name.
Reasoning with generative models
The authors stress that LLM output only resembles reasoning; at writing it was "generally considered" to come from memorisation and pattern matching. Prompting can still mimic reasoning. Borrowing Kahneman's Thinking, Fast and Slow (2011): System 1 is automatic and intuitive, like plain token generation; System 2 is slow, conscious and logical. The reasoning techniques try to give the model a System 2: room to lay out intermediate thoughts, compare alternatives and check itself before committing to an answer.
The chapter's reasoning ladder on the cafeteria problem (23 apples, 20 used, 6 bought). Each step spends more tokens or calls to buy more deliberate output.
- Chain-of-thought (CoT). Wei et al. (2022): show a worked example with its reasoning ("Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11."). Phi-3 then explains "23 - 20 = 3... 3 + 6 = 9" before answering 9. Reasoning tokens let the model spread compute over the problem; "each additional token... allows the LLM to stabilize its output".
- Zero-shot CoT. Kojima et al. (2022): no examples, just append "Let's think step-by-step"; Phi-3 produces Step 1 to Step 3 and the same 9. Variants such as "Take a deep breath and think step-by-step" (Yang et al., LLMs as optimizers) also work.
- Self-consistency. Wang et al. (2022): ask the same prompt several times with sampling (varying temperature and top_p) and take the majority answer, voting only on answers when combined with CoT. Cost: n times slower for n samples.
- Tree-of-thought (ToT). Yao et al. (2023): at each step generate several candidate thoughts, rate them, keep the best and prune the rest. Strong where many paths matter (stories, ideas) but needs many model calls. A zero-shot ToT prompt fakes it in one call: "Imagine three different experts... write down 1 step... If any expert realizes they're wrong... they leave". The experts reach 9 and "all agree".
Output verification
Production systems must check output to avoid breaking the application. Four reasons:
- Structured output. Models default to free text; many uses need JSON.
- Valid output. Asked to choose one of two options, the model must not invent a third.
- Ethics. Some open models have no guardrails; outputs may need to be free of profanity, PII, bias and stereotypes.
- Accuracy. Output must be factually correct, coherent and free of hallucination.
Four reasons production systems check what a model returns.
There are three ways to control output: examples, grammar (control token selection) and fine-tuning (Chapter 12).
n_gpu_layers=-1, n_ctx=2048) with response_format={"type": "json_object"} at temperature 0 returns a full warrior (level 10, strength 18, Ironclad Armor, Steel Greatsword) that json.loads parses cleanly.Grammar and constrained sampling. Few-shot cannot forbid output. Libraries such as Guidance, Guardrails and LMQL go further: some use an LLM to validate output against predefined rules, some generate the known parts of the format themselves and let the model fill only the unknown slots, and some constrain token sampling directly, so a sentiment classifier can only emit "positive", "neutral" or "negative". Sampling parameters still apply inside those limits. Free GPU memory before loading a second model (del model, tokenizer, pipe, gc.collect(), torch.cuda.empty_cache()).
do_sample=False. JSON mode guarantees valid JSON, not a schema: the warrior's fields were whatever the model chose. The Figure 6-13 caption repeats Figure 6-11's ("a complex prompt with many components") instead of describing zero-, one- and few-shot prompts. The reasoning section's own caveat (pattern matching, not true reasoning) predates reasoning models trained with reinforcement learning, which changed the debate.7. Advanced Text Generation Techniques and Tools
How far can you push an LLM without fine-tuning it? The chapter answers with four building blocks, all shown in LangChain: model I/O (loading a quantized GGUF model), chains (prompt templates and multi-step pipelines), memory (three ways to make a stateless model remember) and agents (an LLM that picks its own tools through the ReAct loop). Each block helps on its own; the authors' point is that the combination is "truly where LLMs shine".
The framework and its four blocks
LangChain is "one of the earlier frameworks" that wrap LLMs in useful abstractions; DSPy and Haystack are named as newer alternatives. The chapter covers four of its modules, leaving retrieval to Chapter 8.
- Model I/O. Loading and working with LLMs.
- Memory. Helping LLMs remember.
- Agents. Combining complex behaviour with external tools.
- Chains. Connecting methods and modules.
The four LangChain modules the chapter covers; retrieval waits for Chapter 8.
Model I/O: quantized models
The chapter loads Phi-3 as a GGUF file, a format for quantized models. Quantization reduces the number of bits used to store each parameter while keeping most of the information. Fewer bits mean a narrower range of values: pi stored in float32 keeps more digits than in float16. The result is faster inference and less VRAM, usually at little accuracy cost.
The clock analogy. Asked the time, you say "14:16", not "14:16 and 12 seconds": the seconds are dropped without losing what matters. Quantization does the same to weights. Details come in Chapter 12 and Grootendorst's "A Visual Guide to Quantization".
Quantization drops the "seconds" of each weight. Below 4 bits a smaller model is usually better.
The model is fetched with wget from microsoft/Phi-3-mini-4k-instruct-gguf and loaded through llama-cpp-python as LlamaCpp(model_path=..., n_gpu_layers=-1, max_tokens=500, n_ctx=2048, seed=42). The first llm.invoke("Hi! My name is Maarten. What is 1 + 1?") returns an empty string, because Phi-3 needs its prompt template and LangChain does not add it. Any model works with the chapter's code (Phi-3, ChatGPT via ChatOpenAI, Llama 3); the Open LLM Leaderboard helps choose.
Chains: templates and sequences
A chain connects an LLM to another component (a prompt, a tool, memory), and chains can be linked to each other. LangChain is named after them.
A single link: the prompt template. Phi-3's template has four parts: <s> (prompt start), <|user|> (user turn), <|assistant|> (model turn) and <|end|> (end of either). A PromptTemplate with an {input_prompt} variable, piped as basic_chain = prompt | llm, now answers "The answer to 1 + 1 is 2...". The template is written once; only the variables change. OpenAI's API applies its template itself. Templates can also hold task variables, as in "Create a funny name for a business that sells {product}."
basic_chain = prompt | llm. Without the template, Phi-3 returned an empty string.
Multiple prompts: the sequential chain. Breaking a complex prompt into a sequence of small prompts, each feeding its output to the next, gives each step a focused task and leaves every intermediate result available on its own.
The chapter's story pipeline, title | character | story: one short user input, three LLM calls, and a dictionary holding every intermediate output.
Each link is an LLMChain with an output_key. Given "a girl that lost her mother", the title chain alone returns "Whispers of Loss: A Journey Through Grief". The full chain, run separately (sampling is on, so outputs vary), returns "In Loving Memory: A Journey Through Grief", a two-sentence portrait of Emily, "a resilient young girl", and a one-paragraph story. The user supplied one short prompt; the title is retrievable on its own, which a single big prompt would not guarantee.
Memory: making a stateless model remember
Out of the box, LLMs are stateless. Told "My name is Maarten", Phi-3 answers the sum and then, asked "What is my name?", apologises that it cannot know personal information. Memory has to be added to the chain.
- Conversation buffer. Paste the entire history into the prompt. The template gains a
{chat_history}slot fed byConversationBufferMemory(memory_key="chat_history"), stored as "Human:" and "AI:" turns. Asked again, the model replies "Your name is Maarten." - Windowed buffer. History grows until it exceeds the token limit, so
ConversationBufferWindowMemory(k=2)keeps only the last k exchanges. After "My name is Maarten and I am 33 years old", "What is 3 + 3?" and "What is my name?", the first exchange drops out, and "What is my age?" gets "I'm unable to determine your age". - Conversation summary.
ConversationSummaryMemoryuses an LLM to summarise the history after each turn with a prompt ("Summarize the conversations and update with the new lines. Current summary: ... new lines of conversation: ... New summary:"). Each user question now costs two calls, and the summariser can be a smaller, faster model. Asked "What was the first question I asked?", the model infers "what's 1 + 1?" from the summary, because the exact wording was never stored.
Three ways to make a stateless model remember. The window forgot Maarten's age; the summary costs an extra LLM call per turn.
| Memory type | Pros | Cons |
|---|---|---|
| Conversation buffer | Easiest to implement; no information loss within the context window | Slower generation as tokens pile up; needs large-context LLMs; large histories make retrieval hard |
| Windowed buffer | Large context not needed unless history is big; no loss over the last k turns | Only the last k turns survive; no compression of them |
| Conversation summary | Captures the full history; enables long conversations; fewer tokens | Extra call every turn; quality depends on the summariser |
The trade-off is speed, memory and accuracy: the buffer is instant but "hogs tokens"; the summary is slow but frees them. A bigger context window is no free fix, since those tokens must still be processed before generation starts.
Agents: an LLM that chooses its actions
Chains follow steps the developer defined. Agents "leverage a language model to determine which actions they should take and in what order". On top of model I/O, chains and memory they add two components:
Chains versus agents. The chapter's MacBook agent answered well, but nobody checked it.
- Tools. Things the agent can use to do what the LLM cannot, such as a calculator for arithmetic LLMs are "notoriously bad at", a search engine or a weather API.
- Agent type. The planner that decides which action or tool to use next, able to create and self-correct a roadmap toward a goal.
ReAct (Yao et al., 2022) combines reasoning and acting, so that reasoning shapes actions and observations shape further reasoning. Because an LLM can only emit text, "acting" means generating a query in a format that the framework turns into a tool call.
- Thought. The LLM states what it should do next and why.
- Action. Based on the thought, it names a tool and an input (search, calculator).
- Observation. The tool's result comes back and the LLM summarises what it learned.
- Repeat. The Thought/Action/Action Input/Observation block "can repeat N times" until "Thought: I now know the final answer" and a Final Answer.
The ReAct loop. Acting means emitting text that the framework turns into a tool call.
Book Figure 7-16. Two ReAct cycles for a US MacBook Pro price in euros: cycle 1 searches the web and observes $1,299; cycle 2 calls a calculator for 1,299 x 0.85 = EUR 1,104.15.
ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0). The prompt is the ReAct template with {tools}, {tool_names}, {input} and an {agent_scratchpad} for the running trace. Tools: DuckDuckGo search wrapped as a Tool named "duckduck", plus load_tools(["llm-math"]) for a calculator. create_react_agent builds the agent and AgentExecutor(..., handle_parsing_errors=True) runs it. Asked for the current price in USD and the cost at 0.85 EUR per USD, it answers $2,249.00 and about 1,911.65 EUR. The verbose trace of intermediate steps is the main debugging tool.The double edge. The answer is impressive for two tools, but nobody is in the loop to judge whether it is correct. The authors suggest designing for reliability: have the agent return the source URL for the price, or ask the user to confirm at each step. They also expect smaller local models to handle such agents eventually; larger open models already can, at the cost of more VRAM.
LLMChain, ConversationBufferMemory and the other legacy classes out of the main package into langchain-classic. Agents are now built with create_agent, which runs on LangGraph, instead of AgentExecutor and create_react_agent, and conversation state lives in LangGraph checkpointers or message-history wrappers rather than memory classes. Text-parsed ReAct has largely given way to native tool calling: models return structured tool calls with JSON arguments, so "Action:" lines and handle_parsing_errors are rarely needed, though the thought-act-observe loop is unchanged. Tools are increasingly exposed through the Model Context Protocol (MCP). The chapter's gpt-3.5-turbo is scheduled to shut down on 23 October 2026 (Chapter 4 update). And small models did catch up, as the authors predicted: open models of about 4B parameters, such as Qwen3-4B, are trained for tool calling and can run simple agents locally.8. Semantic Search and Retrieval-Augmented Generation
Search was one of the first places language models earned their keep in industry, and generation needed search to stop making things up. The chapter builds both sides: dense retrieval (find texts whose embeddings sit near the query), reranking (rescore a shortlist with a cross-encoder) and RAG (hand the retrieved text to an LLM for a grounded, cited answer). It closes with how to score a search system with mean average precision and how to judge RAG output along four axes.
Why search came first
Months after the 2018 BERT paper, Google announced it was using BERT in Search and called it "one of the biggest leaps forward in the history of Search". Microsoft Bing said large transformer models had delivered "the largest quality improvements to our Bing customers in the past year". The capability they added is semantic search: searching by meaning, not by keyword overlap. In parallel, users began asking chat models factual questions and got fluent, confident answers that were often wrong or stale, the hallucination problem. Retrieval is the leading fix for hallucination: find the relevant information first, then give it to the model so it generates from evidence rather than from memory.
- Dense retrieval. Embed the query and the archive, then return the nearest neighbours of the query in embedding space.
- Reranking. Take a query plus a shortlist from an earlier stage and reorder it by a relevance score.
- RAG. Text generation systems that incorporate search to reduce hallucinations, increase factuality or ground the model on a specific dataset. Generative search (an answer at the end of a search pipeline) is a subset.
The chapter's three families. Production systems usually chain all three.
Dense retrieval
Embeddings are points in space, and close points mean similar text. A search system embeds the query into the same space as the archive and returns the nearest documents. Two design questions follow. Should a distant third result be returned at all? That is the designer's call, and a maximum-distance threshold filters out results when the corpus holds nothing relevant. Is a question really "similar" to its answer? Not always, which is why retrieval models are trained on question-answer pairs (Chapter 10). Long documents are first chunked, each chunk embedded, and the vectors stored in a vector database.
co.embed(..., input_type="search_document"), giving an array of shape (15, 4096). A FAISS IndexFlatL2 holds the vectors; the search() function embeds the query with input_type="search_query" and calls index.search. For "how precise was the science" the top hit is "It has also received praise from many astronomers for its scientific accuracy..." (distance 10,757), then the Kip Thorne sentence (11,566) and the Double Negative effects sentence (11,922). The best answer shares no keyword with the query. A BM25 baseline (rank_bm25.BM25Okapi, lowercased tokens, punctuation and English stop words stripped) ranks "Interstellar is a 2014 epic science fiction film..." first (1.789), Kip Thorne second (1.373) and the cast list third (0.000): it matched "science" and missed the answer.The dense-retrieval demo. The top answer shares no keyword with the query, which BM25 missed.
Caveats. Dense retrieval has four weak spots:
- It always returns something. "What is the mass of the moon?" still yields the box-office, astronomers and cinematography sentences (distances 1.30 to 1.33). Fixes: a distance threshold, or show results and let users judge, logging clicks and satisfaction to improve later versions.
- Exact phrases. A user hunting for a specific string is better served by keyword matching, which is why hybrid search (semantic plus keyword) is advised over dense retrieval alone.
- Domain shift. A model trained on web and Wikipedia text works worse on, say, legal documents absent from its training data.
- Answers that span sentences. The demo had one fact per sentence; real questions often need several, which raises the chunking question.
Where embedding search fails, with the fix the chapter suggests for each.
Chunking long texts
Transformers have limited context, so long documents cannot be embedded whole. There are two families of approach.
- One vector per document. Either embed a representative part (the title or the opening), which suits a quick demo or Wikipedia-style pages but leaves most text unsearchable, or embed chunks and average them into one vector, which yields a highly compressed vector that loses much of the document.
- Multiple vectors per document. Chunk the document and index chunk embeddings. This gives full coverage and vectors that capture individual concepts, so the index is more expressive. It is the better approach.
Book Figure 8-8. "Llama 2 was trained on 40% more data than Llama 1" split three ways: every 15 characters (words broken, no overlap), every 5 tokens, and every 5 tokens with one overlapping token so neighbouring chunks share context.
The right granularity depends on the texts and queries expected:
- Each sentence a chunk. Often too granular: vectors miss context.
- Each paragraph a chunk. Good for short paragraphs; otherwise every 3 to 8 sentences.
- Add context to chunks. Prepend the document title, or include some text before and after so chunks overlap (Figure 8-10).
Chunk granularity, from too little context to chunks that carry their own.
The authors expect new strategies, including LLMs that split text dynamically into meaningful chunks.
Search infrastructure. For thousands or tens of thousands of vectors, computing every distance with NumPy is fine. At millions, use approximate nearest neighbour libraries such as Annoy or FAISS, which answer in milliseconds and can use GPUs and clusters. Vector databases such as Weaviate and Pinecone add and delete vectors without rebuilding the index and support filtering beyond raw distance.
Fine-tuning for retrieval. Retrieval needs good text embeddings, trained on queries and relevant results. For the sentence "Interstellar premiered on October 26, 2014, in Los Angeles", the positive queries are "Interstellar release date" and "When did Interstellar premier"; a negative is "Interstellar cast". Before tuning, all three may sit equally close to the document (they all mention Interstellar). Training pulls the relevant queries closer and pushes the irrelevant one away, encoding the dataset's own notion of relevance.
Reranking
For organisations that already run search, the easiest upgrade is a language model as the last step, reordering results by relevance. This one step "can vastly improve search results", and it is what Bing added with BERT-like models.
The chapter's full pipeline. Keyword and dense retrieval form a cheap hybrid first stage; a cross-encoder rescores only the shortlist; RAG appends a generation step that cites the retrieved documents.
The Rerank endpoint. co.rerank(query=query, documents=texts, top_n=3, return_documents=True) needs no training. For "how precise was the science" over all 15 sentences it scores the astronomers sentence 0.1698, the box-office sentence 0.0700 and Kip Thorne 0.0044: far more confident about the right answer. Real indexes hold thousands or millions of entries, so a first stage shortlists perhaps 100 or 1,000 results for the reranker. That first stage can be keyword search, dense retrieval or, better, hybrid.
BM25 plus rerank. The chapter's keyword_and_reranking_search takes the top 10 BM25 hits and reranks them to 3. BM25 alone gives scores to just two sentences that share a keyword; after reranking, Kip Thorne rises to first (0.004), followed by the wormhole plot sentence and the Nolan screenplay sentence. On the multilingual MIRACL benchmark a reranker lifts nDCG@10 from 36.5 to 62.8. For local setups, the Sentence Transformers library's "Retrieve & Re-Rank" docs show both steps.
How rerankers work. The query and each candidate are fed together to a cross-encoder, so the model sees both texts before scoring. Documents run as a batch but each is judged independently against the query. This is the monoBERT formulation from "Multi-stage document ranking with BERT", and it reduces search to classification: output a score from 0 (irrelevant) to 1 (highly relevant). "Pretrained Transformers for Text Ranking: BERT and Beyond" surveys the field to about 2021.
Why search runs in two stages: recall with vectors, precision with a cross-encoder.
Retrieval evaluation: mean average precision
IR evaluation needs three components: a text archive, a set of queries and relevance judgments saying which documents are relevant to each query. Two systems answer the same query with three results each. If system 1 returns two relevant documents and system 2 one, system 1 wins on a simple count. Counting fails when both return one relevant result but in different positions: the system with the relevant document in position 1 is better, and the metric must say by how much. Average precision scores that case 1 for system 1 and 0.3 for system 2.
- Precision@k. Relevant results in the top k divided by k. A lone relevant document at position 1 gives 1/1 = 1.0; at position 3 it gives 1/3.
- Average precision (AP). Compute precision@k only at positions holding relevant documents, then average over the query's relevant documents. Nonrelevant positions add nothing directly but push later relevant ones down.
- Mean average precision (MAP). The mean of AP across all queries in the test suite: one number for comparing systems.
- nDCG. Normalized discounted cumulative gain allows graded relevance, so one document can be labelled more relevant than another.
Three steps from one ranked list to a whole test set. Position matters: a hit at rank 3 scores 1/3.
P@k = relevant in top k / k AP = (1/R) Σ P@k · rel(k) MAP = mean of AP over queries
Book Figure 8-22. A query with two relevant documents, returned at positions 1 and 3: precision@1 = 1, precision@3 = 2/3 = 0.67, so AP = (1 + 0.67) / 2 = 0.8. Position 2 (irrelevant) is skipped but costs the third result.
Why "mean average"? Probably because MAP "sounds better than average average precision". The standard reference is the "Evaluation in Information Retrieval" chapter of Introduction to Information Retrieval by Manning, Raghavan and Schütze.
Retrieval-augmented generation
RAG comes from "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Lewis et al., 2020). A search step is followed by grounded generation: the LLM is prompted with the question plus the retrieved information. RAG reduces hallucination, improves factuality and enables "chat with my data" over internal company data or a single source such as a book. Search engines adopted it too: Perplexity, Microsoft Bing AI and Google Gemini summarise results or answer directly.
search(); the top results go to co.chat(message=query, documents=docs_dict). The answer: "The film generated a worldwide gross of over $677 million, or $773 million with subsequent re-releases." The response carries span-level citations: "worldwide gross" (characters 21 to 36), "over $677 million" and "$773 million with subsequent re-releases." each point to doc_0. Local stack. Phi-3-mini (the FP16 GGUF) runs through llama-cpp-python in LangChain (n_gpu_layers=-1, max_tokens=500, n_ctx=2048, seed=42); a small embedding model feeds FAISS.from_texts; a Phi-3 template has a {context} slot ("Relevant information: ... Provide a concise answer...") and a {question} slot; RetrievalQA.from_chain_type with chain_type='stuff' and retriever=db.as_retriever() wires them together. The local answer gets the $677 million and $773 million right but pads them with speculation that the move to digital projectors "might have contributed" to the gross, and it has no span citations.The prompt template is where retrieved documents meet the model, so it is the main lever for answer length and tone.
Advanced RAG techniques
| Technique | Problem it solves | Example from the chapter |
|---|---|---|
| Query rewriting | Verbose chat messages or references to earlier turns confuse retrieval | A rambling essay message about penguins and dolphins becomes "Where do dolphins live"; Cohere's co.chat has a rewriting mode |
| Multi-query RAG | One question needs several searches | "Compare the financial results of Nvidia in 2020 vs. 2023" becomes two queries, one per year; the rewriter may also decide no search is needed |
| Multi-hop RAG | Later queries depend on earlier answers | "Largest car manufacturers in 2023, do they make EVs?" first finds Toyota, Volkswagen, Hyundai, then runs one EV query for each |
| Query routing | Knowledge sits in several systems | HR questions go to the HR system (Notion), customer questions to the CRM (Salesforce) |
| Agentic RAG | Data sources become tools the LLM chooses among | If the agent can search Notion it should also be able to post to Notion |
The drift toward agents. Each enhancement hands the LLM more responsibility: judging what information it needs and which sources to use. At writing time only the largest managed models could attempt this; Cohere's Command R+ "excels" at it and is available as open weights.
RAG evaluation
Evaluation methods are still developing. "Evaluating Verifiability in Generative Search Engines" (Liu, Zhang and Liang, 2023) ran human evaluations of generative search systems along four axes.
- Fluency. Is the generated text fluent and cohesive?
- Perceived utility. Is the answer helpful and informative?
- Citation recall. The share of generated statements about the world that are fully supported by their citations.
- Citation precision. The share of generated citations that actually support their statements.
- Faithfulness (Ragas). Is the answer consistent with the provided context?
- Answer relevance (Ragas). How relevant is the answer to the question?
Four human axes from Liu, Zhang and Liang (2023) plus two Ragas metrics, scored by people or an LLM judge.
Human evaluation is preferred, but LLM-as-a-judge automates it: a capable LLM scores each generation along each axis. Ragas is a library that does exactly this and documents its metric formulas.
cohere.ClientV2 and /v2/chat require a model and a messages list, and documents go in as strings or objects with data, so the chapter's co.chat(message=..., documents=...) is the legacy v1 form. Command R+ (command-r-plus-04-2024) was deprecated on 15 September 2025; Cohere recommends Command A (command-a-03-2025, 256K context), and Command A+ (command-a-plus-05-2026) is its first mixture-of-experts model. Embed v4 (15 April 2025) embeds text, images and mixed PDFs with a 128K context and Matryoshka sizes of 256 to 1,536 dimensions, replacing the 4,096-dimension vectors shown. Rerank 4 (11 December 2025, rerank-v4.0-pro and -fast) has a 32K window, 100+ languages and JSON input. On retrieval quality, Anthropic's contextual retrieval (an LLM prepends chunk-specific context before embedding and BM25 indexing) cut top-20 retrieval failures by 49% (5.7% to 2.9%), and by 67% with reranking, a concrete form of the authors' "add context to chunks". Ragas reached v0.4.3 (January 2026) with a collections API and metrics such as context precision, which is the chapter's AP idea applied to retrieved chunks. Practice has shifted toward agentic retrieval, where the agent decides when and what to search, as a KDD 2026 tutorial reviewing about 2,000 RAG papers (2020 to 2026) describes; hybrid search plus reranking remains the default first stage.9. Multimodal Large Language Models
Language does not live in a vacuum: people communicate with faces, tone and pictures as well as words, and a model that can look at an image is useful in more places. The chapter takes three steps. The Vision Transformer turns images into token-like patch embeddings; CLIP trains text and image encoders into one shared embedding space; BLIP-2 bolts a frozen image encoder onto a frozen LLM through a small trainable bridge, the Q-Former, so a text model can caption and chat about pictures.
What multimodal means
Each type of data (text, images, audio, video, sensor readings) is a modality, and a model that handles more than one is multimodal. A model can accept a modality as input without being able to generate it. LLMs have shown emergent abilities as they grow (generalisation, reasoning, arithmetic; Wei et al., 2022), and the authors expect multimodal input to unlock capabilities that text alone keeps locked.
Transformers for vision
The obstacle is tokenisation: an image has no words. ViT's answer is to cut the image into a grid of patches and treat each patch as a token.
Why no vocabulary. Text tokens recur, so each gets an ID. Image patches are rarely found in other images, so instead of an ID each patch is linearly embedded. Once the patch embeddings reach the encoder they are treated exactly like text tokens, and from that point there is no difference in how text or images are trained. The book's figures use 3 x 3 patches for illustration; the original used 16 x 16, hence the paper's title. That interchangeability is why ViT is the usual route to making language models multimodal, starting with embedding models.
ViT's tokenisation of images. Patches replace words, and a learned linear layer replaces the vocabulary lookup, because image patches almost never repeat exactly across images.
Multimodal embedding models and CLIP
A multimodal embedding model puts text and images in the same vector space, so a text embedding can be compared with an image embedding. Search "pictures of a puppy" and find puppy photos, or ask which documents best match an image. The best known and most used is CLIP (Contrastive Language-Image Pre-training, Radford et al., 2021).
- Zero-shot classification. Compare an image's embedding with embeddings of class descriptions; the closest wins, with no training.
- Clustering. Cluster images together with keywords to see which keywords belong to which image groups.
- Search. Across billions of texts or images, find what relates to a text or image query.
- Generation. Multimodal embeddings drive image generation, as in Stable Diffusion (Rombach et al., 2022).
What a CLIP-style embedding space makes possible.
How CLIP is trained. Start with millions of images with captions. A text encoder embeds each caption and an image encoder embeds each image. The pair is compared with cosine similarity (the dot product divided by the product of the vector lengths). At first similarity is low; training maximises it for matching pairs and minimises it for mismatched ones, then repeats with new batches. This is contrastive learning (Chapter 10). Negative examples are essential, because modelling similarity means knowing what makes things different as well as alike.
CLIP's contrastive loop. Matching pairs are pulled together and mismatched pairs pushed apart.
openai/clip-vit-base-patch32, loaded as three parts: a tokenizer (CLIPTokenizerFast), a processor that resizes images (CLIPProcessor) and the model (CLIPModel). The caption "a puppy playing in the snow" becomes IDs 49406 ... 49407, that is <|startoftext|>, six word tokens ending in </w>, and <|endoftext|>; there is no [CLS] token, because in CLIP that slot represents the image embedding. get_text_features returns a 512-value vector. The 512 x 512 AI-generated puppy image is shrunk to 224 x 224 (tensor [1, 3, 224, 224]) and get_image_features also returns 512 values, the same shape, which is what makes comparison possible. After normalising both, the dot product is 0.33, a number that means little on its own.Book Figure 9-14. Three images against three captions. Each correct pair scores highest in its row (puppy 0.33, pixelated cat 0.35, supercar at sunset 0.31); mismatches score 0.08 to 0.19. CLIP similarities are only meaningful relative to each other.
The short route. sentence-transformers wraps CLIP as SentenceTransformer("clip-ViT-B-32"): model.encode(images), model.encode(captions), then util.cos_sim for the matrix. The open-source reimplementation, OpenCLIP, offers the same workflow.
Making text generation models multimodal: BLIP-2
Text generators such as Llama 2 and ChatGPT reason well over text but are confined to it. The goal is a model that can look at a pizza and list its ingredients, or see the Eiffel Tower and say when it was built. Training such a model from scratch needs billions of images, texts and image-text pairs. BLIP-2 (Bootstrapping Language-Image Pre-training 2, Li et al., 2023) avoids that by connecting a pretrained image encoder and a pretrained LLM through a bridge, the Querying Transformer (Q-Former). Only the bridge is trained; the vision model and the LLM stay frozen, so BLIP-2 reuses what already exists instead of rebuilding it.
The bridge's anatomy. The Q-Former mimics both sides with two modules that share attention layers: an Image Transformer that interacts with the frozen ViT to extract features, and a Text Transformer that interacts with the LLM. It is trained in two stages.
BLIP-2's pipeline. Only the Q-Former and projection train; both big models stay frozen.
- Stage 1, vision-and-language representation learning. Image-caption pairs go in: images through the frozen ViT into the Q-Former's image side, captions into its text side. Three objectives are optimised jointly: image-text contrastive learning (align pair embeddings to maximise mutual information), image-text matching (classify a pair as matched or unmatched) and image-grounded text generation (generate text from what the image shows). The effect is to inject textual information into the frozen ViT's embeddings.
- Stage 2, vision-to-language generative learning. The Q-Former's learnable embeddings, now in the same dimensional space as text, pass through a fully connected linear projection to the shape the LLM expects. They act as soft visual prompts that condition the frozen LLM, much like context in an ordinary prompt.
The two stages that teach the Q-Former to speak the LLM's language.
Book Figure 9-20. The full BLIP-2 procedure: a pretrained, frozen ViT feeds the trainable Q-Former (stage 1); its learnable embeddings are projected into soft visual prompts for a pretrained, frozen LLM (stage 2).
The family that followed. LLaVA (Liu et al., "Visual instruction tuning") and Idefics 2 (Laurençon et al., 2024, built on Mistral 7B) use different architectures but the same idea: connect a CLIP-like visual encoder to a textual LLM and project visual features into language embeddings.
Preprocessing multimodal inputs
The chapter loads Salesforce/blip2-opt-2.7b in float16 with AutoProcessor and Blip2ForConditionalGeneration; model.vision_model and model.language_model reveal the ViT and LLM inside. The processor plays the role a tokenizer plays for text models, for both modalities.
- Images. A 520 x 492 supercar picture comes out as
[1, 3, 224, 224]. Every image becomes a 224 x 224 square, so very wide or tall images get distorted. - Text. The tokenizer is a GPT2TokenizerFast (vocabulary 50,265, with
</s>as BOS, EOS and UNK). "Her vocalization was remarkably melodic" becomes</s>, Her, Ġvocal, ization, Ġwas, Ġremarkably, Ġmel, odic. The Ġ is a space: an internal function shifts certain code points up by 256 to make them printable, so space (32) becomes Ġ (288). It marks the start of a word.
One processor, two modalities: what BLIP-2's processor does to each input.
Two use cases
| Use case | What the chapter got | Lesson |
|---|---|---|
Image captioning (image only, max_new_tokens=20) | Supercar: "an orange supercar driving on the road at sunset". Rorschach blot: "a black and white ink drawing of a bat" | Fits product descriptions or labelling 1,000+ wedding photos; may fail on domain-specific images such as particular cartoon characters |
| Visual question answering (image plus "Question: ... Answer:") | "Write down what you see": "A sports car driving on the road at sunset" | Without a prompt the model only captions |
| Chat-based prompting (image plus prior Q and A plus new question) | "What would it cost me to drive that car?": "$1,000,000" | Memory is the transcript re-sent each turn |
The interactive chat box, built with ipywidgets in Jupyter, keeps a memory list of (question, answer) pairs, rebuilds the prompt as "Question: {} Answer: {}." for each, appends the new question, generates up to 100 tokens and cuts anything after the next "Question". It is the conversation buffer from Chapter 7, applied to images.
openai/clip-vit-base-patch32 through Hugging Face transformers, which is not OpenCLIP. BLIP-2's "$1,000,000" for the cost of driving the car is called "highly specific", but it is an unsupported, hallucinated figure, and the chapter that just taught RAG as the cure for hallucination lets it pass. The "very wide" supercar image is 520 x 492, almost square, so the distortion warning is not shown by the example. Every demo image is AI-generated or iconic (puppy, pixel cat, supercar, a famous inkblot), and no captioning or VQA accuracy is measured. The Rorschach aside ("what do you think it says about the model?") treats a pattern-matcher as having a personality. Missing topics: OCR and document images, charts, audio and video, and the cost of image tokens.10. Creating Text Embedding Models
Part III turns from using models to training them, and starts with the component under almost everything earlier in the book: the text embedding model. The chapter explains contrastive learning, shows why SBERT's bi-encoder replaced the cross-encoder for embeddings, then runs six experiments in sentence-transformers on MNLI data, scored on STSB. The lesson of the numbers: the loss function and the starting model move quality more than anything else, and Augmented SBERT and TSDAE rescue you when labels are scarce.
What an embedding model is for
An embedding model converts text (documents, sentences, phrases) into vectors, and its job is to represent the text as accurately as possible. "Accurate" usually means capturing semantics: documents about similar things get similar vectors, unrelated documents get distant ones (in many dimensions, not the two drawn in figures). But the target is a choice. A sentiment model should put negative reviews together and away from positive ones whatever their topic. An embedding model learns whatever notion of similarity its training examples show it, so the examples you choose decide what "similar" means.
Contrastive learning
Contrastive learning trains a model so that similar documents move closer and dissimilar ones move apart, by feeding it pairs that are similar and pairs that are not. A document's meaning is learned best against another document.
The bank robber. A reporter asks a robber, "Why did you rob a bank?" He answers, "Because that is where the money is" (from Garfinkel, Forms of Explanation, 1982). The answer is correct and misses the point, because the question meant "why rob at all". This is contrastive explanation (Tim Miller): we understand "Why P?" best as "Why P and not Q?", here "Why rob a bank instead of obeying the law?" The same holds for models. Teaching "dog" by listing features (tail, nose, four legs) fails because a cat has them too; asking "why is this a dog and not a cat?" teaches the features that distinguish the concept and the ones that do not matter.
word2vec was contrastive. One of the earliest NLP examples: neighbouring words form positive pairs and randomly sampled words form negatives. It is "not widely known" as such, but it was one of the first major neural NLP breakthroughs to use the idea.
SBERT: from cross-encoder to bi-encoder
<SEP> token, and a classification head outputs a similarity score. Accurate, but it produces no embedding, and finding the most similar pair among 10,000 sentences needs n(n-1)/2 = 49,995,000 inferences. The cheap fix, averaging BERT's output layer or taking its [CLS] token, performs worse than averaged GloVe vectors. Bi-encoder (SBERT). Reimers and Gurevych (2019) drop the classification head and apply mean pooling to the output layer, giving a fixed-size sentence embedding. Each sentence is encoded separately, so embeddings are computed once and compared cheaply. Cross-encoders still score better; bi-encoders are what you can search with.Book Figure 10-7. SBERT's Siamese network: two BERTs with tied weights pool token embeddings into sentence embeddings u and v; the original training fed (u, v, |u - v|) to a softmax classifier.
Siamese training. SBERT trains a Siamese architecture: two identical BERT models sharing weights and architecture (in practice one model fed each sentence in turn). In the original setup the two embeddings are concatenated with their difference and optimised through a softmax classifier. The bi-encoder still learns contrastively, from the (dis)similarity of pairs. Contrastive learning therefore needs two things: data of similar and dissimilar pairs, and a definition of how similarity is measured and optimised.
SBERT's Siamese setup. Training is contrastive, but the output is an embedding per sentence.
Training data and the first model
NLI as free contrast. Embedding models are often pretrained on natural language inference (NLI) data, where a hypothesis either entails, contradicts or is neutral to a premise. "He is in the cinema watching Coco" vs "He is watching Frozen at home" is a contradiction; vs "In the movie theater he is watching the Disney movie Coco" is an entailment. Entailments become positives, contradictions negatives.
Why NLI data is a free source of contrastive pairs for embedding training.
The chapter uses MNLI (Multi-Genre NLI) from the GLUE benchmark (nine language-understanding tasks): 392,702 labelled pairs, of which a 50,000-pair subset keeps training short. Labels are 0 = entailment, 1 = neutral, 2 = contradiction; an example entailment is "One of our number will carry out your instructions minutely" / "A member of my team will execute your orders with immense precision". Smaller datasets make training less stable, so prefer more data if it is good data.
- Base model.
SentenceTransformer('bert-base-uncased'), trained "from scratch" as an embedder; microsoft/mpnet-base often does well. All layers are trainable by default, and freezing is generally not advised. - Loss. First
losses.SoftmaxLoss(..., num_labels=3), the original SBERT loss, kept for illustration. - Evaluator.
EmbeddingSimilarityEvaluatoron the STSB validation set: human-labelled pairs scored 1 to 5, divided by 5, compared by cosine. - Arguments.
SentenceTransformerTrainingArguments: 1 epoch (raise it in practice), batch size 32, 100 warmup steps (learning rate rises linearly from zero), fp16 mixed precision to save memory, eval and logging every 100 steps. - Trainer.
SentenceTransformerTrainer(model, args, train_dataset, loss, evaluator).train().
The softmax-loss model scores pearson_cosine 0.59, the chapter's baseline. Restart the notebook between runs to clear VRAM.
The Massive Text Embedding Benchmark (Muennighoff et al., 2022) spans 8 tasks, 58 datasets and 112 languages, with a public leaderboard, and it records evaluation time because embedding work such as search needs low latency. Run on Banking77Classification, the model gets accuracy 0.493 in 31.8 seconds. A full MTEB run takes hours, so the chapter sticks to STSB.
Loss functions
Softmax loss is generally not advised; more performant losses exist. Two "typically used" ones perform well across the board. Others exist too: MarginMSE, for instance, works well for training cross-encoders.
Cosine similarity loss. For pairs labelled with a degree of similarity between 0 and 1, as in semantic textual similarity tasks. It computes the cosine between the two embeddings and compares it with the label. MNLI is converted with mapping = {2: 0, 1: 0, 0: 1}: entailment = 1, neutral and contradiction = 0. Result: 0.72, against 0.59 for softmax, which shows "the impact the loss function can have".
Multiple negatives ranking (MNR) loss. Also called InfoNCE or NTXentLoss (Henderson et al., 2017; van den Oord et al., 2018; Chen et al., 2020). It takes positive pairs (question/answer, image/caption, title/abstract) or triplets with an added unrelated negative. Negatives come from mixing pairs within the batch: a title with another paper's abstract. These in-batch negatives turn training into classification: cosine similarities feed a cross-entropy loss that must pick the true partner.
MNR: for anchor ai, loss = -log [ esim(ai, pi) / Σj esim(ai, pj) ] bigger batch → more negatives → harder task
For MNR the chapter keeps only entailment rows (50,000 shrink to 16,875), uses the premise as anchor, the hypothesis as positive and a shuffled hypothesis as a soft negative. Result: 0.81. Larger batches help MNR because the right partner must be found among more candidates.
The MNR run. Every other pair in the batch acts as a negative, so bigger batches make the task harder.
Book Figure 10-11. Easy (unrelated: waiting for the bus), semi-hard (on topic: the capital of the Netherlands) and hard (same entities and numbers, wrong answer) negatives.
| Negative type | How to get it | Caveat |
|---|---|---|
| Easy | Randomly sample documents, as in the MNR run | Often trivially unrelated |
| Semi-hard | Embed everything with a pretrained model, take highly similar sentences | Finds similar sentences, not wrong answers, so rarely truly hard |
| Hard | Manual labelling (often starting from semi-hard candidates) or a generative model that judges or writes pairs | The expensive part of building data |
Fine-tuning an existing embedding model
Training from scratch is costly, and sentence-transformers can fine-tune nearly any embedding model.
Supervised. Repeat the MNR run but start from all-MiniLM-L6-v2, small, fast and good across many uses. Result: 0.85, the best so far, with a caveat: MiniLM was already trained on the full MNLI set, so the point is the procedure, not the gain. For domain adaptation, first run masked language modelling on BERT with domain text, then train the embedder on top (Chapter 11). The main difficulty is the data: positive pairs are easy to collect, but adding good hard negatives sharply raises the cost of building a high-quality training set.
Augmented SBERT (Thakur et al., 2020). Large embedders are trained on more than a billion sentence pairs; a real project may have a few thousand labels. Augmented SBERT uses the slow, accurate cross-encoder to label many more pairs for the fast bi-encoder.
- Step 1. Fine-tune a cross-encoder (BERT) on a small, fully annotated gold dataset. Here: 10,000 MNLI pairs, entailment = 1, else 0,
CrossEncoder("bert-base-uncased", num_labels=2), one epoch. - Step 2. Create new sentence pairs. Here: the next 40,000 MNLI pairs, treated as unlabelled. Without extra pairs, recombine premises and hypotheses from the gold set (about 10 times as many pairs), but prefer top-k semantic search with a pretrained embedder over random pairing, which yields mostly dissimilar pairs.
- Step 3. Label them with the cross-encoder (
predict(..., apply_softmax=True), then argmax) to form a silver dataset: fully annotated but not ground truth. - Step 4. Train a bi-encoder (SBERT) on gold plus silver (duplicates dropped) with cosine similarity loss.
Augmented SBERT: the slow, accurate model labels data for the fast one.
Result: 0.71, against 0.72 for the cosine-loss run on all 50,000 gold pairs, from "only 20%" of the labels. To measure what the silver data adds, also train on gold alone and compare.
Unsupervised learning: TSDAE
When there are no labels, options include SimCSE, Contrastive Tension (CT), TSDAE and Generative Pseudo-Labeling (GPL). The chapter picks TSDAE (Transformer-based Sequential Denoising Auto-Encoder; Wang, Reimers and Gurevych, 2021) for its unsupervised and domain-adaptation results.
The idea. Delete a share of the words in a sentence, encode the damaged sentence and pool it into an embedding, then have a decoder reconstruct the original from that embedding alone. The better the embedding, the better the reconstruction. It resembles masked language modelling, but it rebuilds the whole sentence and the model does not know where the gaps are. After training, only the encoder is kept.
TSDAE: a sentence embedding is good if a decoder can rebuild the sentence from it alone.
DenoisingAutoEncoderDataset (which needs NLTK's punkt) damages them, so "Grim faces and hardened jaws are not people-friendly." becomes "Grim jaws are." Pooling uses the [CLS] token, not the mean, because the TSDAE paper found mean pooling loses position information. DenoisingAutoEncoderLoss(embedding_model, tie_encoder_decoder=True) ties the encoder's embedding layer to the decoder's output layer, and batch size drops to 16 because this loss uses more memory. Result: 0.70, "quite impressive" for training with no labels at all.Domain adaptation. Unsupervised methods trail supervised ones and struggle with domain-specific concepts. Domain adaptation moves a model from its source (in-domain) data to a target (out-domain) with different words and subjects. Adaptive pretraining runs TSDAE or masked language modelling on the target corpus first, then adaptive fine-tuning on labelled data, ideally from the target domain, though out-domain data also works after the unsupervised step. The chapter's tools combine: TSDAE on your domain, then supervised training or Augmented SBERT.
dataset[2] should be train_dataset[2]; and the MTEB snippet scores an undefined model. Mapping neutral to 0 treats "unrelated" and "contradicts" as equally dissimilar, a choice the text never questions.SparseEncoder for SPLADE-style sparse models and encode_query / encode_document) and was at v6.1.0 on 18 September 2026; the trainer API used in this chapter carries over. Leading embedders are now LLM-based: Qwen3-Embedding used Qwen3-32B to synthesise about 150M weakly supervised pairs and kept about 12M high-quality ones (cosine above 0.7) for supervised training, with InfoNCE plus in-batch negatives and masking of false negatives. That is the chapter's recipe (contrastive loss, hard negatives, generated labels like Augmented SBERT's silver set) at industrial scale. A BERT-base embedder trained for one epoch on 50,000 pairs is far from today's leaders (about 76 average on English MTEB v2, Chapter 2 update), so the bar for "train your own" is now fine-tuning a strong open embedder on domain pairs rather than starting from BERT.11. Fine-Tuning Representation Models for Classification
Chapter 4 classified text with frozen pretrained models; this chapter lets the weights move. It fine-tunes BERT end to end and measures what freezing layers costs, gets nearly the same F1 from 32 labelled reviews with SetFit, adapts BERT to movie language with continued masked-language-model pretraining, and moves from documents to tokens with named-entity recognition. All of it is nongenerative; generative fine-tuning is Chapter 12.
Supervised classification: fine-tune the whole network
In Chapter 4 both the task-specific model and the embedding model were frozen; only a separate classifier on top learned. Here the pretrained BERT and a classification head (a small feedforward network) train jointly as one architecture, so the backward pass runs from the head through BERT and each part learns from the other.
AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=2); the label count sizes the head. Text is tokenised with truncation=True, and a DataCollatorWithPadding pads each batch to its longest sequence (collators build batches and can also augment data). A compute_metrics function returns F1, which helps detect overfitting during training. TrainingArguments: learning rate 2e-5, batch size 16, 1 epoch, weight decay 0.01. Result: F1 0.85 (eval loss 0.366), against 0.80 for the pretrained task-specific model in Chapter 4, for "a couple of minutes" of training.Full fine-tuning: head and encoder learn together, so each adapts to the other.
Freezing layers
BERT's parameters print as embeddings, then 12 encoder blocks (numbered 0 to 11, each with attention, dense layers and layer norm), then the pooler and finally classifier.weight and classifier.bias. Any of them can be frozen with param.requires_grad = False. Freezing saves compute; the rule is that frozen layers should be followed by trainable ones.
| What trains | How | F1 |
|---|---|---|
| Everything | Default fine-tuning | 0.85 |
| Encoder block 11, pooler and head | Freeze parameters with index below 165, where block 11 starts | 0.80 |
| Classification head only | Freeze every name not starting with "classifier" | 0.63 (loss 0.682), but much faster |
F1 by how much of BERT is allowed to train. Freezing the whole encoder is the costly choice.
Book Figure 11-7. F1 against the number of trainable encoder blocks: about 0.71 with none, a steep climb to about 0.84 at five blocks (red line), then a plateau near 0.85 with all twelve.
Over
multiple epochsthe time and resource gap between freezing and not freezing
widens, so the balance is worth tuning.
Few-shot classification with SetFit
Few-shot classification learns target labels from only a few labelled examples per class, which suits tasks where labels are scarce but a few high-quality ones can be written. SetFit (Tunstall et al., 2022) builds on sentence-transformers and, with a handful of examples, is competitive with fine-tuning BERT on the full labelled set.
SetFit's three steps. The trick is step 1: a few class labels imply many similar and dissimilar pairs, which is exactly the data contrastive learning (Chapter 10) needs.
Step 1, the pair arithmetic. A class with 16 sentences yields 16 x 15 / 2 = 120 positive pairs; pairs across classes give negatives. In the book's illustration, sentences about programming languages pair with each other and against sentences about pets. Step 2 distils the class structure into the embeddings by contrastive fine-tuning. Step 3 embeds every sentence with the tuned model and trains any scikit-learn model or a classification head on top.
The run. sample_dataset(tomatoes["train"], num_samples=16) keeps 32 reviews out of about 8,500. The default head is logistic regression; use_differentiable_head=True with head_params={"out_features": num_classes} swaps in a trainable one. Result: F1 0.836. SetFit also does zero-shot classification by generating synthetic examples from label names, such as "This example is happy" and "This example is sad".
Continued pretraining with masked language modelling
General pretraining may not know domain-specific words. Inserting a middle step, continued pretraining with masked language modelling (MLM) on your own domain text, updates subword representations toward the new vocabulary. The progression is general BERT, then BioBERT for medicine, then a fine-tuned BioBERT that classifies medication; a company can do the same for its own lingo. Continued pretraining has been shown to improve classification (Sun et al., 2019).
Continued pretraining inserts a middle step so the model learns domain words before the task.
- Token masking. Randomly mask 15% of tokens, so parts of a word may be masked. Converges faster.
DataCollatorForLanguageModeling(tokenizer, mlm=True, mlm_probability=0.15). - Whole-word masking. Mask entire words. Harder to predict, so the model learns more precise representations, but it takes longer to converge. Swap in
DataCollatorForWholeWordMask.
mlm. Asked to fill "What a horrible [MASK]!", the original model offers idea, dream, thing, day, thought. The adapted model offers movie, film, mess, comedy, story: it has absorbed the domain. Fine-tuning then starts from AutoModelForSequenceClassification.from_pretrained("mlm", num_labels=2).Named-entity recognition
NER classifies individual tokens rather than whole documents, finding people, organisations and locations; it is especially useful for de-identification and anonymisation of sensitive data. The architecture is the same BERT plus head, but predictions are made per token, with no pooling.
The data. CoNLL-2003 (English) has about 14,000 training samples and five entity classes: person, organisation, location, miscellaneous and none. Example 848 is "Dean Palmer hit his 30th homer for the Rangers." with ner_tags [1, 2, 0, 0, 0, 0, 0, 0, 3, 0]. Labels use the BIO scheme: O = 0, then B and I versions of PER, ORG, LOC and MISC (1 to 8). B marks the beginning of an entity and I a continuation, so B-PER then I-PER means "Dean Palmer" is one person, not two. Other datasets the authors liked: wnut_17 (emerging, rare entities), tner/mit_movie_trivia (actor, plot, soundtrack) and tner/mit_restaurant (amenity, dish, cuisine).
Aligning labels with subword tokens. Labels are per word, but the tokenizer splits "homer" into home and ##r, and "Maarten" into Ma, ##arte, ##n. Giving every piece B-PER would mean three people. The rule: the first subtoken keeps the word's label, later subtokens get the I- version, and special tokens get -100, which the loss ignores.
Label alignment for NER. Without it one person would be counted as three.
Book Figure 11-21. Label alignment: "Maarten" (B-PER) becomes Ma (B-PER), ##arte (I-PER) and ##n (I-PER).
The align_labels function walks word_ids(): a new word takes its own label; a special token takes -100; a continuation turns an odd (B-) label into the next even (I-) one with if updated_label % 2 == 1: updated_label += 1. Example 848 becomes [-100, 1, 2, 0, 0, 0, 0, 0, 0, 0, 3, 0, -100]. Evaluation uses seqeval through the evaluate package, skipping -100 positions and reporting overall F1; training uses DataCollatorForTokenClassification and the same arguments as before. The saved model, run as a token-classification pipeline on "My name is Maarten.", tags Ma as B-PER (0.995) and ##arte and ##n as I-PER (0.993, 0.995).
num_train_epochs=10 while the text says 20 epochs. Figure 11-7 shows about 0.71 with no trainable blocks, but the text measured 0.63; and its axis trains blocks "0 to 4", the bottom of the network, while the code freezes the bottom and trains the top, so the figure and the method disagree about which layers matter. The NER compute_metrics wraps each token in its own list, so seqeval scores isolated tokens rather than entity spans, and the NER F1 is never printed. "16 sentences about sports" appears in an example about programming and pets. The SetFit comparison also mixes models: all-mpnet-base-v2 (already trained on a billion pairs) against bert-base-cased. The cited Sun et al. paper is misprinted as "How to fine-tune GERT".load_metric; use evaluate.load("f1"), as the NER section already does. Transformers v5 no longer accepts tokenizer= in Trainer; pass processing_class=tokenizer. SetFit's evaluation_strategy is now eval_strategy. SetFit v1.2.0 (4 September 2026) supports Transformers v5 and sentence-transformers v6. For the base model, ModernBERT (December 2024, 8,192-token context) and successors such as Ettin (Chapter 4 update) are drop-in replacements for BERT in exactly these pipelines. For NER without training data, GLiNER models extract arbitrary entity types named at inference time, and GLiNER2 (v2.1.0, 9 October 2026) extends this to schema-driven classification, structured extraction and relations. Fine-tuned encoders remain the cheap, fast choice for high-volume classification and NER, where calling an LLM per record costs far more.12. Fine-Tuning Generation Models
The last chapter turns a base generative model into a chat model and then aligns it. It explains the three training steps of a modern LLM, the parameter-efficient methods that make fine-tuning affordable (adapters, LoRA, QLoRA), runs instruction tuning of TinyLlama on a free GPU, surveys how generative models are evaluated, and closes with preference tuning: reward models with PPO, and the simpler DPO that the authors use in practice.
The three LLM training steps
The three steps of Figure 12-3. Steps 2 and 3 are both fine-tuning; only the data and the objective differ.
Step 1, language modelling. The model is pretrained on massive text to predict the next token, a self-supervised method that needs no labels. The result is a base model (also called pretrained or foundation model): a key artefact, but awkward for end users. Asked a question, it completes it rather than answering, and may even write new questions (Figures 12-4 and 12-5). Step 2, supervised fine-tuning (SFT). The base model is trained on instruction data, still with next-token prediction, but now conditioned on a user input, so it learns to follow instructions; SFT can also target tasks like classification. Step 3, preference tuning. A further fine-tune aligns outputs with human preferences or AI safety, as defined by the data supplied, and distils preference into the model.
Full fine-tuning
Full fine-tuning updates all parameters, like pretraining, but with a smaller, labelled dataset (Figure 12-6). Any labelled data works, which also makes it a good way to learn domain-specific representations. For instruction following the data is question-response pairs spanning many tasks (Figure 12-7): the model reads the instruction and applies next-token prediction to the response, so it answers instead of generating more questions. The costs are the problem: full fine-tuning is expensive, slow and storage-hungry, which is what parameter-efficient fine-tuning (PEFT) addresses.
Adapters
Adapters insert small modular components inside each Transformer block, trained while the rest stays frozen. Houlsby et al. (2019) placed them after the attention layer and after the feedforward network and showed that tuning 3.6% of BERT's parameters comes within 0.4% of full fine-tuning on GLUE. One block is not enough, so the components run through every block; the set across all blocks is "an adapter", and adapters for different tasks (say medical classification and NER) can be swapped into the same base model, provided it has the same architecture and weights. AdapterHub (Pfeiffer et al., 2020) became a central repository, mostly for BERT-style models; LLaMA-Adapter (Zhang et al., 2023) carried the idea to generative Transformers.
Houlsby adapters inside one Transformer block. Tuning about 3.6% of BERT came within 0.4% on GLUE.
Low-rank adaptation (LoRA)
LoRA (Hu et al., 2021) trains only a small set of parameters, but instead of adding layers it approximates the change to large weight matrices with two small matrices, kept separately from the base LLM and combined with the frozen weights at the end.
LoRA's arithmetic (Figures 12-12 to 12-14). The update W + BA is low rank, so a rank-8 adapter for one GPT-3 matrix is about 0.13% of its size.
Performance does drop with the compression, and the book is candid about it. The trade-off works because language models "have a very low intrinsic dimension" (Aghajanyan, Zettlemoyer and Gupta, 2020): small ranks approximate even huge matrices. GPT-3 (175B) has 12,288 x 12,288 matrices (150 million parameters each) in each of its 96 blocks; at rank 8 the two factors hold about 197K. LoRA is also selective: it can target, for instance, only the query and value matrices of each layer.
QLoRA: compress the base before adapting it
Weights are numbers at a given precision (float64, float32); fewer bits mean less memory but less accuracy, as pi in float32 vs float16 shows (Figure 12-15). Naive mapping to fewer bits collapses nearby weights onto the same value, erasing the differences between them (Figure 12-16). QLoRA (Dettmers et al., 2023) fixes this in two moves:
- Blockwise quantization. Weights are quantized in blocks, each with its own mapping, so similar weights can still be told apart at low precision.
- Distribution-aware bins. Neural-network weights are roughly normally distributed between -1 and 1, so bins are set by relative density: narrow where weights crowd, wider in the tails. This also reduces outlier problems.
QLoRA's path from a 16-bit model to a trainable 4-bit one.
Book Figure 12-18. Distribution-aware quantization: dense regions near zero get narrow bins, so similar weights land in different bins; tail blocks can be wide because outliers are rare.
Together these go from 16-bit floats to a 4-bit normalised float (NF4) with only a small loss, cutting training memory sharply; quantization also shrinks models for inference, needing less VRAM. QLoRA adds double quantization and paged optimizers, which the book leaves to the paper.
Instruction tuning TinyLlama with QLoRA
- 1. Template the data. Base model TinyLlama-1.1B-intermediate-step-1431k-3T. Data: 3,000 conversations sampled (seed 42) from HuggingFaceH4/ultrachat_200k (
test_sft), a filtered UltraChat of almost 200k user-LLM conversations.format_promptapplies the chat template of TinyLlama-1.1B-Chat-v1.0, which marks turns with<|user|>,<|assistant|>and</s>(Figure 12-19). Sample 2576 is a knock-knock joke ("Hike up your pants, it's cold outside!"). - 2. Quantize (the Q).
BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype="float16", bnb_4bit_use_double_quant=True). Loading takes ~1 GB VRAM instead of ~4 GB; training needs more. Pad token<PAD>, left padding. - 3. Configure LoRA.
LoraConfig(lora_alpha=32, lora_dropout=0.1, r=64, bias="none", task_type="CAUSAL_LM")targeting all seven projections (q, k, v, o, gate, up, down); thenprepare_model_for_kbit_trainingandget_peft_model. - 5. Train.
SFTTrainer(..., dataset_text_field="text", max_seq_length=512, peft_config=peft_config); up to an hour on a free Colab T4. Save the adapter toTinyLlama-1.1B-qlora. - 6. Merge. Reload in 16 bits with
AutoPeftModelForCausalLMand callmerge_and_unload(). Asked "Tell me something about Large Language Models", the merged model now answers instead of continuing the prompt.
| Knob | What it does | Rule of thumb |
|---|---|---|
| r | Rank of the two small matrices; higher means less compression, more capacity | 4 to 64 |
| lora_alpha | Scales how much change is added to the original weights | About 2 x r |
| target_modules | Which layers get adapters; fewer is faster but weaker | Trade speed for quality |
| num_train_epochs | Training rounds; more tends to degrade results | Keep low |
| learning_rate | Step size; QLoRA found higher rates suit models over 33B | Experiment |
| optim | Paged optimizers from the QLoRA paper | paged_adamw_32bit |
Two notes widen the recipe. Dropping quantization_config and peft_config turns it into full instruction tuning. And QLoRA can also tune an existing chat model, for example to emit SQL or JSON in a fixed format, given query-response data. For tuning tips the authors point to Sebastian Raschka's Ahead of AI newsletter; their honest advice is that hyperparameters have no set guidelines.
Evaluating generative models
Generative models serve so many tasks that no single metric judges them, and skill at maths says nothing about coding. Because outputs are probabilistic, production use needs robust evaluation; the authors stress the lack of golden standards.
| Family | What it measures | Limits |
|---|---|---|
| Word-level metrics | Perplexity, ROUGE, BLEU, BERTScore; perplexity asks how probable the model finds each next token ("When a measure becomes a" then "target") | Ignore consistency, fluency, creativity and correctness |
| Benchmarks | MMLU (57 tasks), GLUE, TruthfulQA, GSM8k, HellaSwag (4 choices), HumanEval (164 coding problems) | Overfitting, broad not specific, hours of GPU time |
| Leaderboards | Open LLM Leaderboard: six benchmarks incl. HellaSwag, MMLU, TruthfulQA, GSM8k | Public data invites overfitting to the board |
| LLM-as-a-judge | A separate LLM grades quality; pairwise variant picks the better of two answers | Improves as LLMs improve |
| Human evaluation | Chatbot Arena: two anonymous models, user votes, 800,000+ votes, Elo-style ranking | Crowd taste may not match your use case |
From cheapest proxy to the gold standard. Each level is a proxy for human preference.
The authors' conclusion is to evaluate against the intended use case (HumanEval rather than GSM8k for coding) and that "you are the best evaluator": try models on your own questions. Alammar asks new models questions in Arabic and Grootendorst in Dutch. They end the section with Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." A model optimised only for grammatical sentences could learn to output "This is a sentence." forever.
Preference tuning, alignment and RLHF
An instruction model answers, but how it answers still varies: for "What is an LLM?" we may prefer an explanation of the internals over "It is a large language model". The basic loop: a preference evaluator (human or otherwise) scores a generation, say 4, and the model is updated to encourage high-scoring and discourage low-scoring outputs (Figures 12-21 to 12-23). Thousands of examples are needed, so the evaluator is automated with a reward model.
Two ways to turn preference pairs into an aligned model (Figures 12-30 and 12-32). DPO folds the reward model into the LLM itself.
The reward model. Take a copy of the instruction-tuned model and replace its language-modelling head with a quality classification head, so it outputs one score for a prompt plus generation. It is trained on preference pairs: a prompt with an accepted and a rejected generation (both can be good, one just better). Pairs come from asking the LLM for two generations and having human labelers pick one. The objective: score(accepted) > score(rejected). Llama 2 trained two reward models, one for helpfulness and one for safety.
Building a reward model. Llama 2 trained two: helpfulness and safety.
- Collect preference data. Prompts with accepted and rejected generations, labelled by people.
- Train a reward model. It learns to rank accepted above rejected.
- Fine-tune the LLM with it. The reward model acts as the preference evaluator; the usual algorithm is Proximal Policy Optimization (PPO, Schulman et al., 2017), which keeps the LLM from deviating too far while chasing reward. PPO trained the original ChatGPT (November 2022).
DPO. PPO is complex and trains at least two models. Direct Preference Optimization (Rafailov et al., 2023) drops the reward model and the RL loop: a frozen copy of the LLM serves as the reference, and training pushes the trainable model to raise the probability of accepted generations and lower that of rejected ones relative to the reference. Scores are token-level log probabilities, aggregated per generation, for both models (Figure 12-33). The authors found DPO more stable and more accurate than PPO.
Two routes to preference tuning. The authors found DPO more stable and more accurate.
format_prompt builds <|system|> plus <|user|> prompts with chosen and rejected completions. The SFT adapter is loaded in 4-bit NF4 and merged, the same LoRA config is applied, and DPOConfig copies the earlier arguments except learning rate 1e-5, max_steps=200 (one epoch would take up to two hours) and warmup_ratio=0.1, which ramps the learning rate from 0 over the first 10% of steps to avoid early divergence. DPOTrainer(..., beta=0.1, max_prompt_length=512, max_length=512) saves a second adapter, TinyLlama-1.1B-dpo-qlora. Deployment merges iteratively: SFT adapter into the base, then the DPO adapter into that.SFT + DPO gives chat behaviour first and preference alignment second, at the cost of two training loops and two rounds of hyperparameter tuning. ORPO (Hong, Lee and Thorne, 2024) merges them into one training process, still compatible with QLoRA. The chapter's summary: PEFT through LoRA, made cheaper by quantization, then instruction tuning for chat behaviour and preference tuning to distil what answers people prefer.
SFTConfig(max_length=..., dataset_text_field=...) replaces max_seq_length, processing_class replaces tokenizer, beta lives in DPOConfig, and max_prompt_length is deprecated. TRL v1.15.0 (8 October 2026) scores tokens with a fused LM head in SFT, DPO, KTO, GRPO and RLOO, cutting peak memory 52% to 82% at 8,192 tokens. On evaluation, Hugging Face retired the Open LLM Leaderboard on 13 March 2025 after more than 13,000 models, and Chatbot Arena became LMArena, then Arena (arena.ai, 28 January 2026), which now ranks with a Bradley-Terry model plus style control rather than plain Elo.Main Points
- Two families, one toolbox. Encoders represent text for classification, clustering and search; decoders generate it. Most useful systems combine them, as RAG does with an embedding model, a reranker and a generator.
- Encoders still win on volume. A fine-tuned BERT scored F1 0.85 on sentiment in minutes, and SetFit reached 0.84 from 32 labelled reviews, at a fraction of an LLM call's cost per record.
- Embeddings are the common currency. Tokens, sentences, images and even songs become vectors whose distances carry meaning; once text is a vector, search, clustering, zero-shot labels and recommendations follow.
- Generation is a loop of one forward pass per token. The model scores the whole vocabulary, a sampler picks one token, and the KV cache stops recomputation, which cut generation from 21.8s to 4.5s in the book's test.
- Retrieval quality decides answer quality. Adding a reranker lifted nDCG@10 on the MIRACL benchmark from 36.5 to 62.8, and grounded generation with citations is only as good as the chunks it receives.
- Fine-tuning changes little, so it can be cheap. Low intrinsic dimension lets LoRA train a rank-8 update of about 197K weights in place of a 150M-weight GPT-3 matrix, and QLoRA does it on a 4-bit base model.
- Tokenizers shape what a model can see. Eight tokenizers on one string showed lost newlines, [UNK] emoji, eight-token capitals and whitespace handling that decides whether a model can read code.
- Attention is relevance scoring plus mixing. Queries meet keys to score positions, values are blended by those scores, and many heads do this in parallel; GQA and MQA share keys and values to save memory.
- Topic models are clustering plus representation. BERTopic embeds, reduces with UMAP, clusters with HDBSCAN, then names clusters with c-TF-IDF and optional reranking or LLM labels; each block is swappable.
- Prompting is a set of controls, not magic words. Temperature and top_p set randomness; persona, instruction, context, format, audience and tone shape output; examples and chains add structure.
- Give the model room to reason. Chain-of-thought, self-consistency and tree-of-thought trade more tokens for better answers; reasoning models later built this into training.
- Constrain output when a program will read it. Validation, grammars and JSON modes turn free text into data a pipeline can trust.
- A stateless model needs designed memory. Full buffers, windowed buffers and running summaries trade context cost against recall.
- Agents are ReAct loops. Thought, action, observation repeat until the model answers, which makes tool choice powerful and also fragile.
- Measure retrieval like retrieval. Precision at k, average precision and MAP judge ranked results; RAG adds faithfulness and answer relevance on top.
- Vision joins through shared embedding spaces. ViT turns patches into tokens, CLIP aligns images and captions contrastively, and BLIP-2 bridges a frozen image encoder to a frozen LLM with a small Q-Former.
- Contrastive learning needs good negatives. MNR loss with in-batch and hard negatives beat cosine loss, and choosing negatives that are similar but wrong teaches the finest distinctions.
- Freezing is a dial. Training even a few top encoder blocks recovers most of full fine-tuning; freezing the whole encoder dropped F1 to 0.63.
- Teach domain words before the task. Continued masked-language pretraining shifted BERT's guesses for "What a horrible [MASK]!" from "idea" to "movie".
- Token tasks need label alignment. NER labels belong to words, so subword pieces must inherit B- and I- tags and special tokens must be ignored.
- Preference tuning distils taste. Reward models with PPO, or DPO against a frozen reference, push the model toward accepted answers and away from rejected ones.
- You are the best evaluator. Word metrics, benchmarks, leaderboards and judges are proxies; test on your own task, and remember Goodhart's Law.
Frameworks and Mental Models
| Name | What it says | Use it when |
|---|---|---|
| Representation vs generative | Encoders embed, decoders write; both are Transformers | Choosing a model for a task |
| Tokenisation schemes | Word, subword, character, byte; subword is the default | Debugging odd outputs or costs |
| Forward pass | Tokenizer, embeddings, block stack, LM head, sampler | Reasoning about latency and memory |
| Attention | Score relevance with queries and keys, mix values | Explaining context use and KV cost |
| Five classifiers | Task model, embeddings plus classifier, zero-shot embeddings, encoder-decoder, prompted LLM | Picking the cheapest adequate method |
| BERTopic pipeline | Embed, reduce, cluster, c-TF-IDF, represent | Mining themes from text at scale |
| Prompt components | Persona, instruction, context, format, audience, tone, data | Writing or reviewing prompts |
| Reasoning ladder | CoT, self-consistency, tree-of-thought | Hard multi-step questions |
| Memory types | Buffer, windowed buffer, summary | Building multi-turn assistants |
| ReAct | Thought, action, observation loop | Giving an LLM tools |
| Retrieve, rerank, generate | Hybrid recall, cross-encoder precision, grounded answer | Question answering over documents |
| MAP | Average precision over queries for ranked results | Comparing search systems |
| Contrastive learning | Pull positives together, push negatives apart | Training embeddings or SetFit |
| Bi- vs cross-encoder | Bi-encoders scale, cross-encoders score pairs precisely | Designing search stages |
| Three training steps | Pretrain, SFT, preference tune | Explaining where model behaviour comes from |
| LoRA and QLoRA | Low-rank update on a frozen, optionally 4-bit base | Fine-tuning on one GPU |
| Evaluation stack | Word metrics, benchmarks, leaderboards, judges, humans | Deciding if a model is good enough |
The book's methods as a cost ladder: climb only as far as the task requires.
- Single-run evidence. Most comparisons rest on one run on one dataset, mostly Rotten Tomatoes, without variance; gaps of a few F1 points are not settled results.
- Interests. Cohere APIs and BERTopic feature prominently and come from the authors' own work.
- What holds. The visual explanations of tokens, embeddings, attention, contrastive learning, LoRA and DPO remain among the clearest available, and the encoder chapters are a useful corrective to LLM-only thinking.
About this book
Hands-On Large Language Models: Language Understanding and Generation by Jay Alammar and Maarten Grootendorst, 2024. This is a chapter-by-chapter study of the book, with summaries, maps, tables and diagrams; short quotations keep their page references. It is shared free for personal study, is not for sale, and is not endorsed by the author or publisher. Please read and buy the original.
Insight Editions · rajeshrmahapatra.com/insight-editions