Field note: work from token to governed output

TLDR

• Choose the model route from the task: representation for comparison and classification, generation for constrained sequence production, and a composed route when both are necessary.
• Tokenizers and embedding models are contracts. Their versions, normalisation, truncation, pooling and dimensionality must match between evaluation and use.
• A prompt is executable configuration. Version its instruction, evidence slots, output schema, decoding controls and verification route.
• Retrieval does not ground an answer by itself. Source capture, chunking, retrieval, reranking, citation and abstention must be evaluated as one evidence route.
• Fine-tuning changes a model under a dataset and objective. Compare the adapted candidate with the pinned base model and an independent human-use test.

Reader and route

This edition is for engineers, data scientists, architects, product leaders and control practitioners who need to inspect and adapt language-model systems. Part I covers representation and transformer mechanics. Part II applies classification, clustering and prompt programmes. Part III composes workflows, retrieval and multimodal systems. Part IV trains embeddings and adapts encoders and generators.

Evidence boundary

Merehaven is wholly fictional. Every company example, person, dataset, metric, experiment and operating result in the worked material is synthetic unless the source note identifies a published reference. Model names and code fragments are learning specimens, not current product recommendations. Select a supported implementation, pin its model, tokenizer, library and runtime versions, and verify current official documentation before use. Numerical thresholds are illustrative design inputs rather than universal targets.

Three model routes

The workbench card

For every experiment, record the task, evidence boundary, model and tokenizer versions, transformation route, decoding or similarity controls, evaluation population, failure slices, authority and recovery action. The card keeps a notebook experiment connected to the conditions under which its output may be used.

Part I: Representation before response

Language-model work begins with representation: token boundaries, embedding geometry, transformer state and the distinction between classification and generation.

1. Choose representation before generation

A language model is one component in a system that transforms evidence into an output. The first design choice is whether the task needs a representation, a generated sequence or a controlled combination of both.

Language systems transform representations

Language AI is used here for computational systems that represent, compare or generate human language. The term includes non-generative retrieval and classification routes as well as language models.

Language AI refers to a subfield of AI that focuses on developing technologies capable of understanding, processing, and generating human language. The term can often be used interchangeably with natural language processing (NLP), given the continued success of machine learning methods in tackling language processing problems. Retrieval, representation and generation are treated as distinct components because each needs different evidence and control.

From sparse counts to contextual generation

The history of Language AI encompasses many developments and models aiming to represent and generate language.

Computational models require a numerical representation of language. The representation preserves some structure and discards other structure, so its fitness depends on the task.

Representing Language as a Bag-of-Words

The history begins with a technique called bag-of-words, a method for representing unstructured text. It was first mentioned around the 1950s but became popular around the 2000s.

Bag-of-words works as follows. Assume we have two sentences for which we want to create numerical representations. The first step is tokenisation, the process of splitting up the sentences into individual words or subwords (tokens).

a common method for tokenisation is splitting on whitespace. However, this has disadvantages, as some languages, like Mandarin, do not have whitespaces around individual words. Chapter 2 goes in depth about tokenisation.

Then, each sentence is represented as a vector where each position corresponds to a word in the vocabulary. If the word appears in the sentence, it gets a 1; if not, it gets a 0. This creates the “bag-of-words” representation.

The technique is called “bag-of-words” because it treats the sentence like a bag, losing all information about word order. “The dog chased the cat” and “The cat chased the dog” get the same representation, even though they mean very different things.

Design limit: Bag-of-words has three material limitations: (1) it loses word order entirely, (2) all words are treated as equally different from each other (no notion of similarity), and (3) vocabulary size grows linearly with corpus size, creating extremely sparse, high-dimensional vectors. These problems motivated every subsequent development.

An important technique built on bag-of-words is TF-IDF (term frequency-inverse document frequency), which weights words by their importance: words that appear frequently in one document but rarely across all documents get higher weights. This highlights distinctive words while downweighting common stop words like “the” and “is.”

Better Representations with Dense Vector Embeddings

The limitations of bag-of-words drove the search for better representations. The breakthrough came in 2013 with word2vec, a method for representing words as dense vectors (also called embeddings) in a continuous vector space. Unlike the sparse, high-dimensional vectors of bag-of-words, word2vec creates compact vectors (typically 100-300 dimensions) where geometric relationships capture semantic relationships.

The widely cited demonstration of word2vec’s power is the analogy: king - man + woman ≈ queen. This shows that the model has learned abstract relational concepts (royalty, gender) that can be manipulated through simple vector arithmetic.

Word2vec works through a training process called self-supervised learning: given a large text corpus, the model learns to predict a word from its surrounding context (or vice versa). Through this prediction task, the model gradually positions related words closer together in the embedding space.

Types of Embeddings

Three representation levels serve different tasks:

Token/word embeddings represent individual tokens or words. These are what word2vec produces: a fixed vector for each word in the vocabulary. The limitation is that the word “bank” gets the same embedding whether it means a synthetic Merehaven institution or a river bank.

Sentence/document embeddings represent entire sequences of text. By aggregating word embeddings (through averaging, pooling, or more sophisticated methods), we get a single vector representing the meaning of a full sentence or document. These are important for tasks like semantic search and text classification.

Encoding and Decoding Context with Attention

While word2vec was consequential, it had a material limitation: each word gets a static embedding regardless of context. The word “bank” gets the same vector in “river bank” and “bank account.” This motivated the development of contextual embeddings, where a word’s representation changes based on the surrounding text.

The encoder-decoder architecture was particularly successful for tasks like machine translation, where the entire input (“I love llamas”) must be understood before producing the output (“Ik hou van lama’s”).

However, this context embedding creates a bottleneck: it is a single fixed-size vector that must capture the meaning of the entire input sentence. For short sentences, this works well. For longer sequences, important information gets compressed and lost. This is the material problem that attention was designed to solve.

In 2014, a solution called attention was introduced. Attention allows a model to focus on parts of the input sequence that are relevant to one another and amplify their signal.

As a result, during the generation of “Ik hou van lama’s,” the RNN keeps track of the words it mostly attends to in order to perform the translation. This sequential nature, however, precludes parallelization during training of the model, a limitation that would become the driving motivation for the Transformer.

Attention Is All You Need

Vaswani and peers introduced the Transformer in 2017 as an attention-based sequence architecture without recurrence. Its parallel training path was an important practical advantage over recurrent processing.

In the Transformer, encoding and decoder components are stacked on top of each other.

Both the encoder and decoder blocks revolve around attention instead of leveraging an RNN.

Compared to previous methods of attention, self-attention can attend to different positions within a single sequence, more easily and accurately representing the input.

This is called masked self-attention or causal attention, enforcing the autoregressive property.

Representation Models: Encoder-Only Models

The original Transformer is an encoder-decoder architecture that serves translation tasks well but cannot easily be used for other tasks like text classification. In 2018, BERT (Bidirectional Encoder Representations from Transformers) was introduced, an encoder-only architecture that focuses on representing language.

BERT adopts masked language modelling (MLM) for training: random words in the input are masked, and the model must predict them. This forces bidirectional understanding because the model must use both left and right context.

BERT is then fine-tuned for specific downstream tasks, like classification. Through transfer learning, a model pretrained on a general task (MLM) can be adapted to specific tasks with relatively little additional training data.

Generative Models: Decoder-Only Models

While BERT focuses on understanding, generative models focus on producing text. The GPT (Generative Pre-trained Transformer) family uses a decoder-only architecture.

GPT-family models use a causal language-modelling (CLM) objective: predict the next token from the preceding tokens. Unlike BERT’s bidirectional masking, GPT can only look left (backward), not right (forward). This makes generation natural but understanding less complete.

• GPT-1 (2018): Demonstrated that unsupervised pretraining + supervised fine-tuning works
• GPT-2 (2019): Extended zero-shot task demonstrations at a larger scale
• GPT-3 (2020): Demonstrated few-shot and zero-shot capabilities at 175B parameters
• ChatGPT (2022): Added RLHF (Reinforcement Learning from Human Feedback) alignment

The Training Paradigm of Large Language Models

Self-supervised pretraining constructs targets from the text itself, such as a next-token or masked-token objective. The resulting evidence is behaviour under that objective, not a guarantee of factual knowledge or reasoning.

Match representation and generation to the task

A shared model can support several tasks through different representations or prompt programmes. Each task still requires its own evidence, output contract and authority boundary.

Responsible LLM Development and Usage

The system review covers foreseeable harms alongside capability:

Bias and fairness: LLMs learn from internet text, which contains biases related to gender, race, religion, and other attributes. These biases can be amplified in model outputs.

unsupported generation: LLMs can generate plausible-sounding but factually incorrect text. This is particularly dangerous in high-stakes domains like healthcare and legal advice.

Environmental impact: Training and serving consume energy and hardware resources. Measure the selected route in its actual region, runtime and utilisation pattern rather than transferring one published estimate to another system.

Privacy: Training data may contain personal information that models can memorise and reproduce.

Mechanism: How word2vec Actually Learns

Understanding word2vec’s training process is important because the same contrastive learning principle reappears in Chapter 10 (embedding models) and Chapter 11 (SetFit). Word2vec comes in two flavors:

Skip-gram: Given a centre word, predict the surrounding context words. For example, given “chased” in “The dog chased the cat,” predict “dog” and “cat” appear nearby. The model learns to produce embeddings where words with similar contexts end up close together.

CBOW (Continuous Bag of Words): The reverse: given surrounding context words, predict the centre word. Given “The,” “dog,” “the,” “cat,” predict “chased.”

Both approaches use a shallow neural network with a single hidden layer. The hidden layer weights, after training, become the word embeddings. The training process is contrastive: for each positive example (words that do co-occur), several negative samples (randomly chosen words that do not co-occur) are generated. The model learns to assign high similarity scores to genuine neighbours and low scores to random non-neighbours.

This contrastive learning pattern, learning from positive and negative pairs, is the foundation of modern embedding model training. When you see contrastive learning in Chapter 10 with sentence-transformers, or in Chapter 9 with CLIP, recognize that it is the same material principle that word2vec pioneered, scaled to sentence-level and cross-modal representations.

Select a model route without trusting a catalogue

Start with a task receipt rather than a model list. Record the input, required output, evidence boundary, latency and privacy envelope, authority, abstention condition and recovery action. Then compare the smallest credible representation, generation and composed routes against one transparent baseline. A catalogue entry can identify candidates, but it cannot establish fitness for a decision.

Treat model cards and provider documentation as source material whose version and retrieval date must be recorded. Re-run the comparison whenever a candidate, dependency or operating population changes.

Workbench check

Record the mechanism that matters most to choose representation before generation, the evidence that tests it, the versioned configuration needed to reproduce it, and the fallback when the evidence is weak. The chapter is not operationally complete while any field is blank.

2. Treat tokens and embeddings as contracts

Tokenisation fixes the units a model can observe; embeddings fix the geometry through which it can compare them. Those choices affect length, multilingual behaviour, retrieval quality, cost and failure.

Tokenisation: From Text to Token IDs

tokenisation is the process of converting raw text into a sequence of integers (token IDs) that the model can process. Each token ID maps to a position in the model’s vocabulary. This seemingly simple step has substantial consequences for model performance, efficiency, and capabilities.

The following learning specimen exposes the tokenizer interface:

from transformers import AutoTokenizer

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

# Tokenize a simple sentence
text = "Tokenising text is the first step in any LLM pipeline."
tokens = tokenizer(text)
print(f"Token IDs: {tokens.input_ids}")
print(f"Number of tokens: {len(tokens.input_ids)}")

# Decode individual tokens to see what they represent
for tid in tokens.input_ids:
    print(f"  ID {tid:6d} → '{tokenizer.decode(tid)}'")

Implementation notes: AutoTokenizer.from_pretrained() downloads and loads the tokenizer that was trained alongside the specified model. The tokenizer object contains the vocabulary (mapping from tokens to IDs), the merging rules (how to split text into tokens), and special token definitions. Calling tokenizer(text) produces a dictionary containing input_ids (the sequence of token IDs), attention_mask (which positions are real tokens vs. padding), and optionally token_type_ids (for models like BERT that handle sentence pairs). The decode method reverses the process, converting IDs back to text.

Design limit: A tokenizer and its model form one versioned contract because token identifiers index the learned embedding matrix. Changing the tokenizer requires a separately trained or remapped and validated model route.

On the output side, we can inspect tokens generated by the model:

print(tokenizer.decode(3323))    # Sub
print(tokenizer.decode(622))     # ject
print(tokenizer.decode([3323, 622]))  # Subject
print(tokenizer.decode(29901))   # :

How Does the Tokenizer Break Down Text?

Three major factors dictate how a tokenizer breaks down an input prompt:

First: tokenisation Method. At model design time, the creator chooses a method. Two established methods are Byte Pair Encoding (BPE), widely used by GPT models, and WordPiece, used by BERT. Both aim to optimise an efficient set of tokens, but arrive at it differently.

BPE starts with individual characters and iteratively merges the most frequent adjacent pairs until reaching the target vocabulary size. For example, if “th” appears most frequently, it becomes a single token. Then if “the” appears frequently, it merges “th” + “e” into “the.” This bottom-up process creates subwords that balance between whole words (efficient for common words) and character fragments (fallback for rare words).

WordPiece is similar but uses a likelihood-based criterion for merges rather than pure frequency. It merges pairs that most increase the likelihood of the training data under a language model, which tends to produce slightly different splits than BPE.

Second: Tokenizer Parameters. After choosing the method, design decisions include:

Vocabulary size: How many tokens to include. Larger vocabularies mean fewer tokens per text (more efficient) but larger embedding matrices (more memory). GPT-2 uses ~50K, a selected generation model uses ~100K, BERT uses ~30K.

Special tokens: Tokens with specific meaning for the model: [CLS] (classification), [SEP] (separator), [MASK] (for MLM training), [PAD] (padding), <|endoftext|> (end of document), <|user|> and <|assistant|> (chat roles).

Third: Training Dataset. The tokenizer is trained on a specific dataset. A tokenizer trained on English text produces different merges than one trained on code or multilingual text. This is why code-focused models have tokenizers that efficiently handle programming syntax.

Word Versus Subword Versus Character Versus Byte Tokens

Word tokens: Common with earlier methods like word2vec. One token per word. Simple but creates huge vocabularies and cannot handle new words (OOV problem). The word “apologizing” is one token, but so is “apologize,” “apologetic,” and “apologist,” wasting vocabulary slots on minor variations.

Subword tokens: Contains full and partial words. BPE and WordPiece produce these. “Apologizing” becomes [“apolog”, “izing”], sharing the root “apolog” with all its variants. This is the most efficient scheme and dominates modern LLMs.

Character tokens: Individual characters as tokens. Vocabulary is tiny (~100-300 characters) and can represent any text, but sequences become very long (3-4× more tokens than subword), consuming precious context window space.

Byte tokens: Raw bytes (0-255) as tokens. Vocabulary is exactly 256. Papers like “ByT5: Towards a token-free future” show this can work, especially for multilingual scenarios. Even longer sequences than character tokenisation but handles any encoding.

Token Embeddings

Once text is tokenised into IDs, each token ID is mapped to a dense vector through the model’s embedding matrix. This matrix has dimensions [vocabulary_size × embedding_dimension]. Each row is a learnable vector that starts random and is refined through training.

from transformers import AutoModel

# Load a model to access its embedding matrix
model = AutoModel.from_pretrained("bert-base-uncased")

# The embedding matrix
embeddings = model.embeddings.word_embeddings.weight
print(f"Embedding matrix shape: {embeddings.shape}")
# Output: torch.Size([30522, 768])
# 30,522 tokens × 768 dimensions per token

Implementation notes: The embedding matrix has shape [30522, 768] for BERT-base. This means there are 30,522 tokens in the vocabulary, each represented by a 768-dimensional vector. That is 30,522 × 768 = 23.4 million parameters just in the embedding layer. For a selected generation model with ~100K vocabulary and larger embedding dimensions, this can be hundreds of millions of parameters.

Creating Contextualised Word Embeddings with Language Models

The important distinction in modern NLP is between static embeddings (word2vec, GloVe) and contextualised embeddings (BERT, GPT). Static embeddings assign the same vector to a word regardless of context. Contextualised embeddings produce different vectors for the same word depending on the surrounding text.

To extract contextualised embeddings from a language model:

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-base")
model = AutoModel.from_pretrained("microsoft/deberta-v3-base")

text = "The bank of the river was steep."
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

# outputs.last_hidden_state contains contextualised embeddings
# Shape: [batch_size, sequence_length, hidden_dim]
print(f"Output shape: {outputs.last_hidden_state.shape}")

Implementation notes: return_tensors="pt" returns PyTorch tensors rather than Python lists. torch.no_grad() disables gradient computation since we are doing inference, not training, saving memory. model(**inputs) runs the forward pass through all Transformer layers. The last_hidden_state contains the final-layer embeddings for each token position, which are the most contextualised representations. The shape is [1, num_tokens, 768] where 768 is the hidden dimension.

Text Embeddings: From Tokens to Sentences

While token embeddings represent individual tokens, many applications need embeddings for entire sentences or documents. Text embeddings aggregate token-level representations into a single vector for the whole input.

The sentence-transformers library provides optimised models for producing high-quality text embeddings:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium."
]
embeddings = model.encode(sentences)
print(f"Embedding shape: {embeddings.shape}")
# Output: (3, 384)  -- 3 sentences × 384 dimensions

Computing similarity between sentence embeddings reveals semantic relationships:

from sklearn.metrics.pairwise import cosine_similarity

similarities = cosine_similarity(embeddings)
print(similarities)
# sentences 0 and 1 (both about weather) have high similarity
# sentence 2 (about driving) has low similarity with both

Word2Vec Mechanism: Skip-Gram and Negative Sampling

Word2vec, despite being from 2013, remains foundational because its training principles reappear in modern embedding models. Two key concepts:

Skip-gram: Given a centre word, predict context words. For the sentence “The cat sat on the mat” with centre word “sat” and window size 2, positive pairs are: (sat, cat), (sat, on), (sat, the), (sat, the). The model learns to produce embeddings where “sat” is close to words it typically appears near.

Negative sampling: For each positive pair (sat, cat), generate N negative pairs with random words: (sat, elephant), (sat, quantum), (sat, refrigerator). The model learns to maximise similarity for positive pairs and minimise it for negative pairs. This contrastive training is the direct ancestor of the MNR loss in Chapter 10.

Embeddings Beyond Text: The Song Recommender

Sequence-based embedding objectives can also represent non-text items when co-occurrence expresses the relation of interest.

from datasets import load_dataset

# Load the Spotify playlists dataset
dataset = load_dataset("maharshipandya/spotify-tracks-dataset")

# Example playlist structure
# Each playlist is a list of song IDs, like a sentence of words
playlists = [
    ['0', '1', '2', '3', '4', '5', ..., '43'],   # Playlist 1
    ['78', '79', '80', '3', '62', ..., '210'],    # Playlist 2
]

Training word2vec on playlists:

from gensim.models import Word2Vec

model = Word2Vec(
    playlists,
    vector_size=32,    # 32-dimensional embeddings per song
    window=20,         # Large window (playlists have weaker ordering)
    negative=50,       # High negative sampling (small "vocabulary")
    min_count=1,       # Include all songs
    workers=4          # Parallel training
)

Implementation notes: vector_size=32 produces compact 32-dimensional song embeddings (much smaller than text embeddings because the “vocabulary” of songs is smaller). window=20 is much larger than typical text training (usually 5) because song order within a playlist is less rigid than word order in sentences. negative=50 uses 50 negative samples per positive example, much higher than typical text training (usually 5-15), because with a small vocabulary, the model needs more negatives to avoid learning trivial patterns. min_count=1 includes all songs regardless of frequency. workers=4 enables parallel processing.

Finding similar songs:

song_id = 2172  # Metallica - Fade to Black
model.wv.most_similar(positive=str(song_id))

The recommendations are all heavy metal and hard rock: Van Halen, Dio, Guns N’ Roses, Judas Priest. The embedding space has learned genre similarity from co-occurrence patterns in playlists.

Mechanism: BPE Tokenisation Step by Step

To fully internalize how BPE works, let us trace through a complete example. Suppose our training corpus contains only these words (with frequencies):

"low" (5 times), "lower" (2 times), "newest" (6 times), "widest" (3 times)

Step 0 : Initialize with characters: Vocabulary = {l, o, w, e, r, n, s, t, i, d} Representations: l·o·w (5), l·o·w·e·r (2), n·e·w·e·s·t (6), w·i·d·e·s·t (3)

Step 1 : Count pairs and merge most frequent: Pair counts: (e,s)=9, (s,t)=9, (l,o)=7, (o,w)=7, (n,e)=6, (e,w)=6, (w,e)=8, (i,d)=3, … Merge (e,s) → es (count 9, tied with st; pick first) Vocabulary = {…, es} Representations: l·o·w (5), l·o·w·e·r (2), n·e·w·es·t (6), w·i·d·es·t (3)

Step 2 : Count again and merge: Now (es,t)=9 is highest Merge (es,t) → est Vocabulary = {…, es, est} Representations: l·o·w (5), l·o·w·e·r (2), n·e·w·est (6), w·i·d·est (3)

Step 3 : Continue merging: (l,o)=7 is next Merge (l,o) → lo Representations: lo·w (5), lo·w·e·r (2), n·e·w·est (6), w·i·d·est (3)

And so on until we reach the target vocabulary size.

Mechanism: The Mathematics of Cosine Similarity

Understanding cosine similarity is important because it is the primary metric for comparing embeddings in the following embedding specimens. For two vectors A and B, cosine similarity measures the cosine of the angle between them:

cosine_similarity(A, B) = (A · B) / (||A|| × ||B||)

Where A · B is the dot product (sum of element-wise products) and ||A|| is the L2 norm (square root of sum of squared elements). The result ranges from -1 (opposite directions) to 0 (orthogonal/unrelated) to +1 (same direction/identical meaning).

In practice:

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Example: two similar sentence embeddings
embedding_weather1 = np.array([0.8, 0.6, 0.1, 0.2])  # "It's sunny"
embedding_weather2 = np.array([0.7, 0.7, 0.1, 0.15]) # "The sun is out"
embedding_sports   = np.array([0.1, 0.2, 0.9, 0.8])  # "The game starts at 3"

print(cosine_similarity(embedding_weather1, embedding_weather2))  # ~0.99 (very similar)
print(cosine_similarity(embedding_weather1, embedding_sports))    # ~0.35 (dissimilar)

Why cosine over Euclidean distance? In high-dimensional spaces (768+ dimensions), Euclidean distances tend to converge (all points become roughly equidistant), making comparisons less meaningful. Cosine similarity remains discriminative because it normalizes for vector magnitude, focusing purely on direction.

Compare tokenizers as measured transformations

A tokenizer comparison is meaningful only against the text the system will process. Use a fixed corpus containing ordinary language, specialist terms, identifiers, numbers, punctuation, multiple scripts and long inputs. Compare fragmentation, retained meaning after truncation, round-trip behaviour and sequence-length distribution. A smaller vocabulary or shorter average sequence is not automatically better if critical terms split unpredictably.

Keep the tokenizer and model coupled unless a separately validated adaptation changes both contracts. For every embedded corpus, preserve the tokenizer identifier, model identifier, pooling rule, normalisation rule, dimension and source snapshot. Rebuilding an index with a new representation creates a new artefact, even when the documents are unchanged.

Workbench check

Record the mechanism that matters most to treat tokens and embeddings as contracts, the evidence that tests it, the versioned configuration needed to reproduce it, and the fallback when the evidence is weak. The chapter is not operationally complete while any field is blank.

3. Inspect the transformer as a state machine

A transformer updates token states through attention and feed-forward operations. Inspecting those operations, cache growth and context boundaries makes performance and failure easier to measure.

An Overview of Transformer Models

A transformer consumes token identifiers and updates their hidden states. A causal generator then produces one token at a time through an autoregressive loop.

The Inputs and Outputs of a Trained Transformer LLM

Loading the model and running a forward pass:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
model = AutoModelForCausalLM.from_pretrained(
    "microsoft/Phi-3-mini-4k-instruct",
    device_map="cuda",
    torch_dtype="auto",
    revision="pinned_commit",
)

generator = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    return_full_text=False,
    max_new_tokens=50,
    do_sample=False,
)

The Components of the Forward Pass

The three major components:

1. Tokenizer + Embedding layer: Converts raw text to token IDs, then maps each ID to its embedding vector from the embedding matrix.
2. Stack of Transformer blocks: The core processing engine. Each block contains an attention layer and a feedforward neural network. The blocks are stacked sequentially (6 in the original Transformer, 32 in Phi-3, 80 in Llama 2 70B).
3. Language modelling (LM) Head: A linear projection that maps the final hidden state to a probability distribution over the entire vocabulary.

# Manual forward pass to see the internals
input_text = "The capital of France is"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

# Forward pass through the model
with torch.no_grad():
    model_output = model.model(input_ids)  # Transformer blocks only
    lm_head_output = model.lm_head(model_output[0])  # LM head projection

Implementation notes: model.model(input_ids) runs the input through the embedding layer and all Transformer blocks, producing contextualised hidden states. model.lm_head(model_output[0]) applies the final linear projection. The separation of model.model (Transformer) and model.lm_head (projection) is a Hugging Face convention that lets you access intermediate representations.

The output dimensions reveal the architecture:

model_output[0].shape    # torch.Size([1, 6, 3072])
lm_head_output.shape     # torch.Size([1, 6, 32064])

A batch of 1 input containing 6 tokens, each represented by a 3,072-dimensional vector after the Transformer stack, projected to 32,064-dimensional logits (one score per vocabulary token) by the LM head.

Choosing a Single Token from the Probability Distribution (Sampling/Decoding)

The LM head produces raw scores (logits) for each token in the vocabulary. These are converted to probabilities using softmax, then a decoding strategy selects the actual output token.

import torch.nn.functional as F

# Get logits for the last token position only
next_token_logits = lm_head_output[0, -1, :]

# Convert to probabilities
probs = F.softmax(next_token_logits, dim=-1)

# Greedy decoding: pick the highest probability token
next_token_id = torch.argmax(probs)
print(f"Next token: '{tokenizer.decode(next_token_id)}'")
print(f"Probability: {probs[next_token_id]:.4f}")
# Output: "Next token: 'Paris'" with high probability

Parallel Token Processing and Context Size

A key reason Transformers are so useful is their ability to process tokens in parallel. Unlike RNNs that process one token at a time, a Transformer processes all input tokens simultaneously.

Every deployed transformer has a declared context limit. Inputs beyond that limit require an explicit truncation, segmentation or rejection policy.

For text generation, only the output of the last stream is used to predict the next token. The seemingly wasted computations for earlier streams are important: their intermediate results are used by the attention mechanism to compute the final stream’s output.

Speeding Up Generation by Caching Keys and Values

When generating the second token, the output token is appended to the input and another forward pass is done. If the model caches the results of previous calculations (specifically the key and value vectors in the attention mechanism), the previous streams need not be recomputed. Only the new last stream needs computation.

This optimisation is called the KV cache (keys and values cache).

# Timing with cache (default: enabled)
%%timeit -n 1
generation_output = model.generate(
    input_ids=input_ids, max_new_tokens=100, use_cache=True
)

# Timing without cache
%%timeit -n 1
generation_output = model.generate(
    input_ids=input_ids, max_new_tokens=100, use_cache=False
)

The speed effect of caching depends on context length, batch shape, model architecture, runtime and hardware. The KV cache trades memory (storing cached key and value tensors for each layer and each previous token) for computation (avoiding redundant forward passes).

Cache memory: Key-value state grows with sequence length, batch size, layer count, head configuration and numerical precision. Measure it on the accepted route and define eviction before enabling long contexts or concurrent sessions.

Inside the Transformer Block

The majority of processing happens in the Transformer blocks.

The Feedforward Neural Network at a Glance

Feed-forward layers account for much of a transformer’s parameter count and contribute materially to the token-state update. When you ask a model “The Shawshank” and it predicts “Redemption,” that factual association is stored in the feedforward weights.

A trained model can reproduce memorised associations and interpolate patterns across inputs. Controlled tests are needed to distinguish useful generalisation from unsupported completion.

Base and instruction-tuned routes: A base causal model continues a token sequence. Instruction and preference adaptation can change that response pattern, but the exact behaviour belongs to the pinned model and prompt contract.

The Attention Layer at a Glance

Context is vital to properly model language. Consider: “The dog chased the squirrel because it…” For the model to predict what comes after “it,” it must determine whether “it” refers to the dog or the squirrel. The attention mechanism makes this determination.

How Attention Is Calculated: Queries, Keys, and Values

Attention operates through three learned projection matrices that transform each token’s embedding into three new vectors:

Query (Q): “What am I looking for?” The current token’s query vector encodes what kind of context it needs.

Key (K): “What do I have to offer?” Each previous token’s key vector advertises what kind of information it contains.

Value (V): “Here is my actual information.” Each previous token’s value vector contains the information that will be incorporated if the token is deemed relevant.

Self-attention: Relevance Scoring

The relevance scoring step multiplies the query vector of the current position with the keys matrix of all previous positions. This produces a score stating how relevant each previous token is. Passing through softmax normalizes these scores to sum to 1.

Self-attention: Combining Information

Now that we have relevance scores, we multiply the value vector of each token by its relevance score. Summing these weighted vectors produces the output of attention.

Multi-Head Attention

To give the model richer attention capability, the attention mechanism is duplicated and executed multiple times in parallel. Each parallel application runs inside an attention head.

Different heads learn to attend to different types of relationships: one head might focus on syntactic dependencies (subject-verb agreement), another on semantic relationships (synonyms, antonyms), another on positional patterns (nearby words), and another on coreference (pronouns and their antecedents).

Mechanism: The Complete Attention Mathematics

The scaled dot-product calculation exposes the states and numerical operations that an implementation must preserve:

Step 1: Create Q, K, V matrices. For input X (shape: [seq_len, d_model]), multiply by learned weight matrices: - Q = X × W_Q (shape: [seq_len, d_k]) - K = X × W_K (shape: [seq_len, d_k]) - V = X × W_V (shape: [seq_len, d_v])

Step 2: Compute attention scores. Multiply Q by K transposed and scale: - Scores = (Q × K^T) / sqrt(d_k) (shape: [seq_len, seq_len])

The division by sqrt(d_k) is important: without it, the dot products grow large with high dimensions, pushing softmax into regions with extremely small gradients (the “vanishing gradient” problem). This scaling keeps gradients healthy during training.

Step 3: Apply causal mask (for decoder models). Set all positions above the diagonal to -infinity, preventing the model from attending to future tokens: - Masked_Scores[i][j] = -inf if j > i

Step 4: Apply softmax. Convert scores to probabilities (each row sums to 1): - Attention_Weights = softmax(Masked_Scores) (shape: [seq_len, seq_len])

Step 5: Compute weighted values. Multiply attention weights by V: - Output = Attention_Weights × V (shape: [seq_len, d_v])

In code:

import torch
import torch.nn.functional as F
import math

def scaled_dot_product_attention(Q, K, V, mask=None):
    """
    Q, K, V: [batch, heads, seq_len, d_k]
    mask: [seq_len, seq_len] causal mask
    """
    d_k = Q.size(-1)

    # Step 2: Compute scaled attention scores
    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
    # [Study Note] Without sqrt(d_k) scaling, large d_k causes
    # softmax to produce near-one-hot distributions, killing gradients

    # Step 3: Apply causal mask
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float('-inf'))

    # Step 4: Softmax to get attention weights
    attention_weights = F.softmax(scores, dim=-1)

    # Step 5: Weighted sum of values
    output = torch.matmul(attention_weights, V)

    return output, attention_weights

Implementation notes: The function implements the core attention formula: Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) × V. The masked_fill operation sets future positions to -infinity, which softmax converts to exactly 0, ensuring zero attention to future tokens. The returned attention_weights can be visualised to see what each token attends to, which is useful for interpretability.

Evaluate architectural changes through the route

Attention variants, positional schemes, cache layouts and numerical kernels can change speed, memory and output. Their names do not establish an improvement. Pin the implementation and compare it with the accepted route on sequence lengths, batch patterns and hardware that resemble use. Measure end-to-end latency and memory alongside task quality, because a faster kernel can leave retrieval, serialisation or verification as the binding constraint.

Cache tests should include growth with context, concurrent sessions, eviction, stale state and isolation between users. Numerical changes need regression tests for output distribution and failure slices. An architecture change becomes promotable only when the full route, including rollback, remains inside its evidence and service envelope.

Workbench check

Record the mechanism that matters most to inspect the transformer as a state machine, the evidence that tests it, the versioned configuration needed to reproduce it, and the fallback when the evidence is weak. The chapter is not operationally complete while any field is blank.

Part II: Apply models through evidence routes

Classification, clustering and prompts become useful only when their evidence, uncertainty and fallback routes are explicit.

4. Classify with calibrated routes

Classification can use a task-specific encoder, an embedding baseline or a constrained generator. Calibration and fallback determine how those scores enter a human-owned route.

Text Classification with Representation Models

Classification with pretrained representation models comes in two flavors: using a task-specific model (a representation model fine-tuned for a specific task like sentiment analysis) or an embedding model (a model that generates general-purpose embeddings usable for many tasks).

The following comparison keeps each representation model frozen so that the evaluation measures its unadapted output. Adapted encoders are considered separately in Chapter 11.

Select classifiers through calibration and population fit

Model age, parameter count and leaderboard position are weak substitutes for a controlled comparison. Build a candidate set that includes a transparent rule, a compact task-specific encoder, an embedding-plus-classifier route and, only when justified, a constrained generator. Use the same frozen evaluation population and report uncertainty across decision-relevant slices.

In a task-specific encoder, token states pass through the transformer and a declared pooled representation enters a classification head. In an embedding route, a separately versioned representation becomes input to a simpler classifier. In a generative route, the response must be constrained to the allowed labels and parsed through a schema. These routes expose different failure surfaces, so their scores are not interchangeable.

The selected route needs a threshold policy rather than only an accuracy score. Calibrate scores on data that represents the proposed use, measure the cost of false routing and abstention, and define what happens when inputs fall outside the accepted population. Preserve the candidate manifest and rejection reasons so that later replacement starts from evidence rather than fashion.

Workbench check

Record the mechanism that matters most to classify with calibrated routes, the evidence that tests it, the versioned configuration needed to reproduce it, and the fallback when the evidence is weak. The chapter is not operationally complete while any field is blank.

5. Use clusters as hypotheses

Clustering and topic modelling propose structure rather than discover ground truth. Stability, interpretability and downstream usefulness must be tested across samples and time.

From Text Clustering to Topic Modelling

Finding themes or latent topics in textual data is topic modelling. Classic approaches like latent Dirichlet allocation (LDA) assume each topic is characterized by a probability distribution of words.

Count-based topic models and contextual-embedding pipelines encode different notions of similarity. Compare their stability, interpretability and downstream value on the declared corpus.

Evaluating Topic Models

Topic-model evaluation combines quantitative diagnostics with structured human judgement:

Coherence: Test whether high-weight topic terms co-occur in the evaluation corpus, then inspect whether that measure agrees with blinded human interpretation.

Diversity Score: Measure how different topics are from each other. Low diversity = topics are redundant. Calculate as the fraction of unique words across all topic keyword lists.

Downstream Task Performance: If topics feed into a downstream system (e.g., document routing), evaluate the end-to-end system performance.

Mechanism: How UMAP Works Internally

Understanding UMAP’s internals helps you tune it effectively and diagnose when it produces poor results. UMAP operates in three conceptual stages:

Stage 1: Build a weighted graph in high-dimensional space. For each point, find its k nearest neighbours (controlled by n_neighbors). Create edges between each point and its neighbours, weighted by distance. Points with more neighbours share the same local structure. The n_neighbors parameter determines the “resolution” of this graph: low values create a sparse graph capturing fine local structure, high values create a dense graph capturing broader patterns.

Stage 2: Construct a similar graph in low-dimensional space. Initialize a random low-dimensional layout (e.g., 5D). Build a similar neighbour graph in this low-dimensional space using a t-distribution-like kernel (controlled by min_dist). The goal is to make this low-dimensional graph match the high-dimensional graph as closely as possible.

Stage 3: optimise the low-dimensional layout. Use stochastic gradient descent to adjust the low-dimensional positions so that the two graphs match. Points that are neighbours in high-dimensional space should be neighbours in low-dimensional space (attract). Points that are not neighbours should be pushed apart (repel). After convergence, the low-dimensional positions are the output.

# Visualizing the effect of n_neighbors
from umap import UMAP
import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 3, figsize=(18, 5))

for i, nn in enumerate([5, 15, 50]):
    umap_2d = UMAP(n_components=2, n_neighbors=nn, min_dist=0.0, random_state=42)
    reduced_2d = umap_2d.fit_transform(embeddings[:5000])

    axes[i].scatter(reduced_2d[:, 0], reduced_2d[:, 1],
                    c=clusters[:5000], cmap='Spectral', alpha=0.3, s=2)
    axes[i].set_title(f"n_neighbors={nn}")
    axes[i].set_xticks([]); axes[i].set_yticks([])

plt.suptitle("Effect of n_neighbors on UMAP output")
plt.tight_layout()
plt.show()

What you will observe: With n_neighbors=5, you see many small, tight clusters (fine-grained local structure). With n_neighbors=15 (default), you see a balanced mix of local and global structure. With n_neighbors=50, clusters merge and the global structure dominates, with fewer but larger groups.

Mechanism: The Mathematics of c-TF-IDF

Understanding the c-TF-IDF formula helps you interpret topic keyword scores and diagnose representation quality issues.

The c-TF-IDF score for term x in class (cluster) c is:

w(x, c) = tf(x, c) × log(1 + A / tf(x))

Where: - tf(x, c) = frequency of term x in class c (how often this word appears in this cluster) - A = average number of words per class (normalizes for cluster size differences) - tf(x) = total frequency of term x across ALL classes (how common this word is globally)

Interpretation: A word with high c-TF-IDF in a cluster is: (1) frequent within that cluster (high tf(x, c)) and (2) rare across other clusters (low tf(x), making the log term large). Stop words like “the” appear equally in all clusters, so their log term is approximately log(1 + A/very_large) ≈ 0, giving them near-zero c-TF-IDF regardless of frequency. Domain-specific terms like “ASR” appear primarily in one cluster, giving them high c-TF-IDF.

# Manual c-TF-IDF calculation for one topic
import numpy as np

# Suppose topic 0 has these word frequencies:
topic_0_words = {"speech": 450, "asr": 320, "recognition": 280, "the": 2100, "and": 1800}

# Global frequencies across all clusters:
global_freqs = {"speech": 500, "asr": 330, "recognition": 400, "the": 45000, "and": 42000}

A = 300  # Average words per cluster

for word, tf_c in topic_0_words.items():
    tf_global = global_freqs[word]
    ctfidf = tf_c * np.log(1 + A / tf_global)
    print(f"  {word:15s} → c-TF-IDF: {ctfidf:.2f}")
    # "speech" and "asr" get high scores; "the" and "and" get near-zero

Evaluate clusters as provisional structure

A clustering pipeline combines a representation, distance measure, optional dimensionality reduction and grouping rule. Each choice changes the structure that appears. Compare at least two plausible configurations and repeat the fit across seeds or resampled documents. A cluster that disappears under a modest change should not support an operating claim.

Inspect representative, boundary and outlier documents without showing assessors the generated topic name first. Record whether the group is coherent, useful for the downstream task and distinct from neighbouring groups. Projection plots are navigation aids only; assess distances and assignments in the space used by the algorithm.

Choose minimum cluster size, neighbourhood and density controls from the use case and validation evidence rather than a copied default. Preserve unmatched documents as an explicit outlier set. Monitor changes in assignment, vocabulary and reviewer interpretation when the source population changes. If topics feed routing or reporting, measure that downstream outcome against a non-clustered baseline and retain human authority over labels.

Workbench check

Record the mechanism that matters most to use clusters as hypotheses, the evidence that tests it, the versioned configuration needed to reproduce it, and the fallback when the evidence is weak. The chapter is not operationally complete while any field is blank.

6. Engineer prompts as versioned programmes

A prompt is a versioned programme that combines instruction, evidence, constraints and output schema. It requires evaluation and verification like any other executable interface.

Intro to Prompt Engineering

Prompt design specifies instruction, evidence, constraints and output structure. Treat each revision as an experimental configuration with a version and evaluation result.

The Basic Ingredients of a Prompt

Instruction-Based Prompting

The instruction is an important component. Being specific is important: instead of “Tell me about dogs,” use “List three health benefits of owning a dog, each in one sentence.

Key principles: be specific, mitigate unsupported generations (“Only use information from the provided context” or “If you don’t know, say ‘I don’t know’”), specify format, length, and tone.

Temperature changes the decoding distribution

Understanding the mathematics behind temperature helps you choose values precisely rather than guessing:

The standard softmax function converts logits (raw model scores) to probabilities:

P(token_i) = exp(logit_i) / Σ exp(logit_j)

With temperature T, the formula becomes:

P(token_i) = exp(logit_i / T) / Σ exp(logit_j / T)

When T = 1.0 (default): probabilities are unchanged from standard softmax. This is the model’s “natural” distribution.

When T → 0 (approaching zero): division by a small T makes large logits even larger and small logits even smaller. After softmax, this concentrates almost all probability mass on the top token. In the limit, temperature 0 = greedy decoding (consistently pick the most probable token).

When T > 1 (high temperature): division by a large T compresses all logits toward zero, making the probability distribution more uniform. All tokens become roughly equally likely. As temperature rises, probability mass spreads across more candidates; the resulting behaviour must be measured rather than inferred from one threshold.

import torch
import torch.nn.functional as F

# Example logits for "I am driving a ___"
logits = torch.tensor([5.0, 4.2, 3.8, 1.0, 0.5, -2.0])
tokens = ["car", "truck", "bus", "bike", "horse", "elephant"]

for temp in [0.1, 0.5, 1.0, 1.5, 2.0]:
    probs = F.softmax(logits / temp, dim=-1)
    print(f"\nTemperature = {temp}:")
    for token, prob in zip(tokens, probs):
        bar = "█" * int(prob * 50)
        print(f"  {token:10s} {prob:.4f} {bar}")

What you will see: At T=0.1, “car” gets ~99% probability. At T=1.0, “car” gets ~55%, “truck” ~25%, “bus” ~15%. At T=2.0, all tokens get 10-25% probability, making selection nearly random.

Control decoding without treating settings as policy

Temperature, nucleus sampling, token limits and stop sequences shape a generation distribution. They do not prove factuality, safety or compliance. Change one control at a time, keep the prompt and evidence fixed, and compare repeated outputs on support, schema validity, task outcome and variance. A greedy route can still vary across model or runtime changes and can still generate unsupported claims.

Version the prompt programme

Avoid requests for hidden reasoning. Ask for an answer in the required schema, cited evidence, explicit assumptions and a concise verification record that can be checked independently. Prompt changes should pass regression and adversarial suites before release.

Workbench check

Record the mechanism that matters most to engineer prompts as versioned programmes, the evidence that tests it, the versioned configuration needed to reproduce it, and the fallback when the evidence is weak. The chapter is not operationally complete while any field is blank.

Part III: Compose retrieval and multimodal systems

Workflows, retrieval and multimodal models add interfaces. Each interface needs a bounded contract and a test that matches the operating task.

7. Compose generation without inventing authority

Chains, state and tools extend generation into a workflow. Authority must remain outside the model, with explicit tool permissions, stopping rules and recovery paths.

Compose deterministic control around probabilistic output

A model may propose a route or tool arguments; deterministic software validates whether the call is allowed. Tool access should use a small allow-list, typed arguments, least privilege, bounded time and output, and an idempotency policy. Web retrieval requires destination controls, request limits, content-type checks, rights handling and isolation from internal addresses. Generated text must not be executed as code or treated as authorisation.

from dataclasses import dataclass
from typing import Literal

@dataclass(frozen=True)
class ToolProposal:
    tool: Literal["case_lookup", "evidence_search"]
    case_id: str
    evidence_scope: tuple[str, ...]
    request_id: str

def validate_proposal(p: ToolProposal, allowed_cases: set[str]) -> bool:
    return p.case_id in allowed_cases and bool(p.evidence_scope) and bool(p.request_id)

The validator checks identity, scope and replay information before a separate authorised component performs the action. The model never receives credentials or a generic execution surface.

Keep workflow state explicit

Conversation summaries and agent scratchpads are lossy model outputs, not trusted records. Store authoritative state in a typed system of record and provide only the minimum authorised view to each model call. A workflow receipt should identify the state version read, evidence supplied, model and prompt versions, tool proposal, validation result, human decision and recovery status. Test malformed calls, tool refusal, timeout, partial completion, repeated delivery, stale evidence and attempts to cross the authority boundary.

Workbench check

Record the mechanism that matters most to compose generation without inventing authority, the evidence that tests it, the versioned configuration needed to reproduce it, and the fallback when the evidence is weak. The chapter is not operationally complete while any field is blank.

8. Retrieve evidence before generating claims

Retrieval-augmented generation joins a search system to a generator. The useful object is the evidence route from source capture through retrieval and citation to an answer or abstention.

Overview of Semantic Search and RAG

Three broad categories of language model search applications:

Dense retrieval: Relies on embeddings, turning the search problem into retrieving the nearest neighbours of the search query after both query and documents are converted into embeddings.

Reranking: A reranking language model scores the relevance of a subset of results against the query, then reorders them.

RAG: Generative search systems that include a model generating an answer in response to a query. RAG systems are text generation systems incorporating search capabilities to reduce unsupported generations, increase factuality, and ground generation on specific datasets.

Semantic Search with Language Models

Dense Retrieval

Points close together represent similar text.

Chunking Long Texts

Design limit: Chunking is one of the most impactful design decisions in a RAG system. Chunks that are too small lose context (e.g., “He won the award” without knowing who “He” refers to). Chunks that are too large dilute the signal with irrelevant text. Overlapping chunks help mitigate boundary effects where relevant information spans two chunks.

Retrieval Evaluation Metrics

Normalized Discounted Cumulative Gain (nDCG) handles graded relevance rather than binary.

Retrieval-Augmented Generation (RAG)

The mass adoption of LLMs led to people asking them questions and expecting factual answers. While models can answer some questions correctly, they also confidently answer many incorrectly. The leading remedy is RAG, described in the paper “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (Lewis et al., 2020).

Advanced RAG Techniques

Query rewriting: Reformulating the user’s query to improve retrieval. A user asking “Why is the sky blue?” might be rewritten to “Rayleigh scattering atmosphere blue light wavelength.”

Multi-query RAG: Generating multiple query variations and retrieving documents for each, increasing recall. If the original query is ambiguous, multiple perspectives help capture all relevant documents.

Multi-hop RAG: Iteratively retrieving and reasoning, where results from one retrieval step inform the next query. This can support questions that require synthesizing information from multiple sources.

Query routing: Directing queries to different retrieval systems based on the query type. Technical questions go to a code documentation index, while general questions go to a wiki index.

RAG Evaluation

Evaluating RAG systems requires multiple axes: Fluency (cohesive text), Perceived utility (helpful answer), Citation recall (claims supported by citations), and Citation precision (citations supporting their claims).

Build retrieval as a measured evidence route

Retrieval begins with an authorised, immutable source snapshot. Parse the source while preserving document identity, location, rights and effective date. Chunking should follow the unit a reviewer needs to understand, then be compared with at least one simpler boundary rule. Embed index documents and queries with the exact representation contract expected by the selected model. If the model uses different query and document encoders, record both.

A two-stage route can retrieve a broad candidate set and apply a more expensive pairwise scorer to a smaller subset. The second stage is useful only when it improves the decision-relevant retrieval measures enough to justify latency and operating cost. Validate every retrieved identifier against the authorised source store before exposing text to generation.

Measure retrieval independently from generation. A fluent answer can hide a failed retriever, while a strong retriever can be obscured by a weak synthesis prompt. Keep query, retrieved identifiers, scores, cited spans, model versions and final disposition in one receipt. Caches must include the source and route versions in their keys so that revoked or changed evidence cannot survive invisibly.

Workbench check

Record the mechanism that matters most to retrieve evidence before generating claims, the evidence that tests it, the versioned configuration needed to reproduce it, and the fallback when the evidence is weak. The chapter is not operationally complete while any field is blank.

9. Align modalities through tested representations

Multimodal systems align representations from different sensor and media spaces. Shared geometry is a hypothesis that must be evaluated against the intended cross-modal task.

Transformers for Vision

The Vision Transformer (ViT) applied the Transformer architecture to images by splitting an image into fixed-size patches (typically 16×16 pixels), treating each patch like a token. Each patch is linearly projected into an embedding, and these patch embeddings are processed through Transformer encoder blocks identically to how text tokens are processed.

The vision-transformer result showed that attention-based token processing can also operate on projected image patches.

Multimodal Embedding Models

CLIP: Connecting Text and Images

CLIP (Contrastive Language-Image Pre-training), introduced by Radford and peers, creates a shared embedding space for text and images.

CLIP is trained using contrastive learning (the same principle from word2vec in Chapter 2 and sentence-transformers in Chapter 10, applied across modalities): given a batch of image-text pairs, the model learns to maximise similarity between correct pairs and minimise it for incorrect pairs.

The trained dual encoder supports evaluated image-text retrieval, zero-shot classification against declared label prompts and representation analysis.

Evaluate the cross-modal relation, not the demo

A shared image-text space can support retrieval or candidate classification, but its geometry reflects the training data and objective. Build an evaluation set from the actual visual conditions, terminology and label descriptions expected in use. Include low resolution, cropping, text inside images, unusual composition, multilingual labels, sensitive attributes and examples that should remain unmatched.

Compare image-to-text and text-to-image retrieval separately because their error surfaces can differ. Prompt wording for zero-shot labels is part of the model configuration and must be versioned. Inspect false matches and missing matches by slice, then test whether the downstream user can recognise and correct them.

When a visual encoder feeds a generator, preserve the image transformation, encoder, bridge, prompt and generator as separate versioned interfaces. A plausible caption is not proof that the visual evidence was correctly represented. Require task-specific grounding checks, an abstention route for unreadable or unsupported content, and human authority for consequential interpretation.

Workbench check

Record the mechanism that matters most to align modalities through tested representations, the evidence that tests it, the versioned configuration needed to reproduce it, and the fallback when the evidence is weak. The chapter is not operationally complete while any field is blank.

Part IV: Adapt models with comparative evidence

Embedding training and model fine-tuning should preserve a reproducible base, declare what changes and compare candidates against independent evaluation evidence.

10. Train embeddings against the intended relation

An embedding model is trained against a declared relation: similarity, entailment, retrieval relevance or another task-specific notion. Negative sampling and evaluation must reflect that relation.

Embedding Models

Unstructured textual data must be converted to numeric representations for processing. An embedding model maps text to vectors under a learned objective; fitness means preserving the relation required by the downstream task.

Similarity is not singular. The desired geometry may represent semantic relation, entailment, duplication, relevance or another declared relation.

What Is Contrastive Learning?

Contrastive learning is a major technique for both training and fine-tuning text embedding models. It trains an embedding model such that similar documents are closer in vector space while dissimilar documents are further apart. This is very similar to the word2vec method from Chapter 2.

The contrasting procedure relates to context and is quite useful.

Contrastive learning depends on the relation declared by its positive and negative examples. A useful pair identifies both the intended similarity and the competing alternative the representation must separate.

Word2vec is an early neural example of learning from neighbouring and sampled non-neighbouring words. Later systems apply related contrastive objectives to sentences, documents and modalities.

SBERT

The sentence-transformers framework popularized contrastive learning within NLP. It fixed a major problem with the original BERT implementation for sentence embeddings: computational overhead.

Before sentence-transformers, sentence similarity used cross-encoders with BERT: two sentences passed to the Transformer simultaneously to predict similarity. However, finding the highest-scoring pair in 10,000 sentences requires n·(n-1)/2 = 49,995,000 inference computations, generating significant overhead.

Instead, sentence-transformers uses a Siamese architecture (also called a bi-encoder). Two identical BERT models share the same weights and architecture. Since weights are identical, a single model can process sentences one after another. The mean pooling layer averages the word embeddings and gives back a fixed dimensional output vector, ensuring a fixed-size embedding regardless of input length.

A bi-encoder creates reusable vectors, while a cross-encoder scores a supplied pair. Compare their ranking quality and latency on the target retrieval depth rather than assuming one is uniformly superior.

Train the relation the system actually needs

An embedding objective needs a declared relation. Semantic similarity, duplicate detection, question-to-answer relevance and product substitution are different relations even when they all use cosine similarity. Build positives from evidence that represents the chosen relation and negatives from alternatives the operating route must separate. Random negatives can teach broad separation; hard negatives test the local boundary where errors matter.

Split data by the dependency that could leak information, such as document family, author, case, customer, product or time. A random row split can place paraphrases or near duplicates on both sides and exaggerate generalisation. Deduplicate before splitting and preserve the method and threshold as part of the dataset manifest.

Compare the adapted representation with the unchanged base and a transparent lexical route. Report retrieval, ranking or classification measures at the operating depth, then inspect calibration, outliers and slices. Embedding visualisations are diagnostic projections, not evidence of cluster truth. Promotion requires a versioned index rebuild, query-index compatibility check, rollback artefact and human-use evidence on the downstream task.

Workbench check

Record the mechanism that matters most to train embeddings against the intended relation, the evidence that tests it, the versioned configuration needed to reproduce it, and the fallback when the evidence is weak. The chapter is not operationally complete while any field is blank.

11. Adapt encoders with bounded supervision

Encoder adaptation changes a representation for a bounded supervised task. Label quality, frozen layers, class imbalance and slice behaviour determine whether the improvement is real.

Adapt an encoder under a label contract

A label contract defines the observable evidence, labelling question, allowed classes, ambiguous cases, assessor guidance and adjudication route. Measure agreement before treating the labels as ground truth. If the task or policy changes, create a new label version rather than silently mixing regimes.

Full fine-tuning, partial freezing, adapters and a frozen-embedding classifier create different capacity and operating burdens. Compare them from the same pinned base on the same data split. Hyperparameters are experiment inputs, not universal defaults; preserve their search space, selection criterion and random seeds. Keep a final evaluation population outside tuning.

For token-level tasks, align labels after tokenisation and declare how continuation subwords, truncation and overlapping windows are handled. For few-label settings, repeated resampling and uncertainty intervals matter more than a single favourable split. Synthetic or model-generated labels need independent human verification and must remain distinguishable from observed labels.

Calibration is evaluated after model selection on a population that resembles use. The release route should define thresholds, abstention, human review, monitoring slices and rollback. An encoder is promotable only when its improvement survives dependency-aware splits and when downstream users can recognise and correct weak outputs.

Workbench check

Record the mechanism that matters most to adapt encoders with bounded supervision, the evidence that tests it, the versioned configuration needed to reproduce it, and the fallback when the evidence is weak. The chapter is not operationally complete while any field is blank.

12. Tune generators without losing the base contract

Generator tuning changes response behaviour without rewriting every parameter. Supervised data, low-rank adaptation, preference evidence and independent evaluation form one release argument.

Three adaptation stages: Pretraining, Supervised Fine-Tuning, and Preference Tuning

Three adaptation stages appear frequently in generator development:

Step 1: Language modelling (Pretraining). The first step pretrains the model on one or more large text datasets. This produces a base model (also called a pretrained or foundation model). Base models are key artefacts but are harder for end users to work with because they simply complete text rather than following instructions. This is why Step 2 is important.

Step 2: Supervised Fine-Tuning (SFT). LLMs are more useful if they respond well to instructions. With SFT, the base model is adapted to follow instructions. During this process, parameters are updated to be more in line with the target task. SFT can be used for classification, but is most often used to go from a base generative model to an instruction (or chat) generative model.

Step 3: Preference tuning. A preference objective changes response probabilities using comparative labels. Its claims remain bounded by the rubric, assessor population, sampled candidates and evaluation route.

Supervised Fine-Tuning (SFT)

SFT adapts the base model to follow instructions.

Full Fine-Tuning

A common process involves updating all parameters of a model to align with the target task. The main difference from pretraining is using a smaller but labelled dataset.

During full fine-tuning, the model takes instructions as input and applies next-token prediction on the response. The result: instead of generating new questions, the model follows instructions.

Parameter-Efficient Fine-Tuning (PEFT)

Updating all parameters is costly (slow training, significant storage). PEFT alternatives focus on fine-tuning at higher computational efficiency.

Evaluating Generative Models

Generative evaluation combines task-specific outcome measures, evidence support, structural validity, failure analysis and human-use evidence. No single lexical or model-based score is sufficient.

Word-Level Metrics: Perplexity (how “surprised” the model is by test data; lower is better), ROUGE (overlap between generated and reference summaries), BLEU (overlap for translation tasks), and BERTScore (semantic similarity using BERT embeddings rather than exact token matching).

Benchmarks: standardised test suites like MMLU (large Multitask Language Understanding) that evaluate models across dozens of tasks.

Automated Evaluation: Using a separately versioned model-based evaluator to rate the quality of generated text (model-based evaluator).

Preference Tuning / Alignment / RLHF

After SFT produces an instruction-following model, preference tuning further aligns it with human preferences. The original approach uses Reinforcement Learning from Human Feedback (RLHF).

Reward Models

A reward model learns to score LLM outputs based on human preferences. The reward model learns to assign higher scores to preferred responses.

Reward-model and policy-optimisation routes introduce additional estimation and stability questions. Direct preference objectives provide a different route, not an automatic improvement.

Adapt from a pinned base

Every adapted candidate begins with an immutable base-model and tokenizer identity. Preserve the training-data manifest, licences and rights decisions; preprocessing code; objective; random seeds; optimiser state; adapter configuration; numerical precision; hardware and library versions. An adapter without its compatible base and tokenizer is incomplete.

Low-rank and quantised adaptation can reduce trainable parameters or memory in a particular configuration. Those savings do not predict task quality. Compare full, parameter-efficient and no-adaptation routes using the same independent evaluation set and include training cost, serving cost, recovery and maintenance burden.

Treat preference data as measurement

Preference labels reflect a rubric, sampled outputs, assessor population and collection interface. Preserve all four. Measure agreement, position effects, verbosity preference, subgroup differences and ties before fitting a reward or direct-preference objective. Keep a held-out human-use evaluation independent of both training and model-based grading.

Preference optimisation changes the probability assigned to candidate responses under a chosen dataset and objective. It does not establish truth, safety or alignment in general. Release claims must therefore stay inside the tested population and human-owned operating route.

Workbench check

Record the mechanism that matters most to tune generators without losing the base contract, the evidence that tests it, the versioned configuration needed to reproduce it, and the fallback when the evidence is weak. The chapter is not operationally complete while any field is blank.

Merehaven field lab: evidence-bound complaint routing

The fictional Merehaven service team routes customer complaints to specialist queues. A representation model may classify a complaint when the score is calibrated and the input resembles the accepted population. A constrained generator may draft a short evidence-linked summary. Neither model may close the complaint, make a customer-outcome decision or invent missing account evidence.

Workbench contract

A route receipt

from dataclasses import dataclass
from typing import Literal

@dataclass(frozen=True)
class LanguageRouteReceipt:
    case_id: str
    evidence_ids: tuple[str, ...]
    tokenizer_version: str
    representation_version: str
    prompt_version: str
    output_schema: str
    disposition: Literal["usable", "review", "abstain"]

def may_enter_workbench(receipt: LanguageRouteReceipt) -> bool:
    return receipt.disposition == "usable" and bool(receipt.evidence_ids)

The receipt records reproducibility and evidence fitness. It does not grant authority to the model.

Evaluation matrix

Release ladder

1. Reproduce tokenisation, embeddings and generations from a clean pinned environment.
2. Compare representation, generative and composed routes against transparent baselines.
3. Test multilingual text, rare terms, long inputs, missing evidence and unusual formatting.
4. Measure calibration, retrieval, citation support, schema validity and task outcome separately.
5. Exercise timeout, stale index, weak retrieval, invalid output and abstention routes.
6. Run blinded human-use tests and preserve override reasons.
7. Release behind monitored review with version read-back and rollback.

The useful system is the complete route from authorised evidence to a human-owned decision, not the fluency of one generated response.

Acknowledgements and source note

This edition was developed from a protected study guide derived from Jay Alammar and Maarten Grootendorst’s Hands-On Large Language Models. That lineage is credited here rather than presented as original authorship. Repetitive study aids, duplicated sections, time-sensitive commercial data, model rankings, vendor recommendations and invented banking performance figures were removed or rebuilt. No attempt was made to reproduce either living author’s voice. The supplied source remains byte-identical and has a read-only protected copy.

The publication’s contribution is the workbench framing: three model routes, versioned tokenizer and representation contracts, prompts as executable programmes, retrieval as an evidence route, adaptation receipts, verification, abstention and human authority. Merehaven and every operational result attributed to it are fictional.

Selected foundational references

• Tomas Mikolov and peers, “Efficient Estimation of Word Representations in Vector Space”, 2013.
• Ashish Vaswani and peers, “Attention Is All You Need”, 2017.
• Jacob Devlin and peers, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, 2018.
• Nils Reimers and Iryna Gurevych, “Sentence-BERT”, 2019.
• Patrick Lewis and peers, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, 2020.
• Alec Radford and peers, “Learning Transferable Visual Models From Natural Language Supervision”, 2021.
• Edward Hu and peers, “LoRA: Low-Rank Adaptation of Large Language Models”, 2021.
• Tim Dettmers and peers, “QLoRA: Efficient Finetuning of Quantized LLMs”, 2023.
• Rafael Rafailov and peers, “Direct Preference Optimization”, 2023.
• Jay Alammar and Maarten Grootendorst, Hands-On Large Language Models, source lineage acknowledged.