TLDR

• The agent has no authority by default. A model may propose a tool call, but code, policy and a named owner decide whether that proposal can become an effect.
• LangChain composes transformations, LangGraph carries state across changing routes, and MCP standardises the protocol boundary. They solve different problems and should remain conceptually separate.
• Retrieval is an evidence route, not a truth machine. Chunking, query transformation, store selection, citations and abstention shape what the model is permitted to claim.
• Multi-agent design is an organisational choice. Add a specialist only when its context, tools, evaluation and failure ownership are genuinely distinct.
• A publishable system proves its boundaries. Trace proposal, permission, tool execution, receipt, readback and recovery with versioned evidence.

Reader and route

This edition is for engineers, architects, product leaders and control practitioners building AI applications with LangChain, LangGraph and MCP. Parts I and II establish composition and retrieval. Parts III and IV add state, agency and protocol. Part V turns the stack into an operating system with explicit authority and recovery.

Evidence boundary

Framework APIs, package names, model identifiers and code fragments are version-pinned learning specimens. Revalidate them against the selected runtime and official documentation before use. Merehaven Bank is wholly fictional; every customer, control, metric, incident and route is synthetic.

The duplicate-payment question

A customer asks an assistant to cancel a scheduled payment. The assistant retrieves the mandate, finds a policy note and calls a tool. The call times out. The model now faces a seductive fiction: it can write a confident answer even though nobody knows whether the cancellation occurred.

Composition does not resolve that uncertainty. Memory does not resolve it. A protocol does not resolve it. The system needs an idempotent action contract, a receipt or independent readback, and an authority rule for the unknown state. The difficult engineering begins where fluent text stops being enough.

Three boundaries, one system

Use a chain when the route is fixed, a graph when the route or state can change, and a protocol when tools must cross a process or organisational boundary. None of these grants permission. Permission remains an application and governance decision, expressed outside the model and recorded as evidence.

Part I: Compose without confusion

Start with the smallest useful abstraction. A fixed transformation deserves a chain; branching and persistence deserve a graph; remote capability deserves a protocol boundary.

Name the species before choosing the stack

What exactly are we building? the three species of LLM application

Imagine you run a restaurant. You need three kinds of workers.

The first is a prep cook. You hand them a bag of onions and say, “Dice these.” They dice. You hand them carrots and say, “Julienne.” They julienne. They never ask questions, they never improvise, and when the bag is empty, they stop. They do one thing, they do it well, and they have no memory of what they did yesterday or what they will do tomorrow. In the world of LLM applications, this is an engine.

The second worker is a front-of-house server. They greet the customer, take an order, remember that Table 4 has a nut allergy, and adjust their recommendations accordingly. If the customer says “What did I order last time?”, the server checks their notes. If the customer says “Actually, make that medium-rare instead,” the server updates the order without starting over. The server maintains context across the conversation. This is a chatbot.

The third worker is your general manager. When a party of 30 calls for a reservation on a Friday night, the general manager does not follow a script. They check the seating chart, call the kitchen to confirm capacity, negotiate a prix fixe menu, coordinate with the bar for cocktail service, email the customer a confirmation, and update the reservation system. At each step, they decide what to do next based on what just happened. If the kitchen says they cannot handle 30, the general manager proposes Saturday instead. If the customer insists on Friday, the general manager calls a catering backup. This is an AI agent.

The Engine: Stateless, Bounded, Done

An engine is a stateless function call applied to language. Input goes in. Output comes out. The engine remembers nothing, decides nothing, and stops when it finishes. In software architecture terms, it is a microservice. A summarization engine sits behind a REST API, receives a document payload, processes it through an LLM, returns a summary, and knows nothing about who called it or what happens next.

Here is a concrete example. A financial services firm needs to summarize earnings call transcripts. Each transcript is 15,000 words. The engine receives the transcript, splits it into chunks (because 15,000 words exceeds the LLM’s context window), summarizes each chunk independently, combines those summaries, and returns a final 500-word summary. The firm processes 200 transcripts per quarter. Each run is independent. The engine does not know that Apple’s Q1 transcript came before its Q2 transcript, and it does not care.

Another common engine type is the Question & Answer engine, which works in two phases. First, ingestion: the engine pulls in text from documents, splits the text into chunks, converts those chunks into numerical vectors called embeddings (we will spend a long time on embeddings later; for now, think of them as GPS coordinates that capture meaning instead of geography), and stores both the chunks and their embeddings in a specialized database called a vector store. Second, query: a user asks a question, the engine converts the question into an embedding using the same model, searches the vector store for the most similar chunks, and feeds those chunks plus the question to the LLM, which generates an answer grounded in the retrieved content.

This two-phase architecture, ingestion then query, is the skeleton of every RAG system in the edition. The route can it later in the edition, where we build it from scratch with nothing but OpenAI and ChromaDB. The route can it again later in the edition, where we rebuild it with LangChain’s components. And The route can increasingly sophisticated versions of it in Chapters 8, 9, and 10, where we optimize every layer: multiple embeddings per chunk for better search precision, query rewrites for better question understanding, and multi-store routing for directing questions to the right data source.

The key property of an engine is predictability. You know exactly what it will do because you wrote the pipeline. The execution path is fixed. Testing is straightforward: you can write deterministic test cases for each step. Debugging is manageable: the execution path is known. This predictability is the engine’s strength and its limitation. When the task requires branching based on intermediate results, the engine cannot adapt.

The Thought Experiment: Is It Really an Engine?

Consider these four application descriptions. Before reading the answers, classify each as an Engine, Chatbot, or Agent:

1. A service that receives PDF documents via API, extracts key clauses, and returns them as structured JSON.
2. A Slack bot that answers questions about company HR policies, maintaining conversation context across messages.
3. A system that receives a natural language description of a data analysis task, queries multiple databases, generates visualizations, writes a summary report, and emails it to stakeholders.
4. A decision-time customer support system that can look up order status, process returns, and escalate to human agents when needed.

Application 1 is an engine: stateless, bounded, one input, one output. Application 2 is a chatbot: stateful (conversation context), interactive. Application 3 is an agent: multi-step, tool-using (databases, visualization tools, email), decision-making. Application 4 is tricky, it is all three: it starts as a chatbot (conversation), escalates to an agent (tool use), and invokes engines (summarizing the conversation for the human supervisor). This blending of categories in real systems is the norm, not the exception.

The Chatbot: Conversation as Context

The chatbot adds one capability the engine lacks: memory. It remembers what was said before. This sounds simple. It is not.

Consider a travel chatbot helping a user plan a trip to Cornwall. The user says, “What are the best beaches?” The chatbot responds with a list. The user says, “What about the weather there?” The word “there” refers to Cornwall, which was established two turns ago. Without memory, the chatbot has no idea what “there” means. With memory, it resolves the reference and answers about Cornwall’s weather.

Now the user says, “Can you make it shorter?” “It” refers to the beach description from three turns ago. The chatbot needs to retrieve more than the previous turn but the specific content being referenced. And if the conversation runs for 50 turns, the chatbot needs to decide which prior context is still relevant and which can be dropped, because every token of context costs money and consumes limited space in the LLM’s context window.

Think of the context window as a desk. The LLM can only work with what is physically on the desk at any given moment. Anything not on the desk does not exist for the model. Early desks were tiny: GPT-3.5 could hold about 16,000 tokens, roughly 12,000 words, maybe 30 pages of text. Modern desks are enormous: the pinned capable model and a pinned model, enough for a small book.

But even a million-token desk fills up eventually, and every token on the desk costs money. Production chatbots use strategies to manage this: sliding windows that keep only the last N turns, summarization of older messages into compact context blobs, semantic retrieval over conversation history where only relevant past turns are injected, or external memory stores like LangGraph checkpoints (the relevant section) or Redis-backed stores. The choice depends on expected conversation length, the importance of early context, and budget.

The important difference between a chatbot and an engine is interactivity. A chatbot does more than produce output; it collaborates. The user refines, redirects, and builds on previous exchanges. This makes the chatbot more useful for tasks requiring back-and-forth: summarization refinement (“make it shorter,” “focus on the financial parts”), Q&A with follow-ups (“what about the price?” after discussing a hotel), and collaborative creation (“add a section on sustainability” to a report draft).

But interactivity introduces new failure modes. Context drift: accumulated conversation history subtly shifts the chatbot’s behaviour as older context weighs on newer responses. Context window overflow: old but important information gets pushed out as new messages accumulate. Memory inconsistency: the chatbot contradicts something it said earlier because the relevant turn has been summarized away. These are production bugs, not theoretical concerns. Every production chatbot team encounters at least one of them within the first month.

The Agent: The Machine That Chooses

And now we arrive at the concept that drives the entire book. An AI agent is a system that uses an LLM to choose actions, plan multi-step work, and adapt based on intermediate results. Unlike an engine, the agent does not follow a script. Unlike a chatbot, the agent does more than talk. The agent acts.

Here is the difference in one sentence: engines run workflows; agents manage workflows. In an engine, the developer decides the execution path at design time. In an agent, the LLM decides the execution path at runtime. This trades predictability for flexibility, and that tradeoff is the central tension of the entire book.

Infante illustrates with a tour operator agent that generates holiday packages from natural language requests. A booking website sends: “Family of four, two kids under 10, beach holiday in Cornwall, budget £3,000, first week of August.” The agent must:

1. Research attractions suitable for young children by querying a travel database
2. Find family-friendly accommodations by searching availability APIs
3. Check weather forecasts for the requested dates
4. Calculate total costs including transport, lodging, and activities
5. Generate a formatted itinerary with alternatives

At each step, the agent consults the LLM to decide what to do next. If the accommodation search returns nothing under budget, the agent does not crash. It adjusts: searches for a different area, suggests a different week, or proposes camping instead of a hotel. This adaptive loop, observe, decide, act, observe again, is the ReAct pattern (Reasoning and Acting), and it is the heartbeat of every agent in Chapters 11 through 14.

In high-stakes domains such as finance or healthcare, it is common to include a human-in-the-loop step. The agent proposes an action, a human reviews and approves or modifies it, and only then does the agent execute. In the holiday planning example, the agent could pause and request human approval of the proposed itinerary before sending it to the client. the relevant section covers human-in-the-loop in the context of LangGraph checkpoints, where the graph literally pauses at a checkpoint and waits for human input before resuming.

Here is the production reality that Infante emphasizes: these categories blend in real systems. A modern customer support application starts as a chatbot (handling the conversation), escalates to an agent when tool use is needed (looking up order status, initiating a return), and invokes engines for specific subtasks (summarizing the conversation for a human supervisor). The taxonomy is a design vocabulary, not a rigid classification. The value is in knowing which pattern to apply at each point in your system.

The table above deserves careful study because it encodes the reliability-flexibility tradeoff at the center of every architectural decision in this edition. Moving rightward across the columns, you gain power and lose predictability. The art of production AI engineering is knowing how far right you need to go and no further.

The Production Architecture Insight

This engine-vs-agent distinction maps directly to a reliability-flexibility tradeoff in operating systems. Engines are more predictable, easier to test (you can write deterministic test cases for each chain step), and simpler to debug (the execution path is known). Agents are more capable but harder to control (the LLM might choose unexpected tools or loop indefinitely).

A common production pattern is to use agents for routing and high-level planning but delegate actual execution to deterministic engine-style chains. This gives you the flexibility of agents with the reliability of engines. The trip planner agent decides to check weather, but the weather check itself is an engine: call API, parse response, return result. No LLM decision-making needed for the tool execution. This hybrid approach is exactly what the edition builds toward in Chapters 11 through 14.

The RAG pattern: teaching machines to look things up

Here is a question that seems simple but is actually profound: How does a language model know anything?

The answer is: it does not. Not in the way you know things. You have episodic memory, you remember your first day of school. You have semantic memory, you know that Paris is the capital of France. You have procedural memory, you know how to ride a bicycle. A language model has none of these. What it has is statistical patterns learned from training data. When it tells you that Paris is the capital of France, it is not recalling a fact from memory. It is generating the most statistically likely continuation of the text “The capital of France is…”.

This distinction matters enormously because it explains the model’s two fundamental limitations. First, knowledge cutoff: the model only knows what was in its training data. If your company’s product catalog was updated yesterday, the model knows nothing about the update. Second, hallucination: when the model does not know the answer, it does not say “I don’t know.” It generates a statistically plausible answer that sounds authoritative and is completely wrong.

A hallucination occurs when an LLM generates an incorrect, misleading, or fabricated response. Due to their auto-regressive nature (they predict one token at a time based on previous tokens), LLMs try to generate a response even when relevant content is missing. They fill gaps with plausible-sounding but incorrect information. The confident tone makes hallucinations especially dangerous: there is no difference in how the model sounds when it is right versus when it is fabricating.

In the spring of 2023, an enterprise chatbot built on GPT-4 told a customer that their warranty covered accidental damage, free of charge, for five years. The company’s actual warranty was one year, defects only, no accident coverage. The chatbot had hallucinated a warranty policy more generous than any the company had ever offered. The customer was delighted. The legal team was not.

This is the problem that Retrieval-Augmented Generation, or RAG, was designed to solve.

The Open-Book Exam Analogy

Think of RAG as the difference between a closed-book exam and an open-book exam. A vanilla LLM takes a closed-book exam. It can only draw on what it memorized during training. That knowledge might be outdated, incomplete, or directly wrong for your specific domain. A RAG system takes an open-book exam. Before answering, it looks up the relevant page in a reference document. It still needs understanding to compose a good answer (you cannot pass an open-book exam just by having the edition), but the answer is grounded in actual source material rather than statistical hallucination.

You do RAG every time you answer a pub quiz question. You hear the question (the query). You scan your memory for relevant facts (retrieval). You compose an answer that combines what you retrieved with your general knowledge (generation). The only difference is that an LLM’s “memory” is a vector database, and its “scanning” is a cosine similarity search.

How RAG Works: The Two-Phase Architecture

Phase 1: Content Ingestion. You take your source documents and process them through a pipeline:

1. Load: Extract raw text from the source using a document loader
2. Split: Break the text into manageable chunks using a text splitter
3. Embed: Convert each chunk into a numerical vector using an embedding model
4. Store: Save the chunks and their vectors in a vector store

Think of this phase as building a library. But instead of shelving books alphabetically by author (which is how traditional databases work, by exact matches on fields), you shelve books by meaning. A book about grief sits next to a book about loss, even if one is a novel and the other is a psychology textbook. A travel guide to Cornwall sits next to a blog post about surfing in Newquay, because they are about similar things even though they have different titles and different authors.

This is what embeddings do. They are coordinates in a vast meaning-space where proximity equals semantic similarity. The sentence “Activities you can enjoy in Cornwall” and the sentence “Visitors can walk among the ruins and visit the on-site museum” end up near each other in embedding space, even though they share almost no words, because an embedding model trained on billions of sentences has learned that activities, walking, and visiting are semantically related concepts.

Phase 2: Question Answering. When a user asks a question:

1. Embed the question: Convert it into a vector using the same embedding model (this is critical; using a different model produces vectors in a different mathematical space, making similarity search meaningless)
2. Search: Find the chunks whose vectors are closest to the question vector
3. Augment: Combine the retrieved chunks with the original question into a prompt
4. Generate: Send the augmented prompt to the LLM, which produces an answer grounded in the retrieved context

RAG offers three key benefits that make it the default architecture for most LLM applications:

Grounding an LLM involves crafting prompts that include context pulled from a trusted knowledge source. This ensures the LLM generates its response based on verified facts rather than relying solely on pretrained knowledge.

The Failure Mode That Humbles Everyone

RAG has a failure mode that trips up nearly every team on their first deployment: the chunking problem.

If your documents are split into chunks that break mid-sentence or mid-paragraph, the retrieved chunk might contain half a thought. The LLM receives a fragment and confabulates the rest. The fix is not in the model or the retrieval; it is in how you split your documents. Chunk boundaries must respect semantic units: paragraphs, sections, or sliding windows with overlap. This sounds simple. In practice, it is the number one cause of bad RAG answers, ahead of bad embeddings, bad prompts, and bad models.

the relevant section builds RAG from scratch so you understand every moving part. Chapters 8, 9, and 10 then spend three full chapters fixing the things that go wrong: advanced indexing strategies for better chunk representation (the relevant section), query transformations for better question understanding (the relevant section), and multi-store routing for directing questions to the right data source (the relevant section). This progression from “RAG that works in demos” to “RAG that works in an operating environment” is one of the distinguishing features of Infante’s book.

The three techniques for adapting llms

You have a general-purpose language model. It knows about everything and nothing about your specific business. How do you make it useful for your domain? There are three techniques, arranged from lightest to heaviest, and understanding when to use each is a critical production skill.

Technique 1: Prompt Engineering (Free, Immediate, capable)

Prompt engineering is the practice of designing inputs so that the model understands the task and produces useful, accurate responses. It is the single highest-leverage skill for LLM application developers because a well-crafted prompt can extract expert-level performance from a model that would otherwise produce mediocre results, with zero additional training cost.

A common technique is in-context learning, where the model infers patterns from examples embedded directly in the prompt. The model has never been fine-tuned on your specific task, but by seeing three to five examples, it deduces the pattern and applies it to new inputs. Few-shot prompting provides these examples directly in the prompt. Chain of Thought adds step-by-step reasoning to the examples, enabling the model to handle complex, multi-step logic.

Prompts are often organized as templates: a fixed instruction with variable fields that accept dynamic input. LangChain’s PromptTemplate and FewShotPromptTemplate make templates reusable, testable, and composable. the relevant section covers prompt engineering in depth; it is the most referenced chapter in the entire book because every subsequent chapter’s code quality depends directly on the prompt quality.

Technique 2: RAG (Medium Effort, High Impact)

We just covered this. RAG gives the model access to external knowledge without retraining. It is the sweet spot for most production applications because it combines the model’s language ability with your specific data, can be updated in decision-time (just add new documents to the vector store), and costs a fraction of fine-tuning. Five full chapters (6 through 10) are devoted to RAG because it is the most impactful technique for most worked applications.

Technique 3: Fine-Tuning (Heavy Investment, Specialized Use Cases)

Fine-tuning adapts a pretrained LLM to perform better in a specific domain by training it on a curated dataset. Think of it as a classically trained pianist learning jazz. The fundamental technique (finger positioning, hand independence, reading music) stays, but the musical instincts shift. The pianist’s muscle memory adapts to jazz voicings, swing rhythms, and improvisation patterns. LoRA (Low-Rank Adaptation) is like learning jazz by adjusting only your right hand while keeping your left hand’s classical technique unchanged, a parameter-efficient approximation that materially reduces cost.

Recent advances in LoRA and other parameter-efficient methods have lowered both cost and complexity, but fine-tuning is still the heavyweight option. Preparing high-quality datasets takes time and expertise. Training runs require GPUs. The model becomes specialized, which can hurt general-purpose performance.

Research by Soudani et al. shows that RAG often outperforms fine-tuning by providing context dynamically at runtime, reducing both costs and retraining needs. However, in highly specialized domains where the model needs to learn new reasoning patterns (more than access new facts), fine-tuning remains invaluable. Domain-specific examples include BioMistral (biology), LexiGPT (legal), BloombergGPT (finance), and code-focused models.

The model selection triangle: cost, accuracy, and speed

Choosing which LLM to use for each part of your system is a deceptively important decision. Infante introduces a framework that every production team should internalize: the cost-accuracy-speed triangle.

These three factors exist in tension. Larger models are generally more accurate but slower and more expensive. Smaller models respond faster and cost less but sacrifice capability. No single model optimizes all three dimensions simultaneously.

The professional approach is to use different models for different tasks within the same system. A customer support chatbot might favor speed and cost for simple FAQ responses but escalate to a premium model for complex multi-step reasoning. LangChain’s abstraction layer makes this trivial: each component in your pipeline can use a different model.

Imagine you are building a travel platform with four LLM-powered services:

# Each service uses the model that matches its priority
classifier = ChatOpenAI(model="the pinned low-cost model", temperature=0)  # Fast, cheap
generator = ChatOpenAI(model="the pinned capable model", temperature=0.7)       # Premium

# Both plug into the same LCEL pipeline
classify_chain = classify_prompt | classifier | parser
generate_chain = generate_prompt | generator | parser

Beyond the core tradeoffs, additional considerations include: model purpose (general vs. code-tuned), context window size (128K to 1M+ tokens), multilingual support, instruction vs. reasoning models, and open source vs. proprietary. The distinction between proprietary and open source is blurring rapidly. Meta’s Llama 3.3 achieves quality comparable to GPT-4o on many benchmarks. Mistral, Qwen, and DeepSeek regularly challenge frontier model quality.

The protocol that changed everything: MCP

In late 2024, Anthropic released the Model Context Protocol, or MCP, and the AI development world shifted. To understand why, you need to understand the problem it solved.

Before MCP, every integration was bespoke. If your agent needed to check the weather, you wrote a weather API integration. If it needed to search a database, you wrote a database integration. If it needed to send email, you wrote an email integration. Every agent developer wrote the same integrations. Every integration had the same bugs. And when the weather API changed its endpoints, every agent that used it broke independently.

This is the classic N times M problem. If you have N agents and M tools, you need N times M integrations. Add one new agent, you write M integrations. Add one new tool, you update N agents. The complexity grows as a product, not a sum.

MCP solves this by standardizing the interface. Think of it as USB for AI tools. Before USB, every device had its own proprietary connector: a different cable for your printer, scanner, keyboard, and mouse. USB gave everyone the same port. Suddenly, any device worked with any computer. MCP does the same for AI agents. A tool provider writes one MCP server that exposes their capability. An agent developer writes one MCP client that consumes any MCP server.

The market is bifurcating into three roles: Tool builders who create MCP servers exposing domain expertise, Agent builders who compose agents from MCP tools plus local capabilities, and Platform builders who build the infrastructure connecting tool builders with agent builders. this edition primarily prepares you for the agent builder role (Chapters 1-14) with significant overlap into tool building (the relevant section).

The convergence of RAG (knowledge access), Agents (autonomous decision-making), and MCP (external capability access) produces what the edition builds toward across all fourteen chapters: an autonomous knowledge worker that can access any information source, decide what actions to take, execute those actions through standardised tools, and operate safely within guardrails.

The RAG debugging intuition you need now

You will not build a RAG system until the relevant section, but you need the debugging intuition now because it shapes how you think about every architectural decision.

When a RAG system gives a wrong answer, there are exactly four possible failure points, and you must check them in order:

1. Is the content in the vector store? If the relevant document was never ingested, or if ingestion failed silently, no amount of retrieval optimisation will help. Always verify that your vector store contains the expected content before blaming the retriever.

2. Does the query retrieve relevant chunks? Test retrieval independently from generation. If the top-k results are irrelevant, the problem is either the embedding model (it does not understand your domain’s vocabulary), the chunk boundaries (semantic units are split), or the query itself (the user’s phrasing does not match the document’s vocabulary; the relevant section fixes this with query transformations).

3. Does the LLM use the retrieved context? Sometimes the retrieval is perfect, the relevant chunks are right there in the prompt, but the LLM ignores them and answers from training data instead. This usually means the prompt does not strongly enough instruct the LLM to use only the provided context.

4. Does the prompt prevent hallucination? Test with questions whose answers are NOT in the context. If the LLM invents answers instead of saying “I don’t know,” strengthen the anti-hallucination instructions.

The order matters. If you start debugging at step 4 when the problem is at step 1, you will waste days optimizing prompts for a system that directly does not have the right documents.

This debugging framework, combined with LangSmith traces (the relevant section), enables systematic diagnosis of any RAG quality issue. Commit it to memory now; The route can it constantly later in the edition onward.

The glossary: terms worth memorizing

Every technical term introduced in this section, precisely defined as used in this edition:

Treat prompts as typed interfaces

The machine under the hood: how prompts become completions

Before we learn to write prompts, we need to understand how they are executed programmatically, because in an operating environment, no human types prompts into a chat box. Prompts are assembled from templates, variables, retrieved context, and configuration, then dispatched to the LLM via API calls.

The Raw API Call

The most fundamental operation is sending a prompt to OpenAI’s chat.completions.create method:

from openai import OpenAI
import getpass

OPENAI_API_KEY = getpass.getpass('Enter your OPENAI_API_KEY')
client = OpenAI(api_key=OPENAI_API_KEY)

prompt_input = """Write a concise message to remind 
users to be vigilant about phishing attacks."""

response = client.chat.completions.create(
    model="the pinned low-cost model",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": prompt_input}
    ]
)

The getpass module prevents accidental exposure of credentials. If you hardcode a key and commit the notebook to Git, anyone with repository access can use your API key and run up charges. in an operating environment, use environment variables, a .env file with python-dotenv, or a dedicated secrets manager.

What the Response Object Reveals

The raw response contains metadata that every LLM developer must understand:

ChatCompletion(
    id='chatcmpl-CFkN43Xs80ohhDJVIDeRUSID8RXNo',
    choices=[Choice(
        finish_reason='stop',
        message=ChatCompletionMessage(
            content='Be vigilant against phishing: verify the sender...',
            role='assistant',
            refusal=None
        )
    )],
    model='the pinned low-cost model-2025-08-07',
    usage=CompletionUsage(
        completion_tokens=553,
        prompt_tokens=32,
        total_tokens=585,
        completion_tokens_details=CompletionTokensDetails(
            reasoning_tokens=512
        )
    )
)

Every field in this response has production implications:

finish_reason='stop' means the model completed normally. If it says 'length', the model ran out of max_tokens and was cut off mid-sentence, which means your answer is truncated. If it says 'content_filter', a content policy was triggered. in an operating environment, check this field on every response and handle non-stop cases explicitly.

completion_tokens=553 and prompt_tokens=32 tell you exactly how many tokens were consumed. OpenAI charges per token, so total_tokens=585 is your billing unit. Log this for every API call in an operating environment to track costs.

reasoning_tokens=512 reveals that the model used internal chain-of-thought reasoning tokens. These are counted toward billing but not shown in the output. This is a the pinned capable model feature where the model “thinks” before responding, and it explains why the pinned capable model’s completion token counts are often surprisingly high.

refusal=None indicates the model did not refuse the request. When non-null, the model declined due to content policy, and you need to inspect and adjust the prompt.

To access the actual content:

answer = response.choices[0].message.content
tokens_used = response.usage.total_tokens
cost_usd = tokens_used * 0.00000005  # the pinned low-cost model pricing
print(f"Answer: {answer}")
print(f"Cost: ${cost_usd:.6f}")

The choices is a list because the API supports generating multiple completions per request (via the n parameter). In practice, n=1 is almost always used, so response.choices[0] gives the single result.

The Role-Based Message Format

The messages parameter uses a role-based structure that has become the de facto standard for programmatic LLM interaction:

The system role is the most capable position in a prompt. It sets the ground rules for the entire conversation: persona, constraints, default behaviour. A system message saying “You are a helpful travel assistant specializing in Cornwall, England. Only answer questions related to travel. If asked about unrelated topics, politely decline.” shapes every subsequent response, acting as a persistent behavioral guardrail. This single system message is the simplest and most effective form of guardrailing, and it is the first thing to add when your application’s LLM gives off-topic or inappropriate responses.

LangChain Simplifies Everything

The same operation through LangChain’s ChatOpenAI wrapper:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(openai_api_key=OPENAI_API_KEY, model_name="the pinned low-cost model")
response = llm.invoke("Write a phishing awareness message.")
print(response.content)

Three lines versus eight. LangChain handles message formatting, role assignment, and response extraction internally. Token usage is available via response.response_metadata. The real power emerges when you compose: prompt | llm | parser in LCEL is impossible with the raw OpenAI client.

The eight building blocks of every prompt

Every prompt is assembled from eight possible components. Not every prompt needs all eight. But knowing the full toolkit lets you diagnose and fix failing prompts systematically, the way a mechanic diagnoses an engine by checking fuel, air, spark, and compression rather than randomly replacing parts.

1. Persona tells the model who it is. “You are a senior financial analyst” produces different language than “You are a friendly travel guide.” The persona shapes vocabulary, depth, formality, and risk tolerance.

2. Context provides background the model needs but does not have. “The company operates in renewable energy and recently completed a $2B acquisition.”

3. Instruction is what you want done. “Summarize,” “Classify,” “Extract.” This is the verb of the prompt.

4. Input is the data to process. The document to summarize. The text to classify.

5. Steps decompose complex tasks. “First identify all palindromes. Then calculate their sum.” Explicit step decomposition is one of the most capable prompt improvements.

6. Tone specifies communication style. “Professional,” “casual,” “empathetic.”

7. Output Format defines structure. “Return as JSON with keys: name, category, confidence.” This is not optional in an operating environment. Unspecified format dimensions produce unpredictable output that breaks downstream parsing.

8. Examples teach by demonstration. This is where few-shot learning and chain-of-thought live.

Infante demonstrates all eight components in a fully specified prompt:

<Persona>
You're an experienced LLM developer and renowned speaker.
</Persona>

<Context>
You've been invited to keynote an LLM event.
</Context>

<Instruction>
Write the punch lines for the speech.
</Instruction>

<Input>
Include these facts:
- LLMs became mainstream with ChatGPT in November 2022
- Many popular LLMs have launched since then
- LLMs becoming as popular as search engines
- Many companies want to integrate LLMs in their applications
</Input>

<Tone>
Witty but entertaining.
</Tone>

<OutputFormat>
Two paragraphs of 5 lines each.
</OutputFormat>

The XML-style tags are not decorative. Studies, including the Prompt Report survey from 2024 (https://arxiv.org/abs/2406.06608), found that explicitly naming prompt sections improves model performance. The model can parse the structure more reliably than unstructured prose. Both OpenAI and Anthropic recommend this pattern.

The hierarchy of prompt improvements by impact-per-effort, which serves as a troubleshooting checklist:

1. Add output format specification (biggest impact, easiest). “Return as JSON” or “Use 3 sentences maximum.”
2. Add persona (second biggest). “You are a senior financial analyst” shapes everything.
3. Add examples (few-shot). 3-5 examples calibrate the output pattern.
4. Add reasoning steps (CoT). Explicit decomposition for complex logic.
5. Add defensive instructions. “If unsure, say I don’t know.”
6. Fine-tune the model (most effort, diminishing returns). Last resort.

Most production prompt quality issues are solved by steps 1 through 3. CoT is needed for reasoning-heavy tasks. Defensive instructions are needed for user-facing applications. Fine-tuning is rarely necessary with modern frontier models.

Teaching by example: from zero-shot to chain of thought

The most important dimension of prompt engineering is how many examples you provide and whether those examples include reasoning.

Zero-Shot: When the Model Already Knows

A zero-shot prompt provides no examples. It relies entirely on the model’s pre-existing knowledge. For well-understood tasks, this works:

Classify this review as POSITIVE, NEGATIVE, or NEUTRAL:
"The hotel room was clean but the restaurant was disappointing."

A good model returns “NEUTRAL” or “MIXED” without demonstration. Zero-shot is free in terms of tokens. But it only works for tasks the model already handles. The moment you need a custom rule, zero-shot fails silently, producing an answer that is wrong but confident.

The Six Prompt Types You Will Write Every Day

Before moving to few-shot, it is worth walking through each standard prompt type in detail, because they recur throughout the edition and each reveals a principle about prompt design that The route can constantly.

Text Classification: Where Five Words Changed Everything

In classification, the goal is to assign an input text to a predefined category. Infante demonstrates with a deceptively simple example that reveals one of the most important prompt engineering principles:

Without output specification:

Classify the following text into one of these categories:
history, tech, gardening.
Text: Headphones provide immersive audio experiences for music
lovers and gamers alike.

The model returns an overly detailed explanation: “The text should be classified as ‘tech’ because it discusses technology-related products and their functionality.” In a production pipeline where you need to route documents by category, you want a single word, not a paragraph of justification.

With output specification:

Classify the following text into one of these categories:
history, tech, gardening.
Text: Headphones provide immersive audio experiences for music
lovers and gamers alike.
Output only the category

Result: Tech

Five words, “Output only the category,” transformed the response from a paragraph to a single word. This principle, that output format instructions are as important as task instructions, applies to many prompt interfaces. in an operating environment, unspecified output format is a common cause of downstream parsing failures.

Sentiment Analysis: The Power of Batch Processing

Sentiment analysis classifies text as positive, neutral, or negative. Individual classification is straightforward. The production insight comes from batching:

Classify the sentiment of following stock reports as positive,
neutral or negative

Stock 1: Apple: the launch of the new iPhone has been a success
Stock 2: Nvidia: sales propelled by consumer demand on LLMs
Stock 3: GX oil: demand of carbon energy dropping due to renewables

Output: a table with columns "stock name", "sentiment"

Result:

| Stock Name | Sentiment |
|------------|-----------|
| Apple      | positive  |
| Nvidia     | positive  |
| GX oil     | negative  |

This batch-processing pattern is important for an operating deployment applications. Instead of one API call per text (incurring per-call overhead and latency), you batch multiple items into a single prompt. This can reduce costs by 5-10x for high-volume classification tasks. However, there are limits: the total tokens must fit within the context window, accuracy can degrade for items later in a long batch (the “lost in the middle” phenomenon), and if one item causes an error, the entire batch fails. Find the optimal batch size through experimentation, typically 5-20 items depending on item length.

Text Summarization: The Gateway to the relevant section

Creating a summarization prompt is the simplest prompt type: specify the text and the desired length. The key insight is that you can summarize both provided text and content the LLM knows from training. But summaries from training knowledge are less reliable than summaries of provided text, because the LLM might conflate details or hallucinate specifics. This is why RAG (providing the text explicitly in the prompt) is preferred in an operating environment, and why Chapters 6 through 10 are devoted to getting retrieval right.

Composing Text: The Dramatic Impact of Structure

LLMs can generate new content from a list of facts. Infante demonstrates the dramatic difference between an underspecified and a well-specified prompt with a diver watches article:

Underspecified prompt: Just “Write a piece on diver watches” with a list of facts. Result: verbose, overly formal, complex vocabulary the user did not request, rambling structure.

Well-specified prompt with persona, context, instruction, tone, output format, and facts:

<Persona>You are an experienced copywriter</Persona>
<Context>Writing for a general audience magazine</Context>
<Instruction>Write an engaging article about diver watches</Instruction>
<Input>[list of specific facts]</Input>
<Tone>Witty but entertaining</Tone>
<OutputFormat>Two paragraphs of 5 lines each</OutputFormat>

Result: tight, engaging, appropriately toned, correctly structured. Same model, same facts, materially different output. The difference is entirely in how the prompt was structured.

This comparison is worth memorizing because it demonstrates the ROI of prompt engineering in a single example. The underspecified prompt and the well-specified prompt cost the same in API calls. They take the same time to execute. The only difference is the ten minutes you spent thinking about what you wanted.

Question Answering: The Q&A Format Trick

LLMs understand Q: and A: as shorthand for question-answer pairs. Ending a prompt with A: primes the model to produce a concise, direct answer rather than a discursive explanation:

Text: Java is a popular programming language that compiles code
into bytecode, which is executed by the Java Virtual Machine (JVM)
for platform-independent application development.

Q: Where is Java code executed?
A:

Result: “Java code is executed by the Java Virtual Machine (JVM).”

This is remarkably concise compared to what you would get without the Q&A format. The A: acts as a format constraint, telling the model “give me the answer, not an essay about the answer.” This pattern appears in the RAG prompts of Chapters 6 and 7, where concise answers are essential.

Reasoning: Where Models Surprise and Disappoint

Reasoning is where LLMs simultaneously impress and frustrate. Infante demonstrates with two examples:

Square numbers (success): “Add the square numbers in this sequence: 19, 13, 1, 17, 4, 64, 900.” The model correctly identifies 1, 4, 64, and 900 as perfect squares and produces 969. It not only found the answer but showed the identification step.

Palindromes (failure): “Sum the palindromes in this sequence: 13, 1331, 121, 73, 99, 56, 232, 7.” The model produces 1691, missing 99. This failure motivates the entire next section on in-context learning techniques and is one of the most pedagogically valuable examples in the edition.

The Palindrome Debugging Journey: Why Steps Matter More Than Examples

This is the most instructive sequence in the chapter because it reveals a debugging methodology that transfers to many prompt interfaces. It is worth walking through every attempt in detail because each failure teaches a different lesson.

Infante asks the model to sum the palindromes in a sequence: 13, 1331, 121, 73, 99, 56, 232, 7. The correct answer is 1790 (the palindromes are 1331, 121, 99, 232, and 7; their sum is 1790).

Attempt 1 (Zero-shot): FAILED.

Sum the palindromes in this sequence: 13, 1331, 121, 73, 99, 56, 232, 7

The model returns: “The palindromes are 1331, 121, 232, and 7. Sum: 1331 + 121 + 232 + 7 = 1691.”

It missed 99. Why? The model’s concept of “palindrome” is biased toward multi-digit patterns with distinct digit variations (like 1331 or 121). A two-digit number with repeated digits (99) is technically a palindrome but does not match the model’s strongest associations with the concept. The model “knew” what palindromes were, just not comprehensively.

Lesson 1: Zero-shot works when the model’s training-data understanding aligns with your expectation. When there is a gap, zero-shot fails silently: it produces an answer, just the wrong one. This silent failure is the most dangerous property of LLM reasoning: there is no error message, no exception, no warning. Just a confident, wrong answer.

Attempt 2 (One-shot, added “33 is a palindrome”): FAILED.

Sum the palindromes in this sequence: 13, 1331, 121, 73, 99, 56, 232, 7
Example: 33 is a palindrome

The model apologized for the “oversight” but still returned 1691. One example of a two-digit palindrome was not enough to override the model’s incomplete concept.

Lesson 2: One-shot learning is unreliable for correcting conceptual gaps. A single example might be treated as an anomaly rather than a pattern. The model recognized 33 as a palindrome but did not generalize to “all two-digit repeated numbers are palindromes.”

Attempt 3 (Two-shot, added “44 is a palindrome”): FAILED.

Sum the palindromes in this sequence: 13, 1331, 121, 73, 99, 56, 232, 7
Examples: 33 is a palindrome. 44 is a palindrome.

Still 1691. Two examples of two-digit palindromes were still insufficient. The model recognized 33 and 44 as palindromes but did not generalize to 99.

Lesson 3: More examples of the same type do not necessarily fix the underlying issue. Two examples of “repeated-digit palindromes” did not teach the general rule. The model may have been treating these as specific instances rather than inducing the general pattern.

Attempt 4 (Two-shot plus explicit steps): SUCCESS!

Sum the palindromes in this sequence: 13, 1331, 121, 73, 99, 56, 232, 7
Examples: 33 is a palindrome. 44 is a palindrome.
Steps: 1) identify the palindromes; 2) add them up

The model responds:

Step 1: Palindromes in the sequence: 1331, 121, 99, 232, 7
Step 2: 1331 + 121 + 99 + 232 + 7 = 1790

The breakthrough. Adding “Steps: 1) identify the palindromes; 2) add them up” forced the model to decompose the problem. When trying to identify palindromes and sum them simultaneously (attempts 1 through 3), the model took a cognitive shortcut, jumping to the familiar palindromes and skipping the less obvious 99. When forced to list ALL palindromes first as a separate step, it could not skip 99 without the omission being conspicuous in the explicit list.

Lesson 4: When adding examples does not fix the problem, adding explicit reasoning steps often does. Step decomposition prevents the model from taking shortcuts. The steps make the intermediate work visible, which means errors become visible too. This is the core insight of Chain of Thought: show the work, more than the answer.

Here is Infante’s fascinating meta-observation: this palindrome issue was later fixed in newer model versions. If you enter the original zero-shot prompt in the pinned capable model, it now produces 1790 correctly, decomposing the problem step-by-step on its own using internal reasoning. Reasoning models like the pinned capable model Thinking decompose problems automatically using internal chain-of-thought, without being told to do so.

This is both encouraging and unsettling. The technique you spent an afternoon developing may become unnecessary with the next model release. But in complex cases with deterministic steps, especially in an operating environment where reliability trumps cleverness, explicitly spelling out steps remains valuable. It ensures the LLM follows the intended sequence rather than inventing its own, which could introduce errors. Design your prompt engineering to be modular so you can simplify as models improve, rather than accumulating permanent complexity.

Few-shot learning provides three to ten examples that teach the model a pattern it does not already know. This is one of the most capable techniques in the prompt engineer’s toolkit, and it works through a mechanism that borders on eerie.

Infante demonstrates with a game called “AbraKadabra.” The rules: numbers divisible by 5 are “Abra,” numbers divisible by 7 are “Kadabra,” numbers divisible by both are “Abra Kadabra.” He never states these rules explicitly. Instead, he provides five worked examples using a consistent // delimiter format:

6 // not divisible by 5 nor by 7 // None
15 // divisible by 5 but not by 7 // Abra
12 // not divisible by 5 nor by 7 // None
21 // not divisible by 5 but divisible by 7 // Kadabra
70 // divisible by 5 and by 7 // Abra Kadabra

Classify: 3, 4, 5, 7, 8, 10, 11, 13, 35

The model not only classifies every number correctly but also explains the deduced rules: “Based on the examples, ‘Abra’ means divisible by 5, ‘Kadabra’ means divisible by 7, and ‘Abra Kadabra’ means divisible by both.” It inferred the general pattern from five examples, articulated it, and applied it to nine new cases. No one told it the rules. It figured them out from the structure of the examples alone.

This is the power and the eeriness of few-shot learning. The model is a pattern-matching machine of extraordinary capability. Well-structured examples can teach arbitrary rules without explanation. in an operating environment few-shot prompts, two principles dominate:

Consistent formatting is more important than explicit documentation. The model reads the structure of your examples and mimics it. The // delimiter in AbraKadabra was never explained; the model deduced it from context. If your examples use // as a delimiter, the model will use // in its output. If you switch delimiters between examples, the model gets confused and output format becomes inconsistent.

Example selection matters more than example quantity. Five carefully chosen examples that cover different edge cases outperform twenty examples that all demonstrate the same path. For AbraKadabra, the examples cover: neither condition met (6, 12), only first condition (15), only second condition (21), and both conditions (70). Every possible outcome is demonstrated exactly once. This systematic coverage is what makes the model’s generalization reliable.

Important caveat: Infante deliberately did not use the classic FizzBuzz game (divisible by 3 = “Fizz,” divisible by 5 = “Buzz”) because ChatGPT already knows FizzBuzz from its training data. It would solve it zero-shot, making the few-shot examples a waste of tokens and money. Always check whether the model already knows what you are trying to teach it before investing in few-shot examples. This is more than a cost optimisation; providing redundant examples can actually confuse the model if your examples contradict its pre-existing knowledge.

Chain of Thought (CoT) blends few-shot examples with explicit reasoning. For each example, the prompt shows more than input and output but the intermediate reasoning steps. It is the most capable prompting technique for tasks requiring multi-step logic.

Infante creates a “strange sequence” game to demonstrate. A sequence is “strange” if it meets two conditions: (a) it contains at least two odd numbers, AND (b) the sum of all odd numbers is divisible by 3. This is a compound rule that requires multiple logical steps, the kind of rule that trips up models without CoT.

He first verifies the model does not already know the concept by asking without examples. It does not. Then he provides four carefully designed worked examples:

Q: Is the following sequence strange: 1, 4, 6, 8, 20
A: 1 is an odd number; I need at least two odd numbers // Not Strange

Q: Is the following sequence strange: 5, 6, 7, 8, 20
A: 5 and 7 are odd numbers; the sum of 5 and 7 is 12;
   12 is divisible by 3 // Strange

Q: Is the following sequence strange: 1, 5, 6, 7, 8, 20
A: 1, 5 and 7 are odd numbers; the sum of 1, 5 and 7 is 13;
   13 is not divisible by 3 // Not Strange

Q: Is the following sequence strange: 5, 6, 7, 8, 9, 20
A: 5, 7, 9 are odd numbers; the sum of 5, 7 and 9 is 21;
   21 is divisible by 3 // Strange

When asked about the test sequence 3, 4, 5, 7, 10, 18, 22, 24, the model correctly reasons: “3, 5, and 7 are odd numbers; the sum of 3, 5, and 7 is 15; 15 is divisible by 3 // Strange.” It not only gets the right answer but shows every step of its work, proving it internalized the multi-step logic.

Why These Specific Examples Were Chosen

The pedagogical design of these four examples is not random. Each example covers a different path through the decision logic:

• Example 1 fails at condition (a): only one odd number, so the sequence cannot be strange regardless of the sum. This teaches the “minimum count” requirement.
• Example 2 passes both conditions: two odd numbers whose sum is divisible by 3. This teaches the complete “strange” classification.
• Example 3 passes condition (a) but fails condition (b): three odd numbers whose sum (13) is not divisible by 3. This teaches that meeting one condition is not enough.
• Example 4 passes both conditions with three odd numbers: demonstrates that more than two odd numbers is fine.

This systematic coverage, where every path through the decision tree is demonstrated at least once, is the hallmark of well-designed CoT examples. Random or redundant examples would be far less effective because they might all demonstrate the same path, leaving other paths untaught.

The Mechanism: Why CoT Works

CoT works because it transforms an opaque classification task (“Is this strange?”) into a transparent multi-step computation. Without CoT, the model must hold all the logic in its “head” and produce the final answer directly. With CoT, the model produces intermediate results that anchor subsequent steps.

Think of it as the difference between mental arithmetic and written arithmetic. Most people can multiply 23 times 7 in their head (161). Fewer can multiply 234 times 78 without paper. The paper does not make you smarter; it offloads working memory so you can focus on one step at a time. CoT is paper for the LLM.

The difference between few-shot and CoT is the difference between showing a student the answers to practice problems and showing them the worked solutions. Both help. But worked solutions produce deeper understanding and more reliable performance on novel problems, because the student (or model) learns the reasoning process, more than the input-output mapping.

Tree of Thought (ToT) lets the model explore multiple reasoning paths simultaneously. Instead of a single chain, ToT branches into alternatives, evaluates each, and selects the most promising. Think of a chess player considering multiple moves ahead. ToT helped GPT-4 solve 74% of Game of 24 problems versus just 4% with standard CoT.

Thread of Thought (ThoT) addresses chaotic contexts. It adds a “threading” step where the model first identifies the relevant information within a noisy context before attempting to answer. Think of a detective at a crime scene: separate evidence from clutter before drawing conclusions. ThoT is relevant for agents processing long, messy inputs like email threads or multi-document queries.

Both increase token usage and latency. Worth it for complex planning (ToT) or noisy extraction (ThoT). Overkill for classification, summarization, and simple Q&A.

From strings to systems: LangChain’s prompt templates

PromptTemplate: Separating Structure from Content

Before LangChain, prompt templates were implemented as plain Python functions:

def generate_text_summary_prompt(text, num_words, tone):
    return f"""You are an experienced copywriter.
Write a {num_words} words summary of the following text, 
using a {tone} tone: {text}"""

prompt = generate_text_summary_prompt(
    text=segovia_aqueduct_text,
    num_words=20,
    tone="knowledgeable and engaging"
)
response = llm.invoke(prompt)

This works, but the function is not composable. You cannot pipe it into an LLM with the | operator. You cannot trace it with LangSmith. You cannot version it independently from the code that calls it.

LangChain’s PromptTemplate solves all three problems:

from langchain_core.prompts import PromptTemplate

prompt_template = PromptTemplate.from_template(
    """You are an experienced copywriter. Write a {num_words} words 
    summary of the following text, using a {tone} tone: {text}"""
)

The from_template() factory method automatically detects {variable_name} placeholders and registers them as the template’s input_variables. You can verify this:

print(prompt_template.input_variables)  
# ['num_words', 'tone', 'text']

This metadata is used by LangChain for three purposes: validation (raising errors if you forget a variable at invoke time), composition (automatically wiring outputs of one chain step to inputs of the next), and tracing (LangSmith records which variables were used with which values).

The critical advantage over plain Python functions is that PromptTemplate is a Runnable: it implements invoke(), stream(), batch(), and can be composed with other Runnables via the pipe operator. This means you can write:

chain = prompt_template | llm | StrOutputParser()
result = chain.invoke({
    "text": segovia_text,
    "num_words": "20",
    "tone": "knowledgeable"
})

One line creates a complete, composable, traceable summarization chain. This is the LCEL pattern used throughout the remaining systems.

Partial Variables: Pre-Filling What You Know

When building a multi-stage pipeline, you might know some variables at construction time and others only at runtime:

prompt = PromptTemplate(
    template="Summarize this {text} in {num_words} words with {tone} tone",
    input_variables=["text"],
    partial_variables={"num_words": "50", "tone": "professional"}
)

# Only 'text' needed at runtime
result = prompt.format(text="Some long document...")

This becomes essential later in the edition’s research engine, where the output format and assistant instructions are known at construction time but the user’s query arrives at runtime. Partial variables keep the runtime invoke() call clean with only the dynamic variables.

FewShotPromptTemplate: Teaching at Scale

For few-shot learning, FewShotPromptTemplate cleanly separates three concerns:

from langchain_core.prompts.few_shot import FewShotPromptTemplate

# 1. Examples as structured data (could come from a database)
examples = [
    {"number": 6, "reasoning": "not divisible by 5 nor by 7", 
     "result": "None"},
    {"number": 15, "reasoning": "divisible by 5 but not by 7", 
     "result": "Abra"},
    {"number": 21, "reasoning": "divisible by 7 but not by 5", 
     "result": "Kadabra"},
    {"number": 70, "reasoning": "divisible by 5 and by 7", 
     "result": "Abra Kadabra"}
]

# 2. How each example is formatted
example_prompt = PromptTemplate(
    input_variables=["number", "reasoning", "result"],
    template="{number} \\ {reasoning} \\ {result}"
)

# 3. The actual task instruction
few_shot_prompt = FewShotPromptTemplate(
    examples=examples,
    example_prompt=example_prompt,
    suffix="Classify: {input_numbers}",
    input_variables=["input_numbers"]
)

# Format and invoke
prompt_input = few_shot_prompt.format(
    input_numbers="3, 4, 5, 7, 8, 10, 11, 13, 35.")
response = llm.invoke(prompt_input)

The model correctly classifies every number. It deduced the rules entirely from examples.

What the LLM actually receives after formatting (this is the exact prompt sent to the API):

6 \ not divisible by 5 nor by 7 \ None
15 \ divisible by 5 but not by 7 \ Abra
12 \ not divisible by 5 nor by 7 \ None
21 \ divisible by 7 but not by 5 \ Kadabra
70 \ divisible by 5 and by 7 \ Abra Kadabra
Classify the following numbers as Abra, Kadabra or Abra Kadabra:
3, 4, 5, 7, 8, 10, 11, 13, 35.

What the LLM produces:

3 \ not divisible by 5 nor by 7 \ None
4 \ not divisible by 5 nor by 7 \ None
5 \ divisible by 5 but not by 7 \ Abra
7 \ divisible by 7 but not by 5 \ Kadabra
8 \ not divisible by 5 nor by 7 \ None
10 \ divisible by 5 but not by 7 \ Abra
11 \ not divisible by 5 nor by 7 \ None
13 \ not divisible by 5 nor by 7 \ None
35 \ divisible by 5 and by 7 \ Abra Kadabra

Every classification is correct. The model not only applied the rules but adopted the exact same \ delimiter format from the examples, demonstrating that format mimicry is a fundamental property of few-shot learning.

The architecture of FewShotPromptTemplate separates three concerns that are worth understanding deeply because this separation pattern enables sophisticated production usage: examples (the training data, a list of dicts, which can be stored in a database and dynamically selected), example_prompt (how each example is formatted, controlling presentation without changing the examples), and suffix (the actual task instruction with input variables). You can add, remove, or modify examples without touching the prompt logic. in an operating environment, examples are often dynamically selected using SemanticSimilarityExampleSelector, which picks the examples most similar to the current input, maximizing relevance while keeping the prompt concise.

Common bug: When creating templates that output JSON, use double curly braces {{ }} to escape literal braces. A single { is interpreted as a variable placeholder. This is the most common bug when creating templates that output JSON.

Production prompt patterns: three templates for real systems

Pattern 1: The Defensive Prompt

For operating systems where wrong answers have consequences:

DEFENSIVE_TEMPLATE = """You are a {domain} expert assistant.

RULES (follow these strictly):
1. Only answer questions related to {domain}
2. If the question is outside your domain, say 
   "I can only help with {domain} topics"
3. If you are not confident, say "I'm not certain, but..."
4. Never fabricate information. If you don't know, say so.
5. Always cite which part of the context supports your answer

Context: {context}
Question: {question}
Answer:"""

Pattern 2: The Structured Output Prompt

For machine-parseable responses:

STRUCTURED_TEMPLATE = """Extract information from the text.
Return ONLY a JSON object with these exact keys, no other text:
{{
    "name": "person's full name or null",
    "company": "company name or null",
    "role": "job title or null",
    "sentiment": "positive|negative|neutral"
}}

Text: {text}
JSON:"""

Pattern 3: The Evaluation Prompt

For measuring output quality (used extensively later in the edition):

EVAL_TEMPLATE = """Rate the following answer on a scale of 1-5:
1 = Completely wrong or irrelevant
2 = Partially relevant but mostly wrong
3 = Relevant but missing key information
4 = Good answer with minor issues
5 = Excellent, complete, and accurate

Question: {question}
Context provided: {context}
Answer being evaluated: {answer}

Score (just the number):"""

Each template produces meaningfully different behaviour. The defensive template produces terse, fact-only responses with explicit “I don’t know” handling. The structured template produces machine-parseable JSON. The evaluation template produces numerical scores for quality measurement. Choose based on your application’s requirements.

Worked example: building a production classifier

walk through building a customer support ticket classifier from naive to release-tested. This walkthrough demonstrates the entire debugging methodology from this section applied to a real production use case.

Version 1 (fails in an operating environment):

Classify this ticket: {ticket_text}

Produces inconsistent formats: “Billing issue,” “This is a billing problem,” “BILLING,” “I think this is related to billing.” Downstream parsing breaks because the routing system expects a single-word category.

Why it fails: No output format specification. No category list. No persona. The model has too many degrees of freedom in how to express its answer.

Version 2 (add format):

Classify into one category: billing, technical, shipping, returns, general.
Return ONLY the category name.
Ticket: {ticket_text}

Better format. The model consistently returns a single word. But accuracy is mediocre on ambiguous tickets. “My order never arrived and I want a refund” gets classified as “shipping” when the customer’s primary intent is “returns.”

Why it fails: No examples to calibrate the model’s understanding of each category’s boundaries. The model guesses based on keyword associations rather than intent analysis.

Version 3 (add persona + examples):

You are an expert customer support classifier with 10 years
of experience at a major e-commerce company.

Examples:
"My credit card was charged twice" → billing
"App crashes when I open settings" → technical
"Package hasn't arrived after 2 weeks" → shipping
"Want to return defective headphones" → returns
"Do you sell phone cases?" → general

Classify: {ticket_text}
Return ONLY the category name.

Much better. Handles most cases correctly. The persona adds domain expertise. The examples calibrate category boundaries. But still fails on compound tickets like “I was double-charged AND the product was defective.”

Why it fails: No instruction for handling multi-issue tickets. The model tries to classify by the most prominent keywords rather than by the customer’s primary intent.

Version 4 (add edge case handling):

You are an expert customer support classifier.

Rules:
1. Classify into exactly ONE: billing, technical, shipping,
   returns, general
2. Multiple issues → classify by PRIMARY issue (the one the
   customer wants resolved first)
3. If unclear → classify as "general"
4. Emotional language (anger, frustration) does not change
   the classification; focus on the factual issue

[examples...]

Return ONLY the category name.
Ticket: {ticket_text}

This version handles edge cases explicitly. Rule 2 addresses multi-issue tickets. Rule 3 provides a fallback. Rule 4 prevents emotional language from biasing classification (an angry customer talking about a refund is still “returns,” not some special category).

Each version costs the same in infrastructure. The only investment is the time you spent thinking about what could go wrong. The progression from Version 1 to Version 4 is a microcosm of the entire prompt engineering discipline: start simple, test with diverse inputs, identify failure modes, and add the minimum constraint needed to fix each failure.

The token economics of prompt engineering

Understanding the cost structure of different prompting techniques helps you make informed tradeoffs in an operating environment.

Consider a financial news sentiment classifier processing 10,000 headlines per day:

The batch approach is cheapest because it amortizes the system prompt and examples across 10 items per API call. But it sacrifices some accuracy on items late in the batch. The zero-shot approach is nearly as cheap and works well for well-understood tasks. CoT is the most expensive but produces the most reliable results for complex classification.

For most production classification tasks, few-shot with batching (5 examples, 5-10 items per batch) hits the sweet spot: reliable enough for an operating deployment, cost-effective enough at scale.

The prompt engineering debugging methodology

Prompt engineering is fundamentally empirical. You cannot predict quality from theory alone. A prompt that works for the pinned capable model may fail on a pinned model. A prompt that works for English may fail for Japanese.

The professional approach treats prompts as code:

1. Write an initial prompt based on the patterns in this section
2. Test against 20 or more diverse inputs
3. Measure quality systematically
4. Iterate based on failure analysis
5. Version-control prompts alongside code
6. Re-evaluate when changing models or domains

When a prompt fails, work through these steps in order:

Step 1: Check output format. Add explicit format instructions. Resolves most issues.

Step 2: Check persona. Make it specific. “You are a helpful assistant” is almost never right.

Step 3: Add examples. If content is wrong but format is right, add few-shot examples.

Step 4: Add reasoning steps. If examples do not fix it, add explicit decomposition or convert to CoT.

Step 5: Add defensive instructions. “If unsure, say I don’t know.”

Step 6: Consider fine-tuning. Only when steps 1-5 fail. Rare with modern models.

The Batch Processing Trap

A tempting optimisation: batch multiple items in one prompt. “Classify these 20 reviews.” Reduces API calls. But models lose accuracy on late items. The first three reviews are classified well. Reviews 15-20 show degraded quality. The sweet spot varies by model and task. Systematic experimentation is the only way to find it.

The chatprompttemplate: multi-turn structure

For chat-model APIs, ChatPromptTemplate structures prompts as message sequences:

from langchain_core.prompts import ChatPromptTemplate

chat_prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful travel assistant specializing "
               "in Cornwall, England. Only answer travel questions. "
               "If asked about unrelated topics, politely decline."),
    ("human", "{question}")
])

# Compose into a chain
chain = chat_prompt | llm | StrOutputParser()
result = chain.invoke({"question": "What are the best beaches?"})

The system message establishes persistent behavioral constraints. The human message carries the variable input. In multi-turn conversations (the relevant section), assistant messages carrying previous responses are interspersed to maintain context.

This pattern is the foundation for every agent system prompt in Chapters 11 through 13. The system message defines the agent’s personality, tool usage rules, and safety constraints. The human message carries the user’s request. The quality of the system message is often the single largest determinant of agent behaviour quality.

When models outgrow your prompts: the impermanence of prompt engineering

Infante shares a candid reflection that every practitioner should internalize. He spent considerable time crafting the palindrome fix, the step decomposition, the two-shot examples. Then OpenAI released a model update, and the fix became unnecessary. The model now solved palindrome problems correctly zero-shot, decomposing the problem on its own using internal reasoning.

This is not a one-time event. It happens continuously. Techniques that were essential for GPT-3.5 became unnecessary for GPT-4. Techniques essential for GPT-4 became unnecessary for the pinned capable model Thinking. Each generation of models absorbs the most common prompt engineering patterns into its default behaviour.

This does not make prompt engineering obsolete. It means prompt engineering is a frontier skill: the techniques you need are always at the edge of what models can do autonomously. As models improve, the frontier moves. The easy techniques become built-in. The hard techniques remain your competitive advantage.

The practical implication: design your prompts to be modular and removable. Do not hardcode a 500-token CoT example into a prompt that might not need it next quarter. Make each prompt component (persona, examples, steps, defensive instructions) independently toggleable. When a model update makes one component unnecessary, remove it. Your prompts should simplify over time, not accumulate complexity.

This also means that the meta-skill, knowing how to diagnose prompt failures, design examples, and structure reasoning, is more durable than any specific prompt pattern. The palindrome fix is obsolete. The debugging methodology that produced it (test zero-shot, add examples, add steps, measure improvement) works forever.

Make long context an explicit computation

The context window: a desk only so big

Every LLM has a context window, the maximum amount of text it can process in a single prompt. Think of it as a desk. The model can only work with what is physically on the desk at any given moment. Anything not on the desk does not exist for the model.

Early desks were tiny. GPT-3.5 could hold about 16,000 tokens, roughly 12,000 words, maybe 30 pages of text. GPT-4 expanded to 128,000 tokens. selected long-context models handle over a million tokens, enough for a small book.

What Exactly Is a Token?

Before we can discuss context windows meaningfully, we need to understand tokens, because they are the unit of measurement for everything: context window size, API pricing, and chunk sizing.

A token is the smallest unit of text that an LLM processes. Tokens are often parts of words, not whole words. The word “summarization” might become two tokens: [“summ”, “arization”]. The word “cat” is one token. Common short words like “the,” “is,” and “a” are each one token. Numbers are typically one token per digit group. Punctuation marks are usually separate tokens.

A useful rule of thumb: one token is roughly 0.75 words in English. So 1,000 words is roughly 1,333 tokens. Or equivalently, 1,000 tokens is roughly 750 words. This approximation is good enough for cost estimation and planning, but for precise context window management, use OpenAI’s tiktoken library to count tokens exactly.

The Four Problems With Giant Prompts

Even a million-token desk has four problems that make MapReduce relevant regardless of context window size.

Second, the “lost in the middle” phenomenon. Research has consistently shown that LLMs pay more attention to information at the beginning and end of their context window than information in the middle. A 2023 paper from Stanford, “Lost in the Middle: How Language Models Use Long Contexts,” demonstrated that when relevant information was placed in the middle of a 20-document prompt, model performance degraded by 20-30% compared to placing it at the beginning or end. If your critical paragraph is buried on page 200 of a 400-page document, the model may gloss over it. This is not a bug; it is a fundamental property of the attention mechanism. Shorter chunks avoid this problem because every piece of text is near the beginning or end of its chunk.

Third, latency. One 500,000-token call takes much longer than ten parallel 50,000-token calls. For a summarization engine that needs to process and return results in under 30 seconds, a single enormous prompt is the wrong approach. MapReduce’s parallelism can reduce wall-clock time by a factor of N, where N is the number of chunks processed simultaneously.

Fourth, reliability. If one enormous call fails due to a network error, rate limit, or timeout, you start over from scratch. With MapReduce, only the failed chunk needs retrying. The other five successful chunk summaries are preserved. In operating systems processing hundreds of requests per hour, this resilience prevents cascading failures.

These problems are why MapReduce remains relevant even as context windows grow. The technique is not a workaround for small context windows; it is a fundamentally better architecture for processing large documents in an operating environment.

The Stuff Technique: When the Document Fits

Before we reach MapReduce, there is a simpler technique that should be your default whenever possible: Stuff. Stuff means: put the entire document into a single prompt. No splitting, no combining, no complexity.

stuff_prompt = PromptTemplate.from_template("""
Write a comprehensive summary of the following text.
Include all key points, facts, and conclusions.
Text: {text}
Summary:""")

stuff_chain = stuff_prompt | llm | StrOutputParser()
summary = stuff_chain.invoke({"text": full_document_text})

Stuff is always the highest quality because the model sees everything at once, no chunk boundaries to lose context at, no parallel isolation effects. It is also the simplest: one prompt, one LLM call, one output.

Never over-engineer with MapReduce if the document is small enough for Stuff. The complexity of splitting, mapping, and reducing is only justified when it solves a problem that Stuff cannot. Infante skips the Stuff technique in the chapter precisely because it is trivial, but it is important to document as the baseline that you should always try first.

Mapreduce: the divide-and-conquer strategy

The Analogy: Grading a Stack of Exams

Imagine you are a professor with 200 final exams to grade. You could read all 200 sequentially, from first to last, keeping a running mental model of the class performance. By exam 150, you are exhausted and your judgments are drifting. That is thorough but slow, and it does not parallelize.

Or you could distribute the exams to five teaching assistants. Each TA grades 40 exams independently and writes a one-page summary of the common themes, frequent mistakes, and standout performances. You read the five one-page summaries and write a final assessment of the class.

The first approach is the Refine technique (covered later). The second is MapReduce. Both produce a valid summary. MapReduce is faster (five TAs work in parallel), more resilient (if one TA gets sick, only their 40 exams need re-grading), but potentially less coherent (each TA summarizes in isolation without seeing the others’ exams). Refine is slower but preserves more context, because each step builds on the accumulated understanding from all previous steps.

The Three Stages: Split, Map, Reduce

MapReduce summarization works in three stages:

Split: Break the document into chunks that fit within the context window. Each chunk should be small enough to summarize in a single LLM call, with room left for the prompt instructions.

Map: Summarize each chunk independently and in parallel. Each chunk gets its own prompt: “Write a concise summary of the following text and include the main details.”

Reduce: Combine all the chunk summaries into a single final summary. This is another LLM call: “Write a concise summary of the following text, which joins several summaries.”

Walking Through It With Real Numbers

Trace a concrete example. You have the first 18,000 tokens of Moby Dick (five chapters from the Project Gutenberg text, deliberately shortened to keep API costs manageable). Your chunk size is 3,000 tokens with 100-token overlap. The splitter produces six chunks.

The overlap is a subtle but important detail. If you split at exactly 3,000 tokens with no overlap, you might cut a sentence in half: “Captain Ahab stood at the prow, his eyes fixed on the” becomes chunk 1, and “horizon where the white whale had last been sighted” becomes chunk 2. Neither chunk contains the complete thought. With 100-token overlap, the end of chunk 1 overlaps with the beginning of chunk 2, ensuring complete sentences survive the split.

The Overlap Mechanism: Why 100 Tokens?

Examine chunk overlap with a concrete example. Suppose your text is 300 words (roughly 400 tokens), and you split with chunk_size=200 and chunk_overlap=50:

Full text: [token_1, token_2, ..., token_200, ..., token_400]

Chunk 1:   [token_1   ... token_200]
Chunk 2:   [token_151 ... token_350]
Chunk 3:   [token_301 ... token_400]

Notice: tokens 151-200 appear in both Chunk 1 and Chunk 2. Tokens 301-350 appear in both Chunk 2 and Chunk 3. This redundancy ensures that any sentence spanning a chunk boundary is fully present in at least one chunk.

The overlap value should be 10-20% of chunk_size:

Too small (5-10 tokens): sentences will be split. Too large (50% of chunk_size): you are processing half the document twice, doubling cost for marginal quality gain.

What the Moby Dick Output Actually Looks Like

When you run map_reduce_chain.invoke(moby_dick_book) on the five-chapter excerpt, the model produces a coherent summary similar to:

“The introduction to the Project Gutenberg eBook of Moby Dick by Herman Melville outlines the edition’s availability. The narrative begins with Ishmael, the narrator, who seeks solace at sea to escape his melancholic state. He reflects on his reasons for joining a whaling voyage, driven by a fascination with whales and a thirst for adventure. After arriving in New Bedford, Ishmael faces challenges finding lodging, ultimately settling at The Spouter Inn, where he encounters a chaotic environment and a mysterious harpooneer named Queequeg. As Ishmael shares a bed with Queequeg, whom he initially fears, he gradually overcomes his apprehensions. The morning after highlights their strange yet developing bond, emphasizing themes of fate, choice, and the allure of the unknown in the whaling industry.”

This is a solid summary. It captures the major plot points, key characters, settings, and themes. But notice what is missing compared to a human summary of the same text: Ishmael’s philosophical musings on the sea (a major theme that spans multiple chunks), the specific details of The Spouter Inn’s decor (which sets the atmospheric tone), and the subtle humor in Ishmael’s initial terror at sharing a bed with a “cannibal.”

These omissions illustrate the MapReduce tradeoff. The philosophical musings span three chunks and are diluted in each individual chunk’s summary. The atmospheric details are cut during “concise summary” instructions. The humor, which depends on narrative build-up across paragraphs, is lost when those paragraphs are in different chunks.

For a factual report, these losses are acceptable. For a literary analysis, they would be problematic, and you would want either the Stuff technique (if the text fits) or the Refine technique (if it does not).

The cost of getting it wrong: a cautionary calculation

make the cost of poor summarization architecture concrete.

Scenario: Your company processes 100 research reports per day, each 30 pages (approximately 12,000 tokens). You need a summary of each.

The lesson: always do the arithmetic before choosing your architecture. The best technique is the cheapest one that meets your quality threshold. And the only way to know your quality threshold is to test with real documents and real users, which is what the exercises at the end of this section are designed to help you do.

Stage 1: Split. The TokenTextSplitter produces six chunks: - Chunk 1: Ishmael’s journey to New Bedford (tokens 1-3,000) - Chunk 2: Arrival at The Spouter Inn (tokens 2,901-5,900) - Chunk 3: Meeting Queequeg (tokens 5,801-8,800) - Chunk 4: Queequeg’s backstory (tokens 8,701-11,700) - Chunk 5: Preparations for the voyage (tokens 11,601-14,600) - Chunk 6: Boarding the Pequod (tokens 14,501-18,000)

The overlaps: chunk 2 starts 100 tokens before chunk 1 ends, ensuring continuity.

Stage 2: Map. Each chunk goes through the same summarization prompt independently and in parallel. Six LLM calls run simultaneously. Each produces a ~200-word summary. Total map-stage output: approximately 1,200 words (~1,600 tokens).

Stage 3: Reduce. The six summaries are concatenated and sent through a final summarization prompt. One more LLM call produces the final summary.

Total: 7 LLM calls. 6 ran in parallel (map) + 1 sequential (reduce). If each call takes 3 seconds, total time is approximately 6 seconds (3 for map in parallel + 3 for reduce), not 21 seconds (7 sequential calls). This parallelism is MapReduce’s primary advantage.

The output is a coherent summary covering Ishmael’s journey to New Bedford, his encounter with Queequeg at The Spouter Inn, and the themes of fate and adventure in the whaling industry. The quality is good, though some narrative connections between chunks may be lost because each chunk was summarized without knowledge of the others.

LCEL: the language that makes composition effortless

Now we arrive at the tool that makes MapReduce expressible in four lines of code. LCEL, the LangChain Expression Language, is a composition language built around three primitives: the pipe operator for sequential processing, RunnableParallel for parallel processing, and .map() for fan-out across lists.

These are the three primitives The route can in the remaining systems. Master them here, on the conceptually simple task of summarization, and the rest of the edition becomes variations on these same building blocks.

The Pipe Operator: A Pipeline of Transformations

The pipe operator | says: take the output of the left side and feed it as the input to the right side.

chain = prompt | llm | parser

Read this left to right: format the prompt, send it to the LLM, parse the output. The pipe handles all the plumbing of extracting the right fields from one component’s output and packaging them as the right inputs for the next component.

This is the same concept as Unix pipes. In Unix, cat file.txt | grep "error" | wc -l pipes file content through a filter through a counter. In LCEL, prompt | llm | parser pipes a formatted prompt through a language model through an output parser. The philosophical principle is identical: small, composable units connected by a standard interface.

RunnableLambda: Wrapping Any Function as a Component

Not everything you need to do has a built-in LangChain component. Sometimes you need custom logic: splitting text, transforming data structures, filtering results, computing metrics. RunnableLambda wraps any Python function as a chain component that plugs into the pipe operator:

from langchain_core.runnables import RunnableLambda

text_chunks_chain = RunnableLambda(
    lambda x: [
        {'chunk': text_chunk}
        for text_chunk in TokenTextSplitter(
            chunk_size=3000, 
            chunk_overlap=100
        ).split_text(x)
    ]
)

This lambda takes a string x (the full document), splits it into chunks using TokenTextSplitter, and wraps each chunk in a dictionary with a chunk key. The dictionary format matters because downstream components expect named fields, not raw strings.

Two important details about TokenTextSplitter:

Token-based vs. character-based splitting. TokenTextSplitter works in tokens, not characters. A 3,000-token chunk is exactly 3,000 tokens. Since LLM pricing and context windows are measured in tokens, this gives you precise control over costs. However, TokenTextSplitter may split mid-sentence because it does not understand sentence boundaries. The alternative, RecursiveCharacterTextSplitter, respects sentence boundaries but works in characters, so a “3,000-character chunk” might be anywhere from 600 to 800 tokens depending on vocabulary. For summarization, TokenTextSplitter is appropriate because each chunk gets its own LLM call, and a mid-sentence split in one chunk is unlikely to significantly affect the summary quality. For RAG (Chapters 6+), where chunk precision matters more, RecursiveCharacterTextSplitter is usually the better choice.

The chunk_overlap parameter. A value of 100 means the last 100 tokens of chunk N overlap with the first 100 tokens of chunk N+1. This is a tradeoff: too small risks losing context at boundaries (split mid-sentence, mid-paragraph, or mid-argument); too large wastes tokens (the overlapping content is processed and paid for twice). 100 tokens (~75 words) is a reasonable default for narrative text. For technical documentation with dense paragraphs, consider 150-200.

RunnableParallel: Processing Simultaneously

RunnableParallel takes a dictionary of chains and runs them all on the same input simultaneously:

from langchain_core.runnables import RunnableParallel

summarize_map_chain = RunnableParallel({
    'summary': summarize_chunk_prompt | llm | StrOutputParser()
})

When combined with .map(), this creates a separate chain instance for each item in a list, running all instances in parallel.

The complete mapreduce implementation

Walk through every line of code so the data flow is transparent.

The Map Chain: Summarize Each Chunk

summarize_chunk_prompt_template = """
Write a concise summary of the following text, 
and include the main details.
Text: {chunk}
"""

summarize_chunk_prompt = PromptTemplate.from_template(
    summarize_chunk_prompt_template)
summarize_chunk_chain = summarize_chunk_prompt | llm

summarize_map_chain = RunnableParallel({
    'summary': summarize_chunk_chain | StrOutputParser()
})

The Canonical Pattern: prompt | llm | parser

The three-step pattern prompt | llm | parser deserves special attention because it is the most important pattern in the entire book. You will write it, or a variation of it, in every chapter from here forward. dissect what happens at each step:

Step 1: prompt. The PromptTemplate receives a dictionary (like {'chunk': 'text...'}) and fills in the {chunk} placeholder, producing a formatted string. This string is the complete instruction sent to the LLM.

Step 2: llm. The ChatOpenAI model receives the formatted string, sends it to the OpenAI API, and returns a ChatMessage object containing the response text, token usage, finish reason, and other metadata.

Step 3: parser. The StrOutputParser extracts the text content from the ChatMessage object and returns a plain Python string. Without the parser, you would have a ChatMessage object that you would need to access with .content every time. The parser handles this extraction automatically.

Why wrap in RunnableParallel? The RunnableParallel({'summary': ...}) creates a dictionary output with a named key. This matters because the reduce chain’s lambda accesses i['summary'] from each item. Without the wrapping, the output would be a raw string, and the reduce chain would fail with a TypeError when trying to index a string with ['summary'].

This kind of detail, where the output format of one chain must exactly match the input format of the next, is the most common source of LCEL bugs. The error messages are often unhelpful (“expected dict, got str”), and the fix is always the same: trace the data flow from component to component and verify that each step’s output matches the next step’s expected input.

Why .map() Changes Everything

The .map() operator is a important LCEL feature introduced in this section. Without it, you cannot do MapReduce. Without MapReduce, you cannot process documents larger than the context window in parallel.

Here is what .map() does mechanically. When you write:

summarize_map_chain.map()

You are saying: “For each item in the input list, create a separate instance of summarize_map_chain and run all instances in parallel.” If the input list has 6 items (6 chunks), you get 6 independent chain executions running simultaneously. Each chain instance processes one chunk and returns one summary. The .map() collects all outputs into a list.

Without .map():

Input: [chunk1, chunk2, chunk3]
→ summarize_map_chain receives the entire list as a single input
→ Error: prompt template expects a string for {chunk}, got a list

With .map():

Input: [chunk1, chunk2, chunk3]
→ Instance 1: summarize_map_chain(chunk1) → summary1
→ Instance 2: summarize_map_chain(chunk2) → summary2
→ Instance 3: summarize_map_chain(chunk3) → summary3
→ Output: [summary1, summary2, summary3]

The parallelism is handled by LangChain’s runtime. You do not need to write threading code, manage async contexts, or handle concurrency. .map() abstracts all of that.

This fan-out pattern appears in five subsequent chapters: the relevant section (parallel web scraping), the relevant section (multi-query retrieval), the relevant section (multi-store routing), the relevant section (multi-agent execution), and implicitly later in the edition (parallel tool calls). Mastering it here means you already understand a core pattern used throughout the edition.

The Reduce Chain: Combine Summaries

summarize_summaries_prompt_template = """
Write a concise summary of the following text, 
which joins several summaries, and include the main details.
Text: {summaries}
"""

summarize_summaries_prompt = PromptTemplate.from_template(
    summarize_summaries_prompt_template)

summarize_reduce_chain = (
    RunnableLambda(lambda x: {
        'summaries': '\n'.join([i['summary'] for i in x])
    })
    | summarize_summaries_prompt 
    | llm 
    | StrOutputParser()
)

Data flow trace:

1. Input: A list of dictionaries from the map stage: [{'summary': 'sum1'}, {'summary': 'sum2'}, ..., {'summary': 'sum6'}]
2. Lambda: Extracts the summary field from each dictionary and joins them with newlines: {'summaries': 'sum1\nsum2\n...sum6'}
3. Prompt: Fills {summaries} with the combined text
4. LLM: Generates a final summary from the combined summaries
5. Parser: Extracts the text as a clean string

A potential failure mode: if you have many chunks (say, 50 chunks from a very long document), the combined summaries themselves might exceed the context window. For 6 chunks producing ~200-word summaries each, the combined text is ~1,200 words (~1,600 tokens), well within limits. But for 50 chunks, the combined summaries could be ~10,000 words (~13,000 tokens). The fix is hierarchical MapReduce: summarize summaries in batches, then summarize the batch summaries.

The Complete Pipeline in Four Lines

Here is the payoff. The entire MapReduce pipeline, split, map, reduce, expressed in LCEL:

map_reduce_chain = (
    text_chunks_chain            # Split: string → list of dicts
    | summarize_map_chain.map()  # Map: parallel summarization
    | summarize_reduce_chain     # Reduce: combine summaries
)

# Execute with a single invocation
summary = map_reduce_chain.invoke(moby_dick_book)
print(summary)

Four lines. Split. Map. Reduce. Done.

The .map() call on summarize_map_chain is critical and easy to miss. Without .map(), the chain would try to process the entire list of chunks as a single input and fail. With .map(), it creates a separate chain instance for each item in the list and runs them in parallel. This is the fan-out mechanism. The fan-in happens at the reduce chain, where a lambda function joins all summaries into a single string.

The refine technique: quality through accumulation

MapReduce processes chunks in isolation. Each TA grades their exams without seeing anyone else’s. This means chunk 3’s summary knows nothing about chunk 1’s content. If a character introduced in chunk 1 does something important in chunk 3, the chunk 3 summary may lack the context to capture the significance.

The Refine technique addresses this by processing chunks sequentially, building an accumulated summary that grows with each chunk.

The Analogy: Reading a Book Chapter by Chapter

Imagine reading a 400-page novel. After each chapter, you write a summary of the story so far. Your the relevant section summary covers the setup. Your the relevant section summary integrates the new developments with what you already knew later in the edition. By the relevant section, your running summary contains the complete narrative arc, with each chapter’s contribution woven into the larger story.

This is exactly what Refine does. It starts with the first chunk, summarizes it, then takes the second chunk plus the current summary and produces an updated summary. Then takes the third chunk plus the updated summary and produces another update. And so on, until all chunks have been processed.

The Refine Implementation

The initial summary chain handles the first document:

doc_summary_template = """Write a concise summary of the following text:
{text}
DOC SUMMARY:"""
doc_summary_prompt = PromptTemplate.from_template(doc_summary_template)
doc_summary_chain = doc_summary_prompt | llm

The refine chain handles each subsequent document:

refine_summary_template = """
You must produce a final summary from the current refined summary
which has been generated so far and from the content of an 
additional document.
This is the current refined summary generated so far:
{current_refined_summary}
This is the content of the additional document: {text}
Only use the content of the additional document if it is useful, 
otherwise return the current full summary as it is."""

refine_summary_prompt = PromptTemplate.from_template(
    refine_summary_template)
refine_chain = refine_summary_prompt | llm | StrOutputParser()

The instruction “Only use the content of the additional document if it is useful, otherwise return the current full summary as it is” is important. Without it, each iteration might dilute the summary by incorporating irrelevant content. This guard ensures that adding a document about a completely different topic does not corrupt the running summary.

The orchestration loop:

def refine_summary(docs):
    intermediate_steps = []
    current_refined_summary = ''
    
    for doc in docs:
        intermediate_step = {
            "current_refined_summary": current_refined_summary, 
            "text": doc.page_content
        }
        intermediate_steps.append(intermediate_step)
        current_refined_summary = refine_chain.invoke(
            intermediate_step)
    
    return {
        "final_summary": current_refined_summary,
        "intermediate_steps": intermediate_steps
    }

The intermediate_steps list captures the state at each iteration. If the final summary seems off, you can inspect which step introduced the problem. This observability is important for debugging: did the summary go wrong at document 3 (bad content) or document 15 (accumulated drift)?

The Tradeoff: Quality vs. Speed vs. Cost

Here is the core tradeoff between the two techniques, illustrated with concrete numbers for a 50,000-word document split into 22 chunks:

Refine is approximately twice the cost of MapReduce because each iteration sends the growing summary as part of the input. The cumulative input tokens (155,000) represent the running summary being re-sent with every chunk. It is also 10x slower because no parallelism is possible; each step depends on the previous step’s output.

But Refine produces higher-quality summaries because each step has full context. If your document tells a story with callbacks, cross-references, or evolving themes, Refine captures these connections while MapReduce loses them.

A production hybrid that Infante hints at: use MapReduce for initial rough summaries quickly, then Refine over the most important documents to ensure faithful representation. This gives you MapReduce’s speed for the bulk work and Refine’s quality for the critical pieces.

When to Use Which: The Decision Flowchart

Multi-source summarization: documents from everywhere

worked summarization rarely involves a single source. A research analyst might need to synthesize information from Wikipedia articles, PDF reports, Word documents, and web pages. LangChain’s Document Loaders handle this heterogeneity by converting every source type into the same Document object: text plus metadata.

The Document Object: LangChain’s Universal Container

Every piece of text flowing through LangChain is wrapped in a Document object:

Document(
    page_content="Cornwall has over 300 beaches spanning...",
    metadata={
        "source": "wikipedia", 
        "title": "Cornwall Beaches",
        "language": "en"
    }
)

The page_content field holds the raw text. The metadata dictionary holds provenance information: where the text came from, what page it was on, when it was loaded, and any other information you want to track. This metadata is not decorative. It flows through the entire pipeline. When you split a Document into chunks, each chunk inherits the metadata. When you summarize, you can trace which summary came from which source. in an operating environment, this provenance chain is essential for citation (“This information comes from page 42 of the 2024 Annual Report”), audit trails, and debugging bad summaries.

The Loader Zoo: Getting Data In

LangChain provides dozens of loaders for different formats:

# Wikipedia: Content + hyperlinked articles
from langchain_community.document_loaders import WikipediaLoader
wiki_docs = WikipediaLoader(query="Paestum", load_max_docs=2).load()
# TIP: load_max_docs prevents loading dozens of linked articles
# Each doc has metadata: {"title": "Paestum", "source": "..."}

# PDF: One Document per page
from langchain_community.document_loaders import PyPDFLoader
pdf_docs = PyPDFLoader("report.pdf").load()
# TIP: Returns a list with one Document per page
# Metadata includes page number, enabling page-level citations
# For scanned PDFs, use UnstructuredPDFLoader with strategy="ocr_only"

# Word documents: Preserves text, loses formatting
from langchain_community.document_loaders import Docx2txtLoader
word_docs = Docx2txtLoader("document.docx").load()
# TIP: Images in Word docs are lost. Tables are converted to plain text.

# Plain text: Simplest loader
from langchain_community.document_loaders import TextLoader
txt_docs = TextLoader("notes.txt", encoding="utf-8").load()
# TIP: Always specify encoding. Default assumes system encoding, 
# which varies across OS. UTF-8 is safe for most content.

# Web pages: Raw HTML
from langchain_community.document_loaders import AsyncHtmlLoader
html_docs = AsyncHtmlLoader(["https://example.com"]).load()
# TIP: Returns raw HTML. Use Html2TextTransformer to extract text.

Production Concern: Error Handling Across Sources

When loading from multiple sources, some will fail. The Wikipedia server might be slow. The PDF might be corrupted. The URL might be unreachable. A production pipeline must handle these failures gracefully:

def safe_load(loader, source_name):
    """Load with error handling and logging."""
    try:
        docs = loader.load()
        print(f"Loaded {len(docs)} docs from {source_name}")
        return docs
    except Exception as e:
        print(f"ERROR loading {source_name}: {e}")
        return []  # Return empty, don't crash

all_docs = []
all_docs.extend(safe_load(WikipediaLoader(query="Paestum"), "Wikipedia"))
all_docs.extend(safe_load(PyPDFLoader("report.pdf"), "PDF"))
all_docs.extend(safe_load(TextLoader("notes.txt"), "Text"))

if not all_docs:
    raise ValueError("No documents loaded successfully")

Production Concern: Heterogeneous Quality

When loading from multiple sources, document quality varies widely. A Wikipedia article has been edited by hundreds of contributors and is generally reliable. A scraped web page might be full of navigation text, ads, and boilerplate. A PDF from a company’s annual report is high-signal. A text file of hastily written meeting notes might be low-quality.

Garbage in, garbage out applies at every stage. Pre-filtering or quality-scoring documents before summarization prevents noisy sources from contaminating the final output. A common pattern: use a cheap LLM call to rate each document’s relevance and quality on a 1-5 scale, then only include documents scoring 3 or above.

LCEL anti-patterns: what not to do

Common mistakes when building LCEL chains, collected from production experience:

Anti-Pattern 1: Mutating state inside RunnableLambda.

# BAD: External state mutation
results = []
chain = RunnableLambda(lambda x: results.append(x))  # Side effect!

# GOOD: Return new values
chain = RunnableLambda(lambda x: {"results": [x]})

Mutations inside lambdas cause unpredictable behaviour in parallel execution (.map()) because multiple instances share the same mutable object. Always return new values; let LCEL manage state.

Anti-Pattern 2: Deeply nested lambdas.

# BAD: Unreadable and undebuggable
chain = RunnableLambda(lambda x: 
    RunnableLambda(lambda y: 
        {"text": y["content"].strip().lower()}).invoke(
            {"content": x["raw"]}))

# GOOD: Named functions
def clean_text(state):
    return {"text": state["raw"].strip().lower()}

chain = RunnableLambda(clean_text)

Named functions are debuggable (set breakpoints), testable (call independently), and readable (name documents purpose).

Anti-Pattern 3: Unbounded .map() parallelism.

# BAD: Could be 100 parallel LLM calls
chain = generate_queries | summarize_chain.map()

# GOOD: Cap the parallelism
chain = generate_queries | RunnableLambda(lambda x: x[:10]) | summarize_chain.map()

Always cap items processed by .map(). Without a cap, a query generator that produces 50 variants triggers 50 simultaneous LLM calls, potentially hitting rate limits and running up costs.

Anti-Pattern 4: Using LCEL when a simple function suffices.

# BAD: Over-engineering
chain = (
    RunnableLambda(lambda x: x["text"])
    | RunnableLambda(lambda x: x.strip())
    | RunnableLambda(lambda x: x.lower()))

# GOOD: Just write a function
def clean(state):
    return state["text"].strip().lower()

LCEL shines for composing LLM calls, retrievers, and prompts. For simple data transformations, regular Python functions are clearer and faster.

Operating boundary: what the tutorial does not tell you

Error Handling Per Chunk

In a MapReduce pipeline with 50 chunks, one or two will occasionally fail due to rate limits, timeouts, or malformed content. The naive pipeline crashes entirely. A production pipeline catches per-chunk errors, logs them, skips the failed chunk, and proceeds:

import time

def production_map_reduce(text, llm, chunk_size=3000):
    """MapReduce with error handling and metrics."""
    start_time = time.time()
    splitter = TokenTextSplitter(chunk_size=chunk_size, 
                                 chunk_overlap=100)
    chunks = splitter.split_text(text)
    
    summaries = []
    failed_chunks = 0
    for i, chunk in enumerate(chunks):
        try:
            summary = (summarize_chunk_prompt 
                      | llm 
                      | StrOutputParser()).invoke({"chunk": chunk})
            summaries.append(summary)
        except Exception as e:
            print(f"WARNING: Chunk {i} failed: {e}")
            failed_chunks += 1
    
    if not summaries:
        raise ValueError("All chunks failed to summarize")
    
    combined = '\n'.join(summaries)
    final = (summarize_summaries_prompt 
            | llm 
            | StrOutputParser()).invoke({"summaries": combined})
    
    elapsed = time.time() - start_time
    print(f"Completed in {elapsed:.1f}s | "
          f"{len(chunks)} chunks | "
          f"{failed_chunks} failures | "
          f"~{len(chunks) + 1} LLM calls")
    return final

Hierarchical Reduce for Very Large Documents

For documents exceeding 100 pages where even the combined MapReduce summaries exceed the context window:

def hierarchical_map_reduce(text, llm, chunk_size=3000, 
                             batch_size=10):
    """Two-level MapReduce for very large documents."""
    splitter = TokenTextSplitter(chunk_size=chunk_size, 
                                 chunk_overlap=100)
    chunks = splitter.split_text(text)
    
    # Level 1: Summarize each chunk
    chunk_summaries = []
    for chunk in chunks:
        summary = (summarize_chunk_prompt | llm | StrOutputParser()
                  ).invoke({"chunk": chunk})
        chunk_summaries.append(summary)
    
    # Level 2: Summarize in batches
    batch_summaries = []
    for i in range(0, len(chunk_summaries), batch_size):
        batch = chunk_summaries[i:i+batch_size]
        combined = "\n".join(batch)
        batch_summary = (summarize_summaries_prompt | llm 
                        | StrOutputParser()
                        ).invoke({"summaries": combined})
        batch_summaries.append(batch_summary)
    
    # Final: Combine batch summaries
    final_combined = "\n".join(batch_summaries)
    return (summarize_summaries_prompt | llm | StrOutputParser()
           ).invoke({"summaries": final_combined})

For a 500-page book (200 chunks), Level 1 produces 200 summaries. Level 2 groups them into 20 batches of 10, producing 20 batch summaries. The final step combines those into one. Total: 221 LLM calls, parallelizable at each level.

Token Economics of Summarization

Understanding the cost structure helps you choose the right technique:

Key insight: Refine is approximately 2x the cost of MapReduce because each iteration sends the growing summary as part of the input. Stuff is cheapest but only works when content fits.

Choosing your splitter: a comparison

The choice of text splitter affects summarization quality more than most developers realize. Here is a comparison:

For summarization, TokenTextSplitter is acceptable because each chunk gets its own summarization prompt, and a mid-sentence split in one chunk rarely affects the overall summary quality. For RAG (Chapters 6+), where the precise boundaries of retrieved chunks determine answer quality, RecursiveCharacterTextSplitter is the better default.

Choosing chunk_size and chunk_overlap: A useful rule of thumb is that chunk_overlap should be 10-20% of chunk_size. For a chunk_size of 3,000 tokens, an overlap of 100-300 tokens works well. Too small risks boundary losses. Too large wastes tokens by processing overlapping content twice.

Build research as a visible pipeline

A complete execution trace: following the astorga question

Before diving into the architecture, Trace a complete execution from question to report. This end-to-end walkthrough makes the abstract pipeline concrete and shows what data looks like at each stage.

Input: “What can I see and do in the Spanish town of Astorga?”

Stage 1: Chain 1 (Assistant Selection) produces:

{
  "assistant_type": "Tour guide assistant",
  "assistant_instructions": "You are a knowledgeable tour guide. 
    Your goal is to provide comprehensive travel information 
    including cultural sites, local cuisine, practical tips, 
    and hidden gems.",
  "user_question": "What can I see and do in the Spanish town 
    of Astorga?"
}

The LLM read the question, recognized it as a travel question, and selected the tour guide persona. This selection shapes every subsequent step: the search queries will focus on tourism, the summaries will highlight attractions and cuisine, and the report will read like a travel guide rather than a financial analysis.

Stage 2: Chain 2 (Query Generation) produces:

[
  "Astorga Spain tourist attractions historical sites",
  "things to do in Astorga Spain food culture"
]

Two queries instead of one. The first targets attractions and history. The second targets activities, food, and culture. Together they cover more ground than the original conversational question.

Stage 3: Chain 3 (Search and Summarize) produces:

For query 1, the web search returns 3 URLs. Each is scraped and summarized in parallel: - URL 1 (Wikipedia): Summary about Astorga’s Roman walls and Episcopal Palace - URL 2 (travel blog): Summary about the Chocolate Museum and Cocido Maragato - URL 3 (tourism board): Summary about the Cathedral and pilgrimage routes

For query 2, another 3 URLs are scraped and summarized in parallel: - URL 4 (food blog): Summary about traditional Maragato cuisine - URL 5 (trip advisor): Summary about the Roman Museum and walking routes - URL 6 (culture site): Summary about festivals and local traditions

All 6 summaries are joined into a single text block of approximately 2,000 words.

Stage 4: Chain 4 (Report Synthesis) produces:

A 1,200+ word markdown report with sections on Historical Sites (Roman walls, Cathedral, Episcopal Palace by Gaudi), Cultural Experiences (Chocolate Museum, Roman Museum), Culinary Highlights (Cocido Maragato, local chocolate tradition), Practical Tips (best time to visit, walking routes), and source citations in APA format.

This trace reveals the architecture’s elegance: each stage transforms data into a more useful form. Raw question becomes targeted queries. Queries become URLs. URLs become text. Text becomes summaries. Summaries become a report. The pipe operator connects each transformation.

The manual process: what we are automating

Imagine you are researching the Spanish town of Astorga for a travel article. Manually, you would:

1. Open Google and search “Astorga Spain tourist attractions”
2. Open the top 3-5 results in browser tabs
3. Read each page, highlighting relevant paragraphs
4. Open Google again and search “Astorga Spain things to do”
5. Open 3-5 more results, read, highlight
6. Open your word processor and start combining notes
7. Draft the article, weaving together information from 6-10 sources
8. Go back to verify claims, check that URLs still work
9. Format, edit, publish

A semi-automated approach: copy text from each web page, paste into ChatGPT, ask for a summary, repeat for each page, then combine all summaries into a final prompt asking for a consolidated report. This saves reading time but is still manual and tedious for more than 3-4 sources.

The fully automated approach: type “What can I see and do in the Spanish town Astorga?” and receive a 1,200-word markdown report with source citations. One command. One output. All the intermediate work, searching, scraping, summarizing, synthesizing, handled by a pipeline of LLM chains running in parallel.

Query rewriting: the technique that doubles search quality

Before diving into the four-chain architecture, it is worth pausing on query rewriting because it is one of the highest-impact techniques in the entire book, and it appears here first before returning later in the edition (Multi-Query retrieval) and the relevant section (routing).

The user asks: “What can I see and do in the Spanish town of Astorga?”

This is a reasonable question for a human conversation. But as a web search query, it is suboptimal. It is too long (search engines work better with 3-6 words), it is too broad (combines sightseeing and activities), and it uses conversational phrasing that search engines do not optimize for.

Query rewriting transforms this single question into multiple targeted searches:

Original: "What can I see and do in the Spanish town of Astorga?"

Rewritten:
1. "Astorga Spain tourist attractions historical sites"
2. "things to do in Astorga Spain food culture"

Each rewritten query captures a different facet of the original question. Query 1 targets attractions and historical sites. Query 2 targets activities, food, and culture. Together, they return results that cover more ground than the original question alone.

This same technique reappears as the Multi-Query retrieval pattern later in the edition, where instead of web search queries, the LLM generates multiple vector store queries to improve RAG retrieval precision. The principle is identical; only the search backend changes.

The architecture: four chains, one pipeline

The research engine decomposes the research process into four modular stages, each implemented as an independent LCEL chain that can be tested, debugged, and replaced without affecting the others. This modular architecture is one of the most important design patterns in the edition: it applies equally to RAG pipelines (Chapters 6-10) and agent systems (Chapters 11-14).

Chain 1: Assistant Selection (Dynamic Persona)

The first chain classifies the user’s question and selects the most appropriate research persona. A question about NBA statistics needs a sports analyst. A question about stock valuation needs a financial analyst. A question about tourist destinations needs a travel expert.

This classification uses the few-shot prompting technique later in the edition: the prompt includes examples of question-to-persona mappings, and the LLM outputs structured JSON specifying the selected assistant type and instructions:

ASSISTANT_SELECTION_INSTRUCTIONS = """
You are skilled at assigning the best research assistant...

Question: "What are the growth prospects of NVIDIA stock?"
Response:
{{"assistant_type": "Financial analyst assistant",
 "assistant_instructions": "You are a financial analyst. 
  Your goal is to provide investment analysis backed by data.",
 "user_question": "{user_question}"}}

Question: "What can I see in Astorga?"
Response:
{{"assistant_type": "Tour guide assistant",
 "assistant_instructions": "You are a knowledgeable tour guide. 
  Your goal is to provide comprehensive travel information.",
 "user_question": "{user_question}"}}

Question: "Is Djokovic the GOAT?"
Response:
{{"assistant_type": "Sport expert assistant",
 "assistant_instructions": "You are a sports expert and journalist. 
  Your goal is to provide analysis backed by statistics.",
 "user_question": "{user_question}"}}

Now classify this question:
Question: "{user_question}"
Response:
"""

The double curly braces {{ }} are escape sequences in PromptTemplate. A single { would be interpreted as a variable placeholder. This is the most common bug when creating templates that produce JSON output: forgetting to escape literal braces.

The chain in LCEL:

assistant_instructions_chain = (
    {'user_question': RunnablePassthrough()}
    | ASSISTANT_SELECTION_PROMPT_TEMPLATE
    | get_llm()
    | StrOutputParser()
    | to_obj  # Custom JSON parser: string → Python dict
)

The to_obj function parses the LLM’s JSON string output into a Python dictionary. This is a RunnableLambda wrapping json.loads() with error handling. The dictionary flows to Chain 2, which uses the assistant_instructions field to customize the search query generation.

A single user question rarely produces optimal search results. “Tell me about Astorga” is too broad. The LLM rewrites the question into multiple, more targeted search queries:

WEB_SEARCH_INSTRUCTIONS = """
{assistant_instructions}

Write {num_search_queries} web search queries to gather 
as much information as possible on the following question: 
{user_question}

You must respond with a list of strings in the following format:
["query 1", "query 2", "query 3"]
"""

For the Astorga question, the LLM might generate:

["Astorga Spain tourist attractions historical sites",
 "things to do in Astorga Spain food culture"]

This query rewriting technique, generating multiple perspectives from a single question, is one of the highest-impact techniques for search-based applications. It captures perspectives the user might not have considered (food, culture, history) and breaks broad queries into focused searches that return more relevant results. The exact same technique reappears later in the edition as the Multi-Query retrieval pattern for vector store search.

Chain 3: Search and Summarize (The Parallel Engine)

This is the most architecturally interesting chain because it contains two levels of parallelism, and understanding both levels is essential for any production LCEL pipeline.

Level 1: Each search query runs independently. If Chain 2 generated 2 queries, 2 search-and-summarize instances execute in parallel.

Level 2: Within each search, each URL is scraped and summarized independently. If each search returns 3 URLs, 3 scrape-and-summarize instances execute in parallel.

With 2 queries returning 3 URLs each, you get 6 parallel scrape-and-summarize operations. Trace the wall-clock time:

Without parallelism (sequential):

Query 1: Search (2s) + Scrape URL1 (3s) + Summarize URL1 (3s)
                     + Scrape URL2 (4s) + Summarize URL2 (3s)
                     + Scrape URL3 (2s) + Summarize URL3 (3s)
Query 2: Search (2s) + Scrape URL4 (3s) + Summarize URL4 (3s)
                     + Scrape URL5 (5s) + Summarize URL5 (3s)
                     + Scrape URL6 (2s) + Summarize URL6 (3s)
Total: 2+3+3+4+3+2+3+2+3+3+5+3+2+3 = ~41 seconds

With Level 1 parallelism only (queries parallel, URLs sequential):

Query 1 and Query 2 run simultaneously:
  Query 1: Search (2s) + [URL1 (6s) + URL2 (7s) + URL3 (5s)] = 20s
  Query 2: Search (2s) + [URL4 (6s) + URL5 (8s) + URL6 (5s)] = 21s
  Wall-clock: max(20, 21) = 21 seconds

With both levels of parallelism (queries parallel AND URLs parallel):

Query 1 and Query 2 run simultaneously:
  Query 1: Search (2s) + max(URL1: 6s, URL2: 7s, URL3: 5s) = 2+7 = 9s
  Query 2: Search (2s) + max(URL4: 6s, URL5: 8s, URL6: 5s) = 2+8 = 10s
  Wall-clock: max(9, 10) = 10 seconds

From 41 seconds to 10 seconds: a 4x speedup with zero additional cost. The same 6 scraping operations and 6 LLM calls happen; they just happen concurrently instead of sequentially.

The core components that enable this:

# Web search function (DuckDuckGo, no API key needed)
def web_search(web_query: str, num_results: int) -> List[str]:
    return [r["link"] for r in 
            DuckDuckGoSearchAPIWrapper().results(
                web_query, num_results)]

# Web scraping function
def web_scrape(url: str, max_chars: int = 10000) -> str:
    response = requests.get(url, timeout=10)
    soup = BeautifulSoup(response.text, "html.parser")
    text = soup.get_text(separator="\n", strip=True)
    return text[:max_chars]  # Truncate to control token cost

The max_chars=10000 truncation is a cost control mechanism. 10,000 characters is roughly 2,500 tokens. Without truncation, a single web page could consume 20,000+ tokens, making the summarization call expensive and potentially exceeding the context window. Truncation is crude but effective; a more sophisticated approach uses a readability algorithm (like Mozilla’s Readability) to extract only the main content, stripping navigation, ads, sidebars, and boilerplate.

The summarization prompt for each scraped page:

SUMMARY_INSTRUCTIONS = """
{assistant_instructions}

Using the following text from a web page, extract the relevant 
information for the following question: {user_question}

Text: {search_result_text}

Write a concise summary that captures the key relevant details.
Include the source URL for attribution.
Source URL: {result_url}
"""

The design shows how {assistant_instructions} flows from Chain 1 through Chain 2 into Chain 3. The travel guide persona shapes how each page is summarized: it emphasizes attractions, practical tips, and local cuisine rather than raw facts and statistics. A financial analyst persona would extract different information from the same page.

The LCEL composition for the search-and-summarization chain:

# Chain 3a: Get URLs for a single search query
search_result_urls_chain = (
    RunnableLambda(lambda x: {
        'urls': web_search(x['query'], NUM_SEARCH_RESULTS_PER_QUERY),
        'user_question': x['user_question'],
        'assistant_instructions': x['assistant_instructions']
    })
)

# Chain 3b: Scrape and summarize a single URL
search_result_text_and_summary_chain = (
    RunnableLambda(lambda x: {
        'search_result_text': web_scrape(x['url'])[:RESULT_TEXT_MAX_CHARACTERS],
        'user_question': x['user_question'],
        'assistant_instructions': x['assistant_instructions'],
        'result_url': x['url']
    })
    | SUMMARY_PROMPT_TEMPLATE
    | get_llm()
    | StrOutputParser()
    | RunnableLambda(lambda x: {'summary': x, 
                                 'user_question': ...})
)

# Chain 3 combined: URLs → parallel scrape+summarize → join
search_and_summarization_chain = (
    search_result_urls_chain
    | search_result_text_and_summary_chain.map()  # Level 2 parallelism
    | RunnableLambda(lambda x: {
        'summary': '\n'.join([i['summary'] for i in x]),
        'user_question': x[0]['user_question'] if x else ''
    })
)

The .map() on search_result_text_and_summary_chain creates a separate chain instance for each URL. If search_result_urls_chain returns 3 URLs, .map() creates 3 parallel instances. Each instance scrapes one URL and summarizes it independently. The final RunnableLambda joins all summaries into a single text block.

The master chain then applies .map() again at Level 1:

web_research_chain = (
    assistant_instructions_chain
    | web_searches_chain
    | search_and_summarization_chain.map()  # Level 1 parallelism
    | RunnableLambda(combine_all_summaries)
    | report_chain
)

Two .map() calls, one nested inside the other. Level 1 .map() parallelizes across queries. Level 2 .map() (inside search_and_summarization_chain) parallelizes across URLs within each query. This nested parallelism is the key architectural insight of the chapter.

The .map() Rate Limit Trap

The .map() operator sends all requests simultaneously. With 2 queries returning 3 URLs each, that is 6 simultaneous web scraping requests and 6 simultaneous LLM calls. This can hit API rate limits, especially with OpenAI’s per-minute token limits on lower-tier accounts.

Mitigation strategies: 1. Reduce NUM_SEARCH_RESULTS_PER_QUERY from 3 to 2 2. Use batch() instead of map() for rate-limited execution 3. Add a small delay between chain instances using asyncio.sleep() 4. Use a higher-tier API plan with more generous rate limits

in an operating environment, monitor for 429 Rate Limit errors and implement exponential backoff. LangChain’s .with_retry() method handles this automatically:

resilient_chain = (
    summarize_chain.with_retry(
        stop_after_attempt=3,
        wait_exponential_multiplier=1
    )
)

The LCEL data flow type map

The most common debugging challenge in LCEL is understanding what data type each component expects. Here is the complete type annotation:

Chain 1: str → dict
  Input:  "What can I see in Astorga?"
  Output: {"assistant_type": "Tour guide",
           "assistant_instructions": "You are...",
           "user_question": "What can I see in Astorga?"}

Chain 2: dict → list[dict]
  Input:  {"assistant_instructions": "...", "user_question": "..."}
  Output: [{"query": "Astorga attractions", ...},
           {"query": "things to do Astorga", ...}]

Chain 3 (.map()): list[dict] → list[dict]
  Input:  [{"query": "...", ...}, {"query": "...", ...}]
  Output: [{"summary": "combined for query 1", ...},
           {"summary": "combined for query 2", ...}]

Combine: list[dict] → dict
  Input:  [{"summary": "...", ...}, {"summary": "...", ...}]
  Output: {"research_summary": "all summaries joined",
           "user_question": "..."}

Chain 4: dict → str
  Input:  {"research_summary": "...", "user_question": "..."}
  Output: "# Research Report\n\n## Introduction\n..."

When the pipeline fails with “expected dict, got str,” trace this map from component to component. The output type of each chain must exactly match the input type of the next. This type-mismatch debugging is the most common LCEL development task.

Chain 4: Report Synthesis (The Final Composition)

The report chain receives all summaries plus the original question and generates a broad research report:

RESEARCH_REPORT_INSTRUCTIONS = """
You are an AI critical thinker research assistant. Your sole 
purpose is to write well written, critically acclaimed, 
objective and structured reports on given text.

Information: {research_summary}
Using the above information, answer the following question 
or topic: "{user_question}" in a detailed report.

The report should focus on the answer to the question, 
should be well structured, informative, in depth, with 
facts and numbers if available and a minimum of 1,200 words.

You must write the report with markdown syntax.
You MUST determine your own concrete and valid opinion 
based on the given information. Do not dilly-dally.
Write all used source urls at the end of the report, 
in APA format.

Please do your best, this is very important to my career.
"""

This prompt deserves close examination because it demonstrates five advanced prompting techniques simultaneously, and each one measurably affects the output:

1. Minimum word count (“a minimum of 1,200 words”) forces detailed output. Without this instruction, LLMs tend to produce 200-400 word summaries even when given 2,000 words of source material. The word count instruction pushes the model to elaborate, add context, and develop arguments rather than just listing facts.

2. Format specification (“markdown syntax”) ensures structured output with headers, bullet lists, and emphasis that can be rendered directly in any markdown viewer. Without this, the model defaults to plain prose that is harder to scan.

3. Opinion requirement (“determine your own concrete and valid opinion”) pushes the LLM beyond pure summarization into analysis and synthesis. A summary says “Astorga has Roman walls.” An analysis says “Astorga’s Roman walls, combined with Gaudi’s Episcopal Palace and the pilgrimage route, make it an underappreciated gem for history enthusiasts seeking alternatives to crowded Barcelona.”

4. Source attribution (“source urls in APA format”) forces citation, improving trustworthiness and enabling readers to verify claims. Without this, the report presents facts without provenance, which is unacceptable for professional research.

5. Emotional appeal (“this is very important to my career”) is the most controversial technique. Anecdotal testing suggests it improves output quality, possibly because similar phrases in training data tend to co-occur with carefully written responses. The effect is debated in the prompt engineering community, but Infante includes it as a practical technique worth experimenting with.

From sequential to LCEL: the refactoring journey

the edition provides both a sequential implementation and an LCEL implementation. Walking through the refactoring teaches exactly what LCEL buys you and what it costs.

The Sequential Version

The sequential implementation in research_engine_seq.py uses explicit loops and intermediate variables:

# Sequential: ~80 lines, explicit, debuggable
question = "What can I see in Astorga?"

# Step 1: Classify
assistant = classify_question(question)

# Step 2: Generate queries
queries = generate_queries(question, assistant["instructions"])

# Step 3: Search and summarize (sequential loop)
all_summaries = []
for query in queries:
    urls = web_search(query, num_results=3)
    for url in urls:
        text = web_scrape(url)
        summary = summarize_text(text, question, assistant["instructions"])
        all_summaries.append(summary)

# Step 4: Generate report
combined = "\n".join(all_summaries)
report = generate_report(combined, question)

This is readable, debuggable (set a breakpoint on any line), and easy to understand. But the nested loop in Step 3 processes URLs one at a time. With 6 URLs, each taking 5-8 seconds for scrape plus LLM call, Step 3 takes 30-48 seconds.

The LCEL Version

The LCEL reimplementation replaces loops with composition:

# LCEL: ~60 lines, declarative, parallel
web_research_chain = (
    assistant_instructions_chain
    | web_searches_chain
    | search_and_summarization_chain.map()
    | RunnableLambda(combine_summaries)
    | report_chain
)
report = web_research_chain.invoke(question)

Five lines replace 20 lines of loop logic. More importantly, the .map() operator automatically parallelizes the search-and-summarize stage, reducing Step 3 from 30-48 seconds to 8-12 seconds.

What the Refactoring Teaches

The refactoring reveals three principles that apply to every LCEL conversion:

Principle 1: Loops become .map(). Anywhere you write for item in items: result = chain(item), LCEL expresses as chain.map().invoke(items). The semantic is identical; the execution is parallel.

Principle 2: Intermediate variables become pipe connections. In sequential code, result1 = step1(input); result2 = step2(result1) becomes step1 | step2 in LCEL. The pipe operator replaces variable assignments.

Principle 3: Data transformation functions become RunnableLambda. Any pure function that transforms data (joining strings, extracting fields, filtering lists) wraps in RunnableLambda to participate in the pipe chain.

The cost of refactoring: debugging becomes harder. You cannot set a breakpoint inside a pipe chain. You cannot inspect intermediate values without adding explicit logging. LangSmith compensates by recording every step automatically, but the feedback loop is slower than a debugger breakpoint.

The benefit: parallelism, composability, traceability, and the ability to swap components without rewriting the pipeline. for an operating deployment code that runs thousands of times, this tradeoff always favors LCEL. For development code that you are debugging, start sequential and refactor to LCEL once the logic is correct.

Testing each chain independently: the unit testing principle

One of the most important production practices from this section: test each sub-chain independently before composing them. This follows the same principle as unit testing in software engineering. Each chain has a defined input contract (what dictionary keys it expects) and output contract (what dictionary keys it returns). Testing contracts independently catches integration bugs at the earliest point.

# Test Chain 1: Does classification work?
persona = assistant_instructions_chain.invoke(
    "What are the growth prospects of NVIDIA stock?")
print(f"Type: {persona.get('assistant_type')}")
assert "financial" in persona.get("assistant_type", "").lower()

# Test Chain 2: Does query generation work?
queries = web_searches_chain.invoke({
    "user_question": "Astorga Spain",
    "assistant_instructions": "You are a travel expert",
    "num_search_queries": 2
})
print(f"Queries: {queries}")
assert len(queries) == 2  # Should generate exactly 2 queries

# Test Chain 3: Does search+summarize work for one query?
summaries = search_and_summarization_chain.invoke({
    "query": "Astorga Spain tourist sites",
    "user_question": "What can I see in Astorga?",
    "assistant_instructions": "You are a travel expert"
})
print(f"Summary length: {len(summaries['summary'])} chars")
assert len(summaries['summary']) > 100  # Should have real content

# Test Chain 4: Does report generation work?
report = report_chain.invoke({
    "research_summary": "Astorga is a Spanish town known for...",
    "user_question": "What can I see in Astorga?"
})
print(f"Report length: {len(report)} chars")
assert "Astorga" in report  # Should reference the topic

If the full pipeline fails, the failing test tells you exactly which stage broke. Chain 1 returning an empty dict? The classification prompt needs debugging. Chain 3 returning empty summaries? The web scraping is failing. Chain 4 producing a short report? The report prompt needs strengthening.

This test-first approach is even more important for the LangGraph workflows later in the edition and the agents later in the edition, where the execution path is non-linear and debugging without per-node tests is impractical.

Configuration constants: controlling cost and quality

Three constants define the cost-quality tradeoff for every run:

NUM_SEARCH_QUERIES = 2              # Queries from Chain 2
NUM_SEARCH_RESULTS_PER_QUERY = 3    # URLs per query from search
RESULT_TEXT_MAX_CHARACTERS = 10000  # Max chars scraped per page

These numbers cascade through the entire pipeline:

Doubling any constant roughly doubles both cost and quality. The default (2×3=6 pages) is a reasonable starting point for most research questions. For thorough research reports, increase to 3×5=15 pages. For quick factual lookups, reduce to 1×2=2 pages.

in an operating environment, make these configurable per request so users can choose between “quick answer” (low cost, fast) and “deep research” (higher cost, broad). This is the same principle as model selection later in the edition: route to the configuration that matches the task’s requirements.

Operating boundary: what breaks in the real world

Web Scraping Failures

Web scraping is the most fragile part of the pipeline. Pages change structure, servers block automated requests, JavaScript-rendered pages return empty HTML, and paywalled content returns login forms instead of articles. A production pipeline must handle all of these:

def robust_web_scrape(url: str, max_chars: int = 10000) -> str:
    """Scrape with error handling and content validation."""
    try:
        response = requests.get(url, timeout=10, headers={
            "User-Agent": "Mozilla/5.0 (Research Bot)"
        })
        response.raise_for_status()
        
        soup = BeautifulSoup(response.text, "html.parser")
        
        # Remove navigation, ads, scripts
        for tag in soup(["nav", "header", "footer", "script", 
                         "style", "aside"]):
            tag.decompose()
        
        text = soup.get_text(separator="\n", strip=True)
        
        # Validate: if text is too short, probably a paywall/JS page
        if len(text) < 200:
            return f"[Could not extract content from {url}]"
        
        return text[:max_chars]
    except Exception as e:
        return f"[Failed to scrape {url}: {str(e)}]"

The [Could not extract content] and [Failed to scrape] markers are important: they become visible in the summaries, alerting the report chain that some sources were unavailable. The report can then note “Based on X of Y sources” rather than silently producing a report from incomplete data.

Rate Limiting and Cost Control

The .map() operator sends all requests simultaneously, which can hit API rate limits. The DuckDuckGo search API is particularly sensitive to burst traffic. Fixes:

1. Reduce NUM_SEARCH_RESULTS_PER_QUERY from 3 to 2
2. Use batch() instead of map() for rate-limited execution
3. Add a configurable delay between search calls
4. Track per-request costs and alert when a single query exceeds budget thresholds

Configuration Constants and Cost Awareness

Three constants directly control the cost-quality tradeoff:

NUM_SEARCH_QUERIES = 2          # Queries generated by Chain 2
NUM_SEARCH_RESULTS_PER_QUERY = 3  # URLs per query from Chain 3
RESULT_TEXT_MAX_CHARACTERS = 10000  # Max chars scraped per page

Performance Profiling

Understanding where time is spent helps optimize the pipeline:

import time

profiler = {}
start = time.time()

# Stage 1: Classification (~2-3s)
assistant = assistant_instructions_chain.invoke(question)
profiler["classification"] = time.time() - start

# Stage 2: Query generation (~1-2s)
t = time.time()
queries = web_searches_chain.invoke(assistant)
profiler["query_generation"] = time.time() - t

# Stage 3: Search + summarize (~15-25s, parallelized)
t = time.time()
summaries = search_and_summarization_chain.map().invoke(queries)
profiler["search_and_summarize"] = time.time() - t

# Stage 4: Report generation (~5-8s)
t = time.time()
report = report_chain.invoke(summaries)
profiler["report_generation"] = time.time() - t

total = sum(profiler.values())
for stage, elapsed in sorted(profiler.items(), key=lambda x: -x[1]):
    print(f"  {stage}: {elapsed:.1f}s ({elapsed/total*100:.0f}%)")

Typical output:

  search_and_summarize: 18.2s (64%)
  report_generation: 6.1s (22%)
  classification: 2.8s (10%)
  query_generation: 1.2s (4%)

The search-and-summarize stage dominates because it involves network I/O (web scraping) plus multiple LLM calls. The .map() parallelization already reduces this from ~54 seconds (sequential) to ~18 seconds (parallel). Further optimisation targets: caching (same URL scraped once across queries), async scraping (aiohttp instead of requests), and content extraction (readability algorithms to reduce text volume).

Adapting for enterprise: the internal knowledge assistant

The research engine pattern transfers directly to enterprise knowledge base applications. The adaptation is surprisingly minimal because the architecture cleanly separates the retrieval mechanism from the composition logic:

The LCEL chains remain identical. Only the data source components change. You also add: authentication headers for internal APIs, metadata filtering for access control (users see only documents they are authorized to access), citation tracking for compliance requirements, and audit logging for regulatory environments.

The assistant selection chain (Chain 1) expands to cover enterprise domains: financial analysis, legal review, technical assessment, HR policy, sales intelligence. Each domain gets its own prompt template with domain-specific instructions, output formats, and compliance requirements. A financial analysis persona includes: “Always include data sources and dates. Flag any projections with confidence levels. Include regulatory disclaimers.” A legal review persona includes: “This is not legal advice. Note jurisdictional limitations. Cite specific statutes by number.”

The limits of lines: where the linear route fails

The research engine works. It produces reports. But it has a flaw that only surfaces under pressure, and understanding this flaw motivates the entire next chapter.

Imagine a user asks: “What is the current political situation in Moldova?” The query generation chain produces search queries. The web search returns results. But the results are from 2022, before a recent election that changed the political landscape. The summarization chain dutifully summarizes outdated information. The report chain synthesizes a confident, well-written, completely obsolete report.

In the current architecture, the pipeline marches forward regardless of intermediate quality. It cannot pause after the summarization step, evaluate whether the summaries are recent and relevant, and if they are not, generate different queries targeting more recent sources.

What you want is: search, evaluate the results, and if they are insufficient, generate better queries and search again. This is a loop. And loops cannot be expressed in a linear chain. The pipe operator flows in one direction. There is no pipe-backwards.

The specific adaptations needed: relevance evaluation (do summaries actually answer the question?), recency checking (are sources current enough?), sufficiency verification (is there enough content for a good report?), and retry with refinement (feed evaluation results back to query generation). None of these can be expressed in a linear chain. All of them are natural in a graph with conditional edges and cyclical paths.

This is exactly what the relevant section builds: the research assistant refactored into a LangGraph workflow with a self-improvement loop that evaluates search quality and retries when results are poor.

Part II: Route evidence, not confidence

Grounding quality is governed upstream. The useful unit is not “a vector database”; it is the evidence route from source, through index and query, into a claim.

Build the retrieval foundation

What RAG actually solves: three problems with vanilla llms

Before RAG existed, using an LLM for domain-specific questions had three fundamental problems:

Problem 1: The knowledge cutoff. LLMs are trained on data up to a specific date. GPT-4’s training data ended in April 2024. Ask it about events after that date and it either says “I don’t know” (good) or confidently fabricates an answer (dangerous). RAG solves this by providing current documents as context, bypassing the training cutoff entirely.

Problem 2: Private data. An LLM has never seen your company’s internal documents, your proprietary database, your customer records, or your product specifications. It cannot answer questions about data that was not in its training set. RAG solves this by retrieving from your private knowledge base and including the relevant passages in the prompt.

Problem 3: Hallucination. When an LLM does not have the answer, it often generates a plausible-sounding but incorrect response. This is especially dangerous in domains where accuracy is critical: healthcare, legal, financial, and compliance. RAG reduces hallucination by grounding the LLM’s response in specific retrieved documents, and the hallucination-safe prompt further instructs the LLM to say “I don’t know” rather than guess.

These three problems are why RAG has become the most widely adopted LLM application pattern in enterprise deployments. It transforms a general-purpose language model into a domain-specific expert that answers from your data, not from its training data.

The two stages of RAG

RAG operates in two sequential stages, and both must use the same embedding model:

Stage 1: Content Ingestion (Build the Library)

Before you can search, you must prepare the knowledge base:

1. Extract text from source documents (PDFs, web pages, databases)
2. Split text into chunks small enough for embedding and retrieval
3. Embed each chunk into a numerical vector using an embedding model
4. Store the chunks and their vectors in a vector store

Stage 2: Question Answering (Search the Library)

When a user asks a question:

1. Embed the question using the same embedding model
2. Retrieve the most similar chunks from the vector store
3. Augment the prompt with the retrieved chunks as context
4. Generate an answer using the LLM, grounded in the retrieved context

The critical rule: both stages must use the same embedding model. If you embed documents with OpenAI’s text-embedding-3-small (1,536 dimensions) but embed queries with Chroma’s default model (384 dimensions), the vectors exist in different mathematical spaces. Similarity search becomes meaningless, like comparing temperatures in Celsius to distances in kilometers.

Embeddings: meaning as geometry

The Analogy: A Library Organized by Meaning

Imagine a library where books are shelved not by title, author, or Dewey Decimal number, but by meaning. A book about grief sits next to a book about loss, even if one is a novel and the other a psychology textbook. A book about cooking Italian food sits near a book about Mediterranean cuisine, even though their titles share no words.

This is what embeddings do for text. An embedding converts a piece of text into a point in high-dimensional space (a vector of numbers), positioned so that semantically similar texts are geometrically close together.

The word “king” might be represented as the vector [0.2, -0.4, 0.7, …] with 1,536 dimensions. The word “queen” would be a nearby point: [0.3, -0.3, 0.6, …]. The word “bicycle” would be far away: [-0.8, 0.5, -0.2, …]. The distances between these points encode semantic relationships.

Why This Matters for RAG

When a user asks “What activities can I do at the ruins?”, the embedding model converts this question into a vector. The vector store finds the chunk whose vector is closest to the question’s vector. That chunk might say “Visitors can walk among the ruins and visit the on-site museum.” The words “activities” and “walk among” share zero words, but their embeddings are close because the embedding model learned that activities and walking are semantically related.

This is the fundamental advantage over traditional keyword search (which requires exact word matches) and the foundational insight that makes RAG possible. Keyword search for “activities” would miss a document containing only “walk” and “visit.” Embedding search finds it because meaning, not spelling, determines proximity.

Embedding Models: The Practical Details

OpenAI’s text-embedding-3-small produces 1,536-dimensional vectors. Each piece of text becomes a list of 1,536 floating-point numbers. These numbers have no individual meaning (dimension 742 does not mean “happiness”), but their collective pattern encodes the text’s semantic content.

ChromaDB’s default model (all-MiniLM-L6-v2) produces 384-dimensional vectors. It is free, runs locally, and produces good results for many use cases. OpenAI’s model costs money per API call but generally produces higher-quality embeddings, especially for domain-specific or nuanced content.

A Concrete Numerical Walkthrough

Trace what happens when you embed two sentences and compare them. For illustration, imagine embeddings are 3-dimensional instead of 1,536-dimensional (the math is identical, just easier to visualize):

Sentence A: “The ancient temples of Paestum are well-preserved.” Embedding: [0.8, 0.3, -0.1]

Sentence B: “Paestum contains three Doric temples from the 5th century BC.” Embedding: [0.7, 0.4, -0.2]

Sentence C: “The best pizza in Naples uses San Marzano tomatoes.” Embedding: [-0.3, 0.1, 0.9]

Cosine similarity between A and B: 0.98 (very similar, both about Paestum temples) Cosine similarity between A and C: 0.12 (very different, temples vs. pizza) Cosine similarity between B and C: 0.08 (very different, temples vs. pizza)

When the user asks “Tell me about the temples,” that question gets embedded to approximately [0.75, 0.35, -0.15]. The vector store computes cosine similarity against all stored chunks and returns Sentence B (0.97 similarity) as the top result, followed by Sentence A (0.95). Sentence C (0.10) is not returned because it is far away in embedding space.

This is the entire mechanism of semantic search: convert text to numbers, compare numbers with cosine similarity, return the closest matches. The control point is the embedding model that learned to position semantically similar text at nearby points in 1,536-dimensional space, trained on billions of text pairs.

The Embedding Dimension Mismatch Trap

The most common RAG setup mistake deserves its own callout: embedding dimensions must match between ingestion and query. If you embed documents with OpenAI’s model (1,536 dimensions) but query with Chroma’s default (384 dimensions), the vectors exist in different mathematical spaces. Computing cosine similarity between a 1,536-dimensional vector and a 384-dimensional vector is more than inaccurate; it is mathematically undefined.

This mistake is especially common when: you prototype with Chroma’s free default embeddings, then switch to OpenAI for better quality but forget to re-embed the stored documents; or when you copy code from a tutorial that uses a different model than your ingestion pipeline.

The fix: always specify the embedding model explicitly in both ingestion and query code, and add a validation check that the stored embedding dimensions match the query embedding dimensions.

Text splitting: the hidden quality lever

Before text reaches the vector store, it must be split into chunks. This step is often treated as an afterthought, but chunk quality determines RAG accuracy more than model quality. A perfect LLM cannot answer correctly if the retrieved chunk is a fragment that splits a key sentence in half.

Why Split at All?

Two reasons. First, embedding models have input limits (typically 512-8,192 tokens). A 50-page document cannot be embedded as a single vector. Second, retrieval precision: a small, focused chunk about “Doric temples” will match a question about temples better than a large chunk that also discusses restaurants, hotels, and transportation.

The Chunking Tradeoff

Small chunks (200 characters): high precision (each chunk is focused on one topic), but low context (the chunk may be too short to contain a complete answer). Good for specific factual questions.

Large chunks (2,000 characters): high context (each chunk contains complete paragraphs with surrounding information), but low precision (the chunk may be about multiple topics, diluting the embedding). Good for questions requiring understanding of context.

Medium chunks (500-1,000 characters): the balanced default for most applications. A 500-character chunk typically contains 2-3 complete sentences, enough for both a focused embedding and a complete answer.

Chunk Overlap: Preventing Boundary Losses

Without overlap, a key sentence at the boundary of two chunks gets split: “The Temple of Hera” ends chunk 1, “was built around 550 BC” starts chunk 2. Neither chunk contains the complete fact.

With overlap (typically 10-20% of chunk size), the end of chunk N overlaps with the beginning of chunk N+1. The complete sentence appears in at least one chunk. The cost: some text is embedded and stored twice, slightly increasing storage and embedding costs.

# Recommended starting configuration
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,       # Characters per chunk
    chunk_overlap=100,    # Overlap between chunks
    separators=["\n\n", "\n", ". ", " ", ""]  # Split hierarchy
)

The RecursiveCharacterTextSplitter tries to split at paragraph boundaries first (\n\n), then line breaks, then sentences, then words, and finally characters as a last resort. This hierarchy preserves semantic units whenever possible.

Vector stores: the search infrastructure

A vector store is a database optimized for similarity search. It stores text chunks alongside their vector representations and returns the most semantically similar chunks for a given query vector.

How Similarity Search Works

The most common similarity metric is cosine similarity, which measures the angle between two vectors. Two vectors pointing in the same direction have cosine similarity of 1.0 (identical meaning). Two vectors pointing in opposite directions have cosine similarity of -1.0 (opposite meaning). Perpendicular vectors have cosine similarity of 0.0 (unrelated).

ChromaDB reports cosine distance (1 minus cosine similarity), so lower numbers mean more similar: 0.0 is identical, 2.0 is opposite. When you see a distance of 0.766 for the top result and 1.336 for the third result, the first is explicitly more relevant.

For searching among millions of vectors, exact nearest-neighbor search (comparing against every stored vector) is too slow. Vector stores use Approximate Nearest Neighbor (ANN) algorithms, primarily HNSW (Hierarchical Navigable Small World), that trade tiny accuracy losses for massive speed gains: O(log N) search time instead of O(N).

ChromaDB: The Fastest Path to Working RAG

ChromaDB provides the simplest path from zero to working RAG. Install, create a collection, add documents, and search in under 20 lines:

import chromadb

# Create an in-memory client (data lost on restart)
chroma_client = chromadb.Client()

# Create a collection (like a table in SQL)
tourism_collection = chroma_client.create_collection(
    name="tourism_collection")

# Add documents with metadata and unique IDs
tourism_collection.add(
    documents=[
        """Paestum, Greek Poseidonia, is an ancient city 
        originally a Greek colony founded in the 7th century BC, 
        home to some of the best-preserved major Greek temples.""",
        
        """Poseidonia was probably founded about 600 BC by Greek 
        settlers from Sybaris. The city was conquered by the 
        Lucanians about 400 BC and became a Roman colony, 
        under the name Paestum, in 273 BC.""",
        
        """The ancient Greek part of Paestum contains three 
        well-preserved Doric temples. The oldest temple is the 
        Temple of Hera I, built around 550 BC. The Temple of 
        Athena dates from about 500 BC. The Temple of Hera II 
        was built about 460 BC."""
    ],
    metadatas=[
        {"source": "https://www.britannica.com/place/Paestum"},
        {"source": "https://www.britannica.com/place/Paestum"},
        {"source": "https://www.britannica.com/place/Paestum"}
    ],
    ids=["paestum-br-01", "paestum-br-02", "paestum-br-03"]
)

ChromaDB automatically generates embeddings when you pass documents as strings. It uses its default model (all-MiniLM-L6-v2, 384 dimensions). No embedding API calls, no cost, no setup.

Searching: Seeing Similarity in Action

results = tourism_collection.query(
    query_texts=["How many Doric temples are in Paestum?"],
    n_results=3)

for i in range(3):
    print(f"Rank {i+1} (distance: {results['distances'][0][i]:.3f}):")
    print(f"  {results['documents'][0][i][:80]}...")

Output:

Rank 1 (distance: 0.766):
  The ancient Greek part of Paestum contains three well-preserved Doric temples...
Rank 2 (distance: 0.895):
  Paestum, Greek Poseidonia, is an ancient city originally a Greek colony...
Rank 3 (distance: 1.336):
  Poseidonia was probably founded about 600 BC by Greek settlers from Sybaris...

This output reveals several important properties of semantic search:

Ranking by meaning, not keywords. The top result contains the answer (“three well-preserved Doric temples”) despite the question using “How many” while the chunk uses “three.” The embedding model learned that “How many” questions and numeric answers are semantically related.

Partial relevance decreases gracefully. The second result (distance 0.895) is about Paestum generally, not specifically about temples. It is related but less directly relevant. The third result (distance 1.336) is about Paestum’s founding history, which is topically adjacent but not answering the question. The distances reflect this hierarchy.

No keyword matching required. A traditional keyword search for “How many Doric temples” would match any document containing those exact words. Semantic search matches documents that contain the answer to the question, even if the answer uses completely different words.

Using OpenAI Embeddings Instead of Defaults

ChromaDB’s default embeddings are free and good, but OpenAI’s embeddings are generally higher quality for nuanced queries. To use them:

from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction

openai_ef = OpenAIEmbeddingFunction(
    api_key=os.getenv("OPENAI_API_KEY"),
    model_name="text-embedding-3-small"
)

# Create collection with OpenAI embeddings
collection = chroma_client.create_collection(
    name="tourism_openai",
    embedding_function=openai_ef
)

The RAG pipeline as a diagnostic framework

When a RAG system gives a wrong answer, the three-function decomposition from this section provides a systematic diagnostic framework. Every wrong answer is caused by a failure in one of three stages:

Diagnosis 1: Retrieval Failure (Most Common, ~70% of Issues)

Symptom: The retrieved chunk does not contain the information needed to answer the question.

Test: Print the retrieved chunk. Does it contain the answer? If not, retrieval failed.

Causes: - Chunk too large (the relevant sentence is diluted by irrelevant surrounding text) - Chunk too small (the relevant sentence was split across two chunks) - Wrong embedding model (the model does not capture the semantic relationship) - Missing metadata filter (retrieving from the wrong document category) - Insufficient n_results (the answer was in result 4 but you only retrieved 3)

Fixes (in order of impact): 1. Adjust chunk size and overlap 2. Add metadata filtering 3. Increase n_results 4. Upgrade embedding model 5. Add multi-query retrieval (the relevant section)

Diagnosis 2: Augmentation Failure (~20% of Issues)

Symptom: The retrieved chunk is correct, but the prompt does not guide the LLM to use it properly.

Test: Read the complete prompt (question + context). Is the answer explicitly present? Is the instruction clear?

Causes: - No hallucination-safe instructions (LLM uses training knowledge instead of context) - Context too long (LLM loses the relevant passage in a wall of text) - Conflicting information in multiple retrieved chunks - Poor prompt formatting (question and context not explicitly separated)

Fixes: 1. Use the hallucination-safe prompt template 2. Limit context length (use fewer, more relevant chunks) 3. Add conflict resolution instructions in the prompt 4. explicitly format the prompt with labeled sections

Diagnosis 3: Generation Failure (~10% of Issues)

Symptom: The retrieved chunk is correct, the prompt is well-formed, but the LLM’s answer is wrong.

Test: Show the prompt to a human. Can the human answer correctly from the provided context? If yes but the LLM cannot, it is a generation failure.

Causes: - Model too weak for the task (the pinned low-cost model struggles with complex reasoning) - Context requires multi-step inference that the model cannot perform - Answer requires numerical computation that the model handles poorly

Fixes: 1. Upgrade the model (the pinned low-cost model to the pinned capable model-mini or the pinned capable model) 2. Add chain-of-thought instructions (“Think step by step”) 3. Break complex questions into sub-questions (the relevant section)

This diagnostic framework applies to every RAG system, from the from-scratch implementation in this section to the advanced multi-query, multi-store systems in Chapters 8-10. When something goes wrong, always diagnose in order: retrieval first, then augmentation, then generation.

Building RAG from scratch: three functions

The complete RAG pipeline is three function calls. Building it from scratch ensures you understand what every LangChain abstraction hides later in the edition. When the LangChain version misbehaves, you can reason about which of these three steps is failing.

Function 1: Retrieve

def query_vector_database(question):
    results = tourism_collection.query(
        query_texts=[question],
        n_results=1)
    return results['documents'][0][0]

The double indexing [0][0] is because query() supports batch queries: the outer [0] selects the first query’s results (we sent one query), and the inner [0] selects the first (most relevant) document. in an operating environment, retrieve n_results=3 or more for richer context.

The n_results parameter directly controls the quality-cost tradeoff. With n_results=1, you get a relevant chunk: fast, cheap, but brittle (if the top result is slightly off-topic, the answer fails). With n_results=5, you get five chunks: more context for the LLM, more likely to contain the answer, but more tokens consumed and higher cost. The typical production range is 3-5 chunks. Start with n_results=4 and adjust based on answer quality.

Understanding the results structure:

results = tourism_collection.query(
    query_texts=["How many Doric temples?"],
    n_results=3)

# results is a dict with keys:
# 'ids': [['paestum-br-03', 'paestum-br-01', 'paestum-br-02']]
# 'distances': [[0.766, 0.895, 1.336]]
# 'documents': [['The ancient Greek...', 'Paestum, Greek...', 'Poseidonia...']]
# 'metadatas': [[{'source': 'https://...'}, ...]]

Each field is a list of lists (supporting batch queries). The results are ordered by similarity: index 0 is the closest match.

Function 2: Augment

def prompt_template(question, context):
    return f'Read the following text and answer this question: \
    {question}. \nContext: {context}'

This is the simplest possible augmentation: concatenate the question and the retrieved context into a single prompt. The LLM receives both and generates an answer informed by the context.

In practice, you would concatenate multiple retrieved chunks, add source attribution, and include the hallucination-safe instructions. The augmentation step is where prompt engineering later in the edition directly applies to RAG quality.

Function 3: Generate

def execute_llm_prompt(prompt_input):
    response = openai_client.chat.completions.create(
        model='the pinned low-cost model',
        messages=[
            {"role": "system", 
             "content": "You are an assistant for question-answering tasks."},
            {"role": "user", "content": prompt_input}
        ])
    return response

The Complete Chatbot: R-A-G in Three Lines

def my_chatbot(question):
    context = query_vector_database(question)       # R: Retrieve
    prompt = prompt_template(question, context)      # A: Augment
    response = execute_llm_prompt(prompt)            # G: Generate
    return response.choices[0].message.content

Three function calls. Retrieve the relevant context. Augment the prompt. Generate the answer. This is the entire RAG pattern. Every framework, library, and production system is a variation on these three steps.

Metadata: The Underappreciated Feature

When we added documents to ChromaDB, we included metadata:

metadatas=[
    {"source": "https://www.britannica.com/place/Paestum"},
    {"source": "https://www.britannica.com/place/Paestum"},
    {"source": "https://www.britannica.com/place/Paestum"}
]

This metadata travels with the chunk through the entire pipeline. When you retrieve a chunk, you also get its metadata. This enables:

Source attribution: “According to Britannica (https://www.britannica.com/place/Paestum), the Temple of Hera I was built around 550 BC.” Without metadata, the answer has no provenance.

Metadata filtering: in an operating environment, you might store documents from multiple departments. A user in the finance department should only search finance documents. Metadata filtering restricts the search: where={"department": "finance"}.

Debugging: When the chatbot gives a wrong answer, metadata tells you which source document produced the retrieved chunk. You can check: was the chunk relevant? Was the source reliable? Was the source outdated?

Always include meaningful metadata during ingestion. At minimum: source URL or file path, ingestion date, and document type. For operating systems, add: author, department, access level, version, and any domain-specific fields.

The hallucination problem: when RAG lies

Testing with “How many columns do the three temples have?” reveals RAG’s most dangerous failure mode. The answer is not in the retrieved context (the chunks describe temples but do not mention column counts). The basic prompt produces a confident, detailed, and completely fabricated answer about column counts.

This is hallucination: the LLM generates plausible-sounding information not supported by the provided context. The danger is amplified in RAG because the user trusts the answer more (they believe it comes from their documents) while the LLM is actually drawing from training data or inventing entirely.

Why Hallucination Happens in RAG

Three scenarios trigger hallucination:

Scenario 1: The answer is not in the context. The user asks about column counts but the retrieved chunk only describes temples generally. The LLM fills the gap with training knowledge (“Doric temples typically have 6 columns on the short side and 13 on the long side”). This is the most common scenario and the most dangerous because the fabricated answer sounds authoritative.

Scenario 2: The context is ambiguous. The retrieved chunk says “The temples are among the best preserved in the world.” The user asks “Are these the best preserved Greek temples?” The LLM may answer “Yes” even though “among the best” is not the same as “the best.” Subtle but important in legal, medical, and compliance contexts.

Scenario 3: Multiple chunks conflict. One chunk says “founded in the 7th century BC” and another says “founded about 600 BC.” These are actually consistent (600 BC is in the 7th century BC), but if chunks genuinely conflict, the LLM might synthesize a response that averages the conflicting claims, producing a statement that matches neither source.

The Fix: The Hallucination-Safe Prompt

def prompt_template(question, text):
    return f'''Use the following pieces of retrieved context to \
answer the question. Only use the retrieved context to answer \
the question. If you don't know the answer, or the answer is \
not contained in the retrieved context, just say that you don't \
know. Use three sentences maximum and keep the answer concise.
Question: {question}
Context: {text}
Remember: if you do not know, just say: I do not know. 
Do not make up an answer.
Answer:'''

Four key phrases prevent hallucination:

• “Only use the retrieved context” blocks training knowledge from leaking in
• “If you don’t know, just say that you don’t know” gives explicit permission to not answer
• “Do not make up an answer” is a direct anti-hallucination instruction
• “Use three sentences maximum” prevents verbose hedge-filled responses that obscure uncertainty

This prompt comes from the LangChain Hub’s rlm/rag-prompt template, one of the most battle-tested RAG prompts available. With this prompt, the column question correctly returns “I do not know” instead of a fabricated answer.

The Tradeoff: Safety vs. Helpfulness

The hallucination-safe prompt has a cost: it may refuse to answer questions it could answer correctly. If the retrieved context contains indirect evidence (the chunk mentions “three temples” and the question asks “more than two?”), the safe prompt might say “I don’t know” rather than inferring “yes, three is more than two.”

In most production deployments, this tradeoff favors safety. A “I don’t know” response is always better than a fabricated answer in healthcare, legal, financial, and compliance domains. In consumer-facing chatbots where helpfulness matters more, you can soften the prompt: replace “only use the retrieved context” with “primarily use the retrieved context, and explicitly mark any information from general knowledge.”

The RAG quality hierarchy: what to fix first

When RAG answers are wrong, developers often blame the LLM and reach for a more expensive model. But the hierarchy of impact is:

1. Chunking strategy (highest impact). Bad chunks produce bad retrieval regardless of everything else. Fix: appropriate chunk size (500-1,000 characters for most use cases), overlap (10-20%), and semantic boundary detection (RecursiveCharacterTextSplitter over TokenTextSplitter).

2. Retrieval parameters. Wrong n_results (too few chunks miss the answer, too many dilute relevance). Missing metadata filtering (retrieving from wrong document categories). Fix: test with different n_results values, add metadata filtering.

3. Prompt engineering. The hallucination-safe prompt vs. the basic prompt is the difference between 5% and 30% hallucination rates. Fix: always use the safe prompt as default.

4. Embedding model. OpenAI’s text-embedding-3-small produces better embeddings than Chroma’s default for most use cases, especially for domain-specific content. Fix: upgrade embedding model and re-embed all content.

5. LLM model (lowest impact). Switching from the pinned low-cost model to the pinned capable model improves answer quality only when the retrieved context is already relevant. If retrieval is wrong, a better LLM just generates a more eloquent wrong answer. Fix: only upgrade after optimizing steps 1-4.

This hierarchy saves money and time: fix the cheap, high-impact issues first.

Interpreting distance scores: the trap that misleads

Different vector stores report similarity differently, and confusing them causes subtle bugs:

Cosine distance (ChromaDB default): 0 = identical, 2 = opposite. Lower is better.

Cosine similarity: 1 = identical, -1 = opposite. Higher is better.

Euclidean distance (L2): 0 = identical, unbounded above. Lower is better.

When you see a distance of 0.766 in ChromaDB, that means “moderately similar.” If you see 0.766 in a system reporting similarity (not distance), it means something entirely different. Always verify your store’s convention before interpreting scores.

A practical threshold guide for cosine distance:

These thresholds are approximate and vary by embedding model and domain. Calibrate for your specific use case by testing with known relevant and irrelevant queries.

Use abstractions without hiding failure

The LangChain object model: two stages, six components

LangChain organizes RAG into two stages, each with specific component families. Understanding this object model is essential for reading, writing, and debugging any LangChain RAG application.

The object model is organized into class hierarchies beginning with abstract base classes from which multiple concrete implementations derive. This means every loader, regardless of format (PDF, HTML, CSV, database), implements the same .load() method returning list[Document]. Every vector store, regardless of backend (ChromaDB, Pinecone, FAISS, pgvector), implements the same .add_documents() and .similarity_search() methods. This uniformity is what makes components swappable.

Stage 1: Content Ingestion Components

Stage 2: Q&A Components

The power of these abstractions: you can swap any component without changing the rest. Replace ChromaDB with Pinecone? Change one class. Replace OpenAI embeddings with Cohere? Change one class. Replace the pinned low-cost model with a pinned model? Change one class. Everything else stays identical.

The Abstract Base Class Pattern

Every component family follows the same design: an abstract base class defines the interface, and concrete implementations handle the specifics:

# BaseLoader defines: .load() → list[Document]
# WikipediaLoader implements: .load() queries Wikipedia API
# PyPDFLoader implements: .load() parses PDF pages
# TextLoader implements: .load() reads text files

# VectorStore defines: .add_documents(), .similarity_search()
# Chroma implements: stores in ChromaDB
# Pinecone implements: stores in Pinecone
# FAISS implements: stores in Facebook's FAISS library

This is classic object-oriented polymorphism applied to LLM infrastructure. If you understand one loader, you understand all loaders. If you understand one vector store, you understand all vector stores. The interface is identical; the implementation varies.

Alternative RAG implementations: retrievalqa

LangChain provides a specialized RetrievalQA class that wraps the retrieve-augment-generate pattern in a single call:

from langchain.chains import RetrievalQA

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=retriever,
    return_source_documents=True
)

result = qa_chain.invoke({"query": "Tell me about the temples"})
print(result["result"])           # The answer
print(result["source_documents"]) # The retrieved chunks

The chain_type parameter controls how retrieved chunks combine:

The explicit LCEL chain from the earlier section is equivalent to chain_type="stuff" and is preferred for most applications because it is more composable, more transparent (each component visible in LangSmith traces), and aligns with LangChain’s modern design direction. Use RetrievalQA for quick prototyping; use explicit LCEL for an operating deployment.

Production RAG architecture: beyond the tutorial

An operating RAG system adds layers not shown in the tutorial code:

The tutorial pipeline (retrieval, prompt, generation) is the core. Production adds:

Input validation: Reject queries that are too long, contain injection attacks, or violate content policies.

Query enhancement: Fix typos, expand abbreviations, add synonyms for better retrieval.

Context ranking: After retrieving top-K chunks, re-rank them using a cross-encoder model (more accurate than embedding similarity but too slow for initial retrieval).

Output validation: Check the generated answer for hallucination (does the answer’s content appear in the retrieved context?), safety (does the answer contain harmful content?), and formatting (does the answer follow the requested format?).

These production layers are covered in Chapters 8-10 (query enhancement, context ranking) and the relevant section (input/output validation, guardrails, evaluation).

A thought experiment: choosing the right abstraction level

You have three options for building RAG:

Option A: From scratch (the relevant section). Raw ChromaDB API, raw OpenAI API, manual prompt construction. Full control, full understanding, maximum effort for each change.

Option B: LangChain abstractions (the relevant section). Composable components, swappable providers, LangSmith tracing, conversation memory. Moderate control, good abstraction, minimum effort for changes.

Option C: High-level frameworks (LangChain RetrievalQA, LlamaIndex). One function call produces answers. Minimal control, maximum abstraction, zero effort for standard use cases.

The right choice depends on your situation:

Most production applications use Option B: the canonical LCEL chain gives maximum composability with good debugging (via LangSmith). Teams that start with Option C often migrate to Option B when they need more control over retrieval quality, conversation management, or error handling.

Multi-source ingestion: loading any format

worked RAG systems ingest from diverse sources. A company knowledge base might include Wikipedia articles, PDF reports, Word documents, plain text files, and web pages. LangChain provides a loader for each format, and they all produce the same Document object.

The Loader Zoo: One Interface, Many Formats

from langchain_community.document_loaders import (
    AsyncHtmlLoader, PyPDFLoader, TextLoader, 
    WikipediaLoader, Docx2txtLoader, CSVLoader
)

# Web pages (returns raw HTML, use Html2TextTransformer for clean text)
web_loader = AsyncHtmlLoader(["https://en.wikipedia.org/wiki/Paestum"])
web_docs = web_loader.load()

# PDF files (one Document per page, metadata includes page number)
pdf_loader = PyPDFLoader("travel_guide.pdf")
pdf_docs = pdf_loader.load()

# Wikipedia articles (auto-fetches article content)
wiki_loader = WikipediaLoader(query="Paestum", load_max_docs=2)
wiki_docs = wiki_loader.load()

# Word documents (text extracted, formatting lost)
word_loader = Docx2txtLoader("policy.docx")
word_docs = word_loader.load()

# CSV files (one Document per row)
csv_loader = CSVLoader("data.csv", source_column="url")
csv_docs = csv_loader.load()

Every loader produces the same output: a list of Document objects, each containing page_content (the text) and metadata (source information). This uniformity means everything downstream, splitting, embedding, storing, searching, works identically regardless of where the data came from.

The Document Object: LangChain’s Universal Container

Document(
    page_content="Paestum contains three well-preserved Doric temples...",
    metadata={
        "source": "https://en.wikipedia.org/wiki/Paestum",
        "title": "Paestum",
        "language": "en"
    }
)

Metadata flows through the entire pipeline. When you split a Document, each chunk inherits the parent’s metadata. When you retrieve a chunk, you get its metadata alongside the text. This enables source attribution (“According to the Wikipedia article on Paestum…”), filtered retrieval (“search only PDF sources”), and debugging (“this wrong answer came from chunk paestum-br-02 sourced from Britannica”).

Production Ingestion: Error Handling and Monitoring

in an operating environment, some sources will fail. The PDF might be corrupted. The Wikipedia server might be slow. The web page might return a 403. A production pipeline handles each failure gracefully:

import time

def safe_load(loader, source_name):
    """Load with error handling, timing, and logging."""
    start = time.time()
    try:
        docs = loader.load()
        elapsed = time.time() - start
        print(f"Loaded {len(docs)} docs from {source_name} "
              f"in {elapsed:.1f}s")
        return docs
    except Exception as e:
        print(f"ERROR loading {source_name}: {e}")
        return []  # Skip, don't crash

all_docs = []
all_docs.extend(safe_load(
    WikipediaLoader(query="Paestum", load_max_docs=2), "Wikipedia"))
all_docs.extend(safe_load(
    PyPDFLoader("guide.pdf"), "PDF guide"))
all_docs.extend(safe_load(
    AsyncHtmlLoader(["https://visitpaestum.com"]), "Travel site"))

print(f"\nTotal: {len(all_docs)} documents from "
      f"{sum(1 for d in all_docs if d.page_content)} sources")

if not all_docs:
    raise ValueError("No documents loaded! Check sources.")

Ingesting from a Folder

For bulk ingestion, LangChain’s DirectoryLoader processes an entire folder of mixed-format files:

from langchain_community.document_loaders import DirectoryLoader

loader = DirectoryLoader(
    "./documents/",
    glob="**/*.*",  # All files, all subdirectories
    show_progress=True
)
docs = loader.load()

This is particularly useful for ingesting a company’s document repository: point the loader at a folder and it processes PDFs, text files, and other supported formats automatically.

Splitting: The Quality Lever

Splitting is where most RAG quality is won or lost. The splitter determines what units of text become searchable, and poorly split text produces poor retrieval regardless of everything else.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = splitter.split_documents(wiki_docs)
print(f"Split {len(wiki_docs)} documents into {len(chunks)} chunks")

RecursiveCharacterTextSplitter is LangChain’s most versatile splitter. It tries paragraph boundaries first (\n\n), then line breaks (\n), then sentences (.), then words (), then characters (""). This hierarchy preserves semantic units: a chunk should ideally be a complete paragraph, but if a paragraph is too long, it splits at a sentence boundary rather than mid-sentence.

Each chunk inherits the parent document’s metadata, maintaining source attribution throughout the pipeline. If the parent document has metadata={"source": "Wikipedia", "title": "Paestum"}, every chunk from that document carries the same metadata.

Chunk Size and Overlap: The Two Most Important RAG Parameters

These two numbers affect RAG quality more than the choice of embedding model, LLM, or vector store:

chunk_size=500 means each chunk targets approximately 500 characters (~100 words, ~125 tokens). This is the sweet spot for most use cases: large enough to contain a complete thought, small enough for a focused embedding. Smaller chunks (200) are more precise but lose context. Larger chunks (2000) preserve context but dilute embeddings with irrelevant content.

chunk_overlap=100 means consecutive chunks share 100 characters. This prevents the boundary problem: a key sentence at the junction of two chunks appears in full in at least one chunk. The cost is approximately 20% more chunks (and 20% more embedding cost), which is almost always worth the quality improvement.

The rule of thumb: chunk_overlap should be 10-20% of chunk_size. For a 500-character chunk, 50-100 overlap. For a 1000-character chunk, 100-200 overlap.

A Complete Ingestion Walkthrough With Output

Trace a real ingestion to see what happens at each step:

# Step 1: Load from Wikipedia
from langchain_community.document_loaders import WikipediaLoader
docs = WikipediaLoader(query="Paestum", load_max_docs=2).load()
print(f"Loaded {len(docs)} documents")
print(f"First doc: {len(docs[0].page_content)} chars")
print(f"Metadata: {docs[0].metadata}")

Output:

Loaded 2 documents
First doc: 12847 chars
Metadata: {'title': 'Paestum', 'source': 'https://en.wikipedia.org/wiki/Paestum'}

# Step 2: Split into chunks
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=100)
chunks = splitter.split_documents(docs)
print(f"Split into {len(chunks)} chunks")
print(f"Chunk sizes: min={min(len(c.page_content) for c in chunks)}, "
      f"max={max(len(c.page_content) for c in chunks)}, "
      f"avg={sum(len(c.page_content) for c in chunks)//len(chunks)}")

Output:

Split into 32 chunks
Chunk sizes: min=87, max=498, avg=402

Notice: the minimum chunk size (87 characters) is much smaller than 500 because the last section of a document may be shorter than chunk_size. The maximum (498) is slightly under 500 because the splitter respects sentence boundaries rather than cutting at exactly 500 characters.

# Step 3: Embed and store
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_db = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_paestum"
)
print(f"Stored {vector_db._collection.count()} chunks in ChromaDB")

Output:

Stored 32 chunks in ChromaDB

The entire ingestion pipeline: 3 lines of code (load, split, store). The from_documents() method handles embedding all 32 chunks via the OpenAI API and storing them in ChromaDB in a single call. The persist directory ensures data survives Python restarts.

Embedding and Storing: What Happens Under the Hood

When you call Chroma.from_documents(chunks, embeddings), LangChain:

1. Extracts page_content from each Document
2. Calls embeddings.embed_documents([text1, text2, ..., text32]), which sends all 32 texts to the OpenAI API in a batch
3. Receives 32 vectors, each with 1,536 dimensions
4. Calls collection.add(documents=texts, embeddings=vectors, metadatas=metadata_list, ids=auto_generated_ids)
5. ChromaDB stores each chunk with its vector and metadata

Deduplication: Preventing Redundant Chunks

When ingesting from multiple sources about the same topic, the same information often appears in multiple documents. Wikipedia and Britannica both describe Paestum’s temples, producing near-identical chunks.

Without deduplication, the vector store contains redundant entries. The retriever returns four chunks that all say the same thing, wasting context window space and providing no additional information.

A simple deduplication strategy checks for near-duplicate content before adding:

def add_with_dedup(vector_db, new_chunks, threshold=0.1):
    """Add chunks, skipping near-duplicates."""
    added = 0
    skipped = 0
    for chunk in new_chunks:
        existing = vector_db.similarity_search_with_score(
            chunk.page_content, k=1)
        if existing and existing[0][1] < threshold:
            skipped += 1  # Too similar to existing chunk
        else:
            vector_db.add_documents([chunk])
            added += 1
    print(f"Added {added}, skipped {skipped} duplicates")

The threshold=0.1 means chunks with cosine distance less than 0.1 (extremely similar) are considered duplicates. Adjust based on your tolerance for redundancy.

The canonical RAG chain: the pattern The route can forever

This is the most important code in the chapter. Every RAG application you build, from this section through the relevant section, is a variation of this pattern:

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI

# Create retriever from vector store
retriever = vector_db.as_retriever(search_kwargs={"k": 4})

# The hallucination-safe prompt (later in the edition)
prompt = ChatPromptTemplate.from_template("""
Use the following context to answer the question. 
Only use the provided context. If the answer is not in the 
context, say "I don't know."

Context: {context}
Question: {question}
Answer:""")

# The LLM
llm = ChatOpenAI(model="the pinned low-cost model")

# The canonical RAG chain
rag_chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# Use it
answer = rag_chain.invoke("How many temples are in Paestum?")

Dissecting Every Pipe Step

Trace the data flow through each component when the user asks “How many temples are in Paestum?”:

Step 1: {"context": retriever, "question": RunnablePassthrough()}

This is a RunnableParallel. It receives the input string and sends it to two places simultaneously:

• retriever receives “How many temples are in Paestum?”, embeds it, searches the vector store, and returns the top-4 most similar Document objects
• RunnablePassthrough() receives “How many temples are in Paestum?” and forwards it unchanged

The output is a dictionary:

{
    "context": [Document(page_content="The ancient Greek part...three Doric temples..."), 
                Document(page_content="Paestum, Greek Poseidonia..."), 
                Document(page_content="The Temple of Hera I...550 BC..."),
                Document(page_content="Poseidonia was founded...")],
    "question": "How many temples are in Paestum?"
}

Step 2: prompt

The ChatPromptTemplate fills {context} with the retrieved documents (LangChain automatically formats them by joining their page_content fields) and {question} with the original question. The output is a formatted prompt string ready for the LLM.

Step 3: llm

The ChatOpenAI model receives the formatted prompt and generates a response. Because the context contains “three well-preserved Doric temples,” the model answers correctly: “Paestum contains three Doric temples: the Temple of Hera I (550 BC), the Temple of Athena (500 BC), and the Temple of Hera II (460 BC).”

Step 4: StrOutputParser()

Extracts the text content from the LLM’s ChatMessage response object, returning a clean Python string.

Why This Pattern Matters

This four-component pattern is the foundation of every RAG application in the edition. Chapters 8-10 modify what goes into the “context” slot:

Understanding this canonical pattern deeply means you already understand the skeleton of every subsequent chapter. The flesh changes; the skeleton stays.

Querying the Vector Store Directly (Without a Chain)

Before building the chain, you can test retrieval in isolation:

# Direct similarity search (returns Documents)
docs = vector_db.similarity_search(
    "Tell me about the temples", k=3)
for doc in docs:
    print(f"Source: {doc.metadata.get('source', 'unknown')}")
    print(f"Content: {doc.page_content[:100]}...")
    print()

# With scores (reveals distance/similarity values)
docs_with_scores = vector_db.similarity_search_with_score(
    "Tell me about the temples", k=3)
for doc, score in docs_with_scores:
    print(f"Score: {score:.3f} | {doc.page_content[:80]}...")

Testing retrieval separately is essential for debugging. If similarity_search returns irrelevant chunks, the problem is in ingestion (bad chunking, wrong embedding model) or the query (too vague, wrong terminology). No amount of prompt engineering can fix a retrieval that returns the wrong documents.

Conversation memory: from stateless q&a to chatbot

The RAG chain so far is stateless: each question is independent. A user asks “Tell me about the temples in Paestum” and gets a great answer. Then asks “What about their construction dates?” and the system has no idea what “their” refers to. Without memory, every question is a fresh start.

Conversation memory solves this by maintaining chat history across turns.

The Prompt With History

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.schema import HumanMessage, AIMessage

prompt_with_history = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant that answers questions "
               "using the provided context. If the answer is not in "
               "the context, say you don't know."),
    MessagesPlaceholder(variable_name="chat_history"),
    ("human", "Context: {context}\n\nQuestion: {question}")
])

The MessagesPlaceholder is the key: it inserts the entire conversation history into the prompt at that position. The LLM sees the system instruction, then all previous exchanges, then the current question with its retrieved context.

Managing Chat History

from langchain_community.chat_message_histories import ChatMessageHistory

chat_history = ChatMessageHistory()

def ask_with_memory(question):
    # Retrieve context
    context_docs = retriever.invoke(question)
    
    # Build and invoke the chain
    response = (prompt_with_history | llm | StrOutputParser()).invoke({
        "context": context_docs,
        "question": question,
        "chat_history": chat_history.messages
    })
    
    # Update history
    chat_history.add_user_message(question)
    chat_history.add_ai_message(response)
    
    return response

# Conversation flow
print(ask_with_memory("Tell me about the temples in Paestum"))
# → "Paestum contains three well-preserved Doric temples..."

print(ask_with_memory("What are their construction dates?"))
# → "The Temple of Hera I was built around 550 BC, 
#     the Temple of Athena around 500 BC, and the 
#     Temple of Hera II around 460 BC."

The second question works because the chat history contains the first exchange. The LLM reads the history, sees that “their” refers to “the temples in Paestum,” and answers correctly.

A Subtle Bug: Why RunnableLambda Matters for History

A common mistake when integrating memory into LCEL chains deserves special attention because it bites every developer exactly once:

# BAD: Captures messages at construction time (empty list!)
chain = (
    {"context": retriever, 
     "question": RunnablePassthrough(),
     "chat_history": chat_history.messages}  # Evaluated NOW, empty
    | prompt_with_history | llm | StrOutputParser()
)

# GOOD: Wraps in lambda to evaluate at invocation time
from langchain_core.runnables import RunnableLambda

chain = (
    {"context": retriever, 
     "question": RunnablePassthrough(),
     "chat_history": RunnableLambda(lambda _: chat_history.messages)}
    | prompt_with_history | llm | StrOutputParser()
)

The bad version captures chat_history.messages when the chain is constructed. At construction time, the history is empty. Python evaluates the expression and binds the empty list object to the chain definition. Even after adding messages to chat_history, the chain still sees the empty list from construction time.

The good version wraps the history access in a RunnableLambda, which is evaluated at each .invoke() call. Each invocation gets the current messages, not the construction-time snapshot. This is the same RunnableLambda pattern from Chapters 3 and 4, applied to mutable state.

This bug is subtle because the chain works for the first turn (the history should be empty for the first turn!). It only fails on the second turn, when the history should contain the first exchange but the chain still sees an empty list. The symptom: follow-up questions like “What about their dates?” fail because the chain has no memory of the previous exchange.

Without memory, the conversation fails on the second turn:

Turn 1: "Tell me about the temples in Paestum"
→ "Paestum contains three well-preserved Doric temples..." ✓

Turn 2: "What about their construction dates?"
→ "I don't know what you're referring to by 'their'." ✗

With memory, the LLM sees the full conversation:

Turn 1: "Tell me about the temples in Paestum"
→ "Paestum contains three well-preserved Doric temples..." ✓

Turn 2: "What about their construction dates?"
→ (LLM sees Turn 1 in chat_history, resolves "their" to "the temples")
→ "The Temple of Hera I was built around 550 BC, the Temple of
    Athena around 500 BC, and the Temple of Hera II around 460 BC." ✓

The critical detail: the retriever receives “What about their construction dates?” without context. If the retriever cannot find chunks about construction dates using this vague query, the answer will be wrong even with perfect memory. A more sophisticated approach (covered later in the edition) uses the conversation history to rewrite the query before retrieval: “What about their construction dates?” becomes “What are the construction dates of the Doric temples in Paestum?” This query-rewriting technique materially improves retrieval for follow-up questions.

The History Management Problem: Three Strategies With Code

Conversation history grows with each turn. After 20 turns with retrieved context, the prompt might consume 8,000+ tokens. Three strategies handle this:

Strategy 1: Full History (simplest, good for short conversations)

# Just pass all messages, no filtering
def get_history():
    return chat_history.messages

Works for conversations under 10 turns. Token cost grows linearly. At turn 20 with an average response of 200 words, the history alone consumes ~5,300 tokens, leaving less room for retrieved context. Eventually exceeds the context window and fails.

Strategy 2: Sliding Window (practical default)

def get_recent_history(max_turns=5):
    """Keep only the last N exchanges."""
    messages = chat_history.messages
    # Each turn = 1 human message + 1 AI message = 2 messages
    return messages[-max_turns * 2:]

Simple, predictable token consumption (capped at ~2,600 tokens for 5 turns), handles 95% of real conversations. The user rarely references something from 15 turns ago. If they do, the information was probably important enough to exist in the vector store, and the retriever will find it.

Strategy 3: History Summarization (sophisticated, for long conversations)

def get_summarized_history(max_recent=5):
    """Summarize old turns, keep recent ones verbatim."""
    messages = chat_history.messages
    
    if len(messages) <= max_recent * 2:
        return messages  # Short enough, no summarization needed
    
    # Separate old and recent messages
    old_messages = messages[:-max_recent * 2]
    recent_messages = messages[-max_recent * 2:]
    
    # Summarize old messages
    old_text = "\n".join([
        f"{'User' if i % 2 == 0 else 'Assistant'}: {m.content}" 
        for i, m in enumerate(old_messages)
    ])
    summary = llm.invoke(
        f"Summarize this conversation history in 2-3 sentences, "
        f"preserving key facts and topics discussed:\n{old_text}"
    )
    
    # Return summary + recent messages
    return [AIMessage(content=f"[Previous context: {summary.content}]")] \
           + recent_messages

For most production applications, the sliding window is the right default. Start with max_turns=5. If users consistently reference earlier context, increase to 10 or switch to summarization.

Common RAG mistakes and how to fix them

Mistake 1: Different Embedding Models for Ingestion and Query

# BAD: Ingested with OpenAI, querying with Chroma default
vector_db = Chroma.from_documents(chunks, OpenAIEmbeddings())
# Later, in a different script:
vector_db = Chroma(persist_directory="./data")  # Uses default embeddings!
retriever = vector_db.as_retriever()  # Queries with wrong model

The stored vectors are 1,536-dimensional (OpenAI). The query vectors are 384-dimensional (Chroma default). Similarity search produces garbage. The fix: always specify the same embedding model when loading an existing collection:

vector_db = Chroma(
    persist_directory="./data",
    embedding_function=OpenAIEmbeddings()  # Same model as ingestion
)

Mistake 2: Not Testing Retrieval Independently

# BAD: Only testing the full chain
answer = rag_chain.invoke("Tell me about the temples")
print(answer)  # Looks correct, but is it using the right chunks?

# GOOD: Test retrieval separately first
docs = retriever.invoke("Tell me about the temples")
for doc in docs:
    print(f"[{doc.metadata.get('source', '?')}] {doc.page_content[:100]}...")
# Then test the full chain

If the answer is wrong, The design needs whether it is a retrieval problem (wrong chunks) or a generation problem (right chunks, wrong answer). Testing retrieval separately answers this immediately.

Mistake 3: Chunks Too Large or Too Small

# BAD: Chunks of 2000 characters (entire paragraphs)
splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
# Problem: embeddings are diluted, retrieval precision drops

# BAD: Chunks of 100 characters (sentence fragments)
splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)
# Problem: chunks lack context, answers are incomplete

# GOOD: Start with 500, test and adjust
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=100)

The only way to find the right chunk size is to test with your specific content and questions. 500 characters with 100 overlap is a good starting point. Test with 10 representative questions and adjust.

Mistake 4: Ignoring Metadata During Ingestion

# BAD: No metadata
vector_db = Chroma.from_documents(chunks, embeddings)
# Problem: no source attribution, no filtering, no debugging info

# GOOD: Rich metadata from the start
for chunk in chunks:
    chunk.metadata["ingestion_date"] = datetime.now().isoformat()
    chunk.metadata["document_type"] = "policy"
    chunk.metadata["department"] = "engineering"

Metadata costs nothing to store but is invaluable for: source attribution in answers, filtered retrieval (search only engineering policies), debugging (which source produced this wrong chunk?), and freshness checks (is this chunk outdated?).

Mistake 5: No Hallucination Guard in the Prompt

# BAD: Basic prompt that invites hallucination
prompt = "Answer this question: {question}\nContext: {context}"

# GOOD: Hallucination-safe prompt (always use this as default)
prompt = """Use ONLY the following context to answer the question.
If the answer is not in the context, say "I don't know."
Do not make up information.
Context: {context}
Question: {question}"""

The basic prompt allows the LLM to supplement retrieved context with training data. For questions where the context is incomplete, the LLM fills gaps with plausible-sounding but potentially wrong information. The hallucination-safe prompt prevents this by explicitly constraining the LLM to the retrieved context.

The retriever-memory interaction: a subtle quality issue

There is a subtle interaction between conversation memory and retrieval that is not obvious until you encounter it in an operating environment.

Consider this conversation:

Turn 1: "What temples are in Paestum?"
→ Retriever searches: "What temples are in Paestum?" ✓ (clear query)
→ Returns chunks about Doric temples ✓

Turn 2: "How old are they?"
→ Retriever searches: "How old are they?" ✗ (ambiguous query!)
→ Returns chunks about... age? old buildings? elderly people?

The conversation memory helps the LLM understand that “they” means “the temples,” but the retriever does not see the conversation history. The retriever receives the raw question “How old are they?” and searches the vector store for semantically similar chunks. This vague query produces poor retrieval.

The fix is query contextualization: use the conversation history to rewrite the query before sending it to the retriever. This is a preview of the relevant section’s query transformation techniques:

# Contextualize the query using conversation history
contextualize_prompt = ChatPromptTemplate.from_messages([
    ("system", "Given the chat history and latest question, "
               "reformulate the question to be standalone. "
               "Do NOT answer the question."),
    MessagesPlaceholder("chat_history"),
    ("human", "{question}")
])

contextualize_chain = contextualize_prompt | llm | StrOutputParser()

# "How old are they?" → "How old are the Doric temples in Paestum?"
standalone_question = contextualize_chain.invoke({
    "question": "How old are they?",
    "chat_history": chat_history.messages
})

Now the retriever searches for “How old are the Doric temples in Paestum?” which is a clear, standalone query that produces excellent retrieval. This contextualization step adds one LLM call per turn but materially improves retrieval quality for follow-up questions.

This pattern, using the LLM to improve the input before the main processing, appears throughout the Advanced RAG chapters: the relevant section’s multi-query retrieval generates multiple search queries, the relevant section’s HyDE generates a hypothetical answer to search for, and the relevant section’s query routing classifies the query type before selecting the data store. All follow the same principle: invest a small LLM call upfront to materially improve the quality of the main operation.

The retriever: more than default similarity

The vector_db.as_retriever() call hides significant configurability that directly impacts answer quality:

Default: Top-K Similarity

retriever = vector_db.as_retriever(search_kwargs={"k": 4})

Returns the 4 chunks with highest cosine similarity to the query. Simple, fast, effective for most use cases. The k parameter is the most impactful retriever setting: too low (k=1) and you miss relevant context; too high (k=10) and irrelevant chunks dilute the signal. Start with k=4 and adjust based on answer quality.

MMR: Maximum Marginal Relevance

retriever_mmr = vector_db.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 4, "fetch_k": 20}
)

MMR solves a specific problem: when the top-4 results by pure similarity are all from the same paragraph (because the paragraph was split into overlapping chunks), the LLM gets four versions of the same information. MMR retrieves 20 candidates (fetch_k=20), then selects the top 4 that are both relevant to the query AND diverse from each other.

Use MMR when your chunking produces overlapping content (which it should, if you use chunk_overlap). The diversity penalty ensures the LLM gets context from different sections of the knowledge base.

Score Threshold: Quality Gates

retriever_threshold = vector_db.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={"score_threshold": 0.7}
)

Only returns chunks scoring above 0.7 similarity. If no chunk passes the threshold, the retriever returns an empty list, and the LLM correctly responds with “I don’t know” (assuming you use the hallucination-safe prompt). This prevents the failure mode where the system generates an answer from marginally relevant chunks.

The tradeoff: a strict threshold may reject borderline-relevant chunks that actually contain useful information. A lenient threshold may include irrelevant chunks. Calibrate the threshold by testing with known answerable and unanswerable questions.

Choosing Your Retriever

For most applications, start with default similarity. Switch to MMR if your answers seem repetitive. Add score threshold for domains where “I don’t know” is better than a mediocre answer (healthcare, legal, compliance).

Swapping providers: the abstraction payoff in practice

The abstraction layer’s value becomes concrete when you need to change a component. Each swap is two lines of code:

Swap Vector Store (ChromaDB to Pinecone)

# Before: ChromaDB (self-hosted, free)
from langchain_chroma import Chroma
vector_db = Chroma(persist_directory="./data", embedding_function=embeddings)

# After: Pinecone (managed, SOC 2 compliant)
from langchain_pinecone import PineconeVectorStore
vector_db = PineconeVectorStore(index_name="prod", embedding=embeddings)

Everything downstream (retriever, chain, prompt, memory) stays identical.

Swap Embedding Model (OpenAI to Cohere)

# Before: OpenAI (1,536 dimensions, $0.02/M tokens)
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# After: Cohere (1,024 dimensions, different pricing)
from langchain_cohere import CohereEmbeddings
embeddings = CohereEmbeddings(model="embed-english-v3.0")

Warning: Changing the embedding model requires re-embedding all stored documents. You cannot mix embeddings from different models in the same collection.

Swap LLM (GPT to a pinned model)

# Before: OpenAI the pinned low-cost model
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="the pinned low-cost model")

# After: Anthropic a pinned model = ChatAnthropic(model="a pinned model-sonnet-4-20250514")

The Migration Checklist

When migrating any component in an operating environment:

1. Install the new provider package (e.g., pip install langchain-pinecone)
2. Instantiate with the same embedding model (critical for vector stores)
3. Re-ingest all documents if changing vector store or embedding model
4. Update the retriever in your RAG chain (one line)
5. Run your test suite against the new configuration
6. Monitor retrieval quality for the first week

The LangChain abstraction ensures step 4 is literally one line. All downstream code (retriever, chain, prompts) remains identical. This is the power of the abstraction layer: infrastructure decisions become configuration changes, not architectural rewrites.

The complete RAG pipeline: from documents to answers

Assemble the route into a complete, operating RAG chatbot:

import os
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_chroma import Chroma
from langchain_community.document_loaders import WikipediaLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables import RunnablePassthrough, RunnableLambda
from langchain_core.output_parsers import StrOutputParser
from langchain_community.chat_message_histories import ChatMessageHistory

# Configure LangSmith
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_PROJECT"] = "rag-chatbot"

# 1. INGEST
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
docs = WikipediaLoader(query="Paestum", load_max_docs=3).load()
chunks = RecursiveCharacterTextSplitter(
    chunk_size=500, chunk_overlap=100
).split_documents(docs)
vector_db = Chroma.from_documents(chunks, embeddings, 
                                   persist_directory="./chroma_db")

# 2. BUILD CHAIN
retriever = vector_db.as_retriever(search_kwargs={"k": 4})
llm = ChatOpenAI(model="the pinned low-cost model")
chat_history = ChatMessageHistory()

prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer questions using ONLY the provided context. "
               "If unsure, say 'I don't know'."),
    MessagesPlaceholder("chat_history"),
    ("human", "Context: {context}\n\nQuestion: {question}")
])

rag_chain = (
    {"context": retriever, 
     "question": RunnablePassthrough(),
     "chat_history": RunnableLambda(lambda _: chat_history.messages)}
    | prompt | llm | StrOutputParser()
)

# 3. USE
def chat(question):
    answer = rag_chain.invoke(question)
    chat_history.add_user_message(question)
    chat_history.add_ai_message(answer)
    return answer

# Interactive loop
while True:
    q = input("\nYou: ")
    if q.lower() in ("quit", "exit"):
        break
    if q.lower() == "/reset":
        chat_history.clear()
        print("History cleared.")
        continue
    print(f"Bot: {chat(q)}")

This is a complete, operating RAG chatbot in 45 lines. It ingests from Wikipedia, splits intelligently, stores in persistent ChromaDB with OpenAI embeddings, retrieves with the hallucination-safe prompt, maintains conversation memory, traces everything in LangSmith, and provides a terminal REPL with a /reset command.

Index at the granularity of the question

The core problem: one size does not fit all

Naive RAG from Chapters 6-7 uses one embedding per chunk at one fixed chunk size. This forces a tradeoff that becomes painfully visible with a concrete example.

The Tradeoff in Numbers

Consider a Wikivoyage article about Cornwall with 10,000 characters. Here is what happens with different chunk sizes:

Chunk size = 200 characters (50 chunks):

Query: “What is the Cornwall Ranger bus ticket?” The retriever finds a focused 200-char chunk: “The Cornwall Ranger ticket allows unlimited travel on most bus services for one day. It costs £14 for adults.” Distance score: 0.45 (excellent). But the chunk is too short to explain the bus network or connecting routes.

Query: “Tell me about transportation in Cornwall.” The retriever finds three isolated fragments about buses, trains, and ferries. Each is disconnected. The LLM produces a disjointed list rather than a coherent overview.

Chunk size = 2,000 characters (5 chunks):

Query: “Tell me about transportation in Cornwall.” The retriever finds one broad chunk spanning buses, trains, ferries, and driving. Distance score: 0.72 (acceptable). The LLM produces a coherent, broad answer.

Query: “What is the Cornwall Ranger bus ticket?” The retriever finds the same 2,000-char chunk. The Ranger is mentioned in one sentence among 15 others. Distance score: 0.89 (mediocre). The embedding is diluted by five subtopics.

The fundamental insight: No single chunk size serves both types of questions. The four techniques in this section solve this by maintaining multiple representations at different granularities.

The solution is not to choose one size. The solution is to decouple the search representation from the synthesis representation. Search with small, focused embeddings for precision. Return large, contextual documents for synthesis. This “search small, return big” principle is the foundation of every technique in this section.

Technique 1: parentdocumentretriever (search small, return big)

The most impactful single technique in advanced RAG. The architecture uses two stores:

• A vector store containing small child chunks with focused embeddings (for search)
• A document store containing large parent documents with full context (for synthesis)

When a query arrives, the retriever searches the vector store for matching child chunks, then looks up their parent documents in the document store, and returns the parents to the LLM.

The Analogy: Library Index Cards

Think of a library with an old-fashioned card catalog. Each index card (child chunk) contains a brief, focused description: “Doric temple architecture, 550 BC, Hera I.” The card is small enough to match a specific search. But the card references a full book chapter (parent document) with rich context about the temple’s history, construction methods, and cultural significance.

You search the cards (precise matching). You read the chapters (rich context). This is exactly what ParentDocumentRetriever does.

The Two-Store Architecture

Vector store (ChromaDB, Pinecone): Contains child chunk embeddings optimized for fast similarity search. Each child has metadata linking to its parent’s ID.

Document store (InMemoryByteStore, Redis): Contains full parent documents. No embeddings, no search. Just key-value storage: given a parent ID, return the parent text.

Complete Implementation

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryByteStore
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

# Child splitter: small chunks for precise search
child_splitter = RecursiveCharacterTextSplitter(
    chunk_size=200, chunk_overlap=50)

# Parent splitter: large chunks for rich context
parent_splitter = RecursiveCharacterTextSplitter(
    chunk_size=2000, chunk_overlap=200)

# Two stores
vectorstore = Chroma(
    embedding_function=OpenAIEmbeddings(), 
    collection_name="child_chunks")
docstore = InMemoryByteStore()

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=docstore,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter
)

# Ingest: automatically splits into parents and children
retriever.add_documents(documents)

Execution Trace: What Happens Step by Step

When you call retriever.invoke("What year was the Temple of Hera built?"):

Step 1: Embed the query. The question becomes a 1,536-dimensional vector.

Step 2: Search child chunks. The vector store finds the most similar child: “The oldest temple is the Temple of Hera I, built around 550 BC. The Temple of Athena dates from about 500 BC.” (200 chars). Distance: 0.38 (excellent).

Step 3: Look up parent ID. The child’s metadata contains parent_doc_id: "paestum-parent-03".

Step 4: Retrieve parent. The document store returns the 2,000-char parent covering all three temples, their architecture, the on-site museum, visitor tips, and historical context.

Step 5: Return parent to LLM. The LLM receives rich context and produces a broad answer about the Temple of Hera’s construction date, architectural style, and historical significance.

The child chunk served as a precise search index. It was never sent to the LLM. The parent document provided the synthesis context. This separation is the core architectural insight.

Choosing Child and Parent Sizes

Rule of thumb: the child should be small enough to focus on one fact or one sentence. The parent should be large enough to contain the complete context for any question the child might match. A 10:1 ratio (parent 10x larger than child) is a good starting point.

Technique 2: summary embeddings (search the gist)

Instead of embedding raw text, generate an LLM summary of each chunk and embed the summary. The insight: summaries are more focused than raw text. They strip away filler words, transition phrases, and tangential details, producing denser embeddings that match natural language queries better.

Why Summaries Produce Better Embeddings

Consider a 2,000-character chunk about Paestum’s history. The raw text contains: founding date, conquerors, name changes, archaeological timeline, temple descriptions, museum details, visitor hours, and ticket prices. The embedding of this chunk reflects all these topics, diluting the signal for any specific query.

A summary might be: “Paestum is an ancient Greek colony in southern Italy, founded around 600 BC, containing three remarkably well-preserved Doric temples and an archaeological museum.” This 150-character summary captures the essential meaning. Its embedding focuses on the key concepts: Greek colony, Doric temples, archaeological site. Queries about ancient temples, Greek architecture, or Italian archaeological sites match this focused embedding more precisely than the raw 2,000-character text.

Implementation with MultiVectorRetriever

from langchain.retrievers.multi_vector import MultiVectorRetriever
from langchain.storage import InMemoryByteStore
from langchain_core.documents import Document
import uuid

# Summary generation chain
summary_chain = (
    ChatPromptTemplate.from_template(
        "Write a concise 2-3 sentence summary of this text "
        "that captures its key topic and main facts:\n{text}")
    | llm | StrOutputParser()
)

# Generate summaries and link to originals
doc_ids = [str(uuid.uuid4()) for _ in chunks]
summaries = []
for chunk, doc_id in zip(chunks, doc_ids):
    summary_text = summary_chain.invoke({"text": chunk.page_content})
    summaries.append(Document(
        page_content=summary_text,
        metadata={"doc_id": doc_id}
    ))

# Store summaries in vector store (for search)
vectorstore.add_documents(summaries)

# Store originals in doc store (for context)
docstore.mset(list(zip(doc_ids, chunks)))

# Create retriever
retriever = MultiVectorRetriever(
    vectorstore=vectorstore,
    byte_store=docstore,
    id_key="doc_id"
)

When a query matches a summary embedding, the retriever looks up the doc_id in the document store and returns the original full chunk. The summary served as a search index; the original text is what the LLM receives.

When Summary Embeddings Excel

Summary embeddings work best for narrative or technical content where the surface text does not explicitly convey the conceptual meaning. A paragraph describing a complex chemical process might embed poorly as raw text (too many specific terms diluting the embedding), but its summary (“This section describes catalytic methane-to-methanol conversion at 300C using a copper-zeolite catalyst”) produces a focused embedding that matches related queries precisely.

They work less well for factual, concise content that is already essentially a summary of itself (product specifications, FAQ entries, data tables). For such content, the summary adds little value because the original is already focused.

A Concrete Comparison: Summary vs. Raw Retrieval

Consider a chunk about Cornwall’s Eden Project:

Raw chunk (1,500 chars): “The Eden Project, near St Austell, is a popular visitor attraction and educational charity. Opened in 2001, it features two massive biomes: the Rainforest Biome, the world’s largest indoor rainforest, and the Mediterranean Biome with plants from the Mediterranean, South Africa, and California. The outdoor gardens display plants from Cornwall’s own temperate climate. The project was built in a reclaimed kaolinite pit and has become an icon of sustainable architecture. Visitors can explore the biomes, enjoy zip-lining across the pit, attend concerts in the summer, and learn about ecology through interactive exhibits. Tickets cost approximately £30 for adults…”

Summary embedding (150 chars): “The Eden Project is a world-renowned ecological attraction near St Austell, featuring massive biomes with global plant collections and sustainable architecture.”

Query: “Where can I learn about ecology and sustainability in Cornwall?”

Raw embedding distance: 0.78 (the raw text mentions ecology and sustainability, but buried among ticket prices, zip-lining, and biome details that dilute the embedding)

Summary embedding distance: 0.42 (the summary highlights “ecological attraction” and “sustainable architecture” prominently)

The summary-based retrieval finds this chunk more confidently because the summary distills exactly the conceptual essence that matches the query. The raw text’s embedding is diluted by operational details (ticket prices, zip-lining) that are irrelevant to the ecology query.

The Summary Generation Prompt Matters

The quality of the summary directly affects embedding quality. A poor summary prompt produces vague summaries that embed poorly:

# BAD: Vague summary prompt
bad_prompt = "Summarize this text briefly."
# Produces: "This text is about the Eden Project in Cornwall."
# Embedding: too vague to match specific queries

# GOOD: Focused summary prompt
good_prompt = """Write a concise 2-3 sentence summary of this text 
that captures: (1) the main topic, (2) the key facts, and 
(3) the primary themes or concepts discussed."""
# Produces: "The Eden Project is a world-renowned ecological attraction 
# near St Austell featuring massive biomes with global plant collections. 
# Built in a reclaimed pit, it exemplifies sustainable architecture and 
# offers educational exhibits on ecology and environmental conservation."
# Embedding: focused on the conceptual essence

Invest effort in your summary prompt. Test it on 10 diverse chunks and verify that each summary captures the searchable essence of the original. A good summary should make you think “if someone were looking for this content, what would they search for?”

Technique 3: hypothetical question embeddings (bridge the vocabulary gap)

The most capable technique for user-facing Q&A applications. For each chunk, generate 3-5 questions that the chunk could answer, then embed those questions instead of (or alongside) the raw text.

The Vocabulary Gap Problem

Users ask: “When was this temple built?” Documents say: “The Temple of Hera I, constructed circa 550 BCE during the archaic period of Greek architecture, features six columns on its front facade.”

The embedding of the question focuses on: “when,” “built,” “temple.” The embedding of the document focuses on: “Hera I,” “550 BCE,” “archaic,” “columns,” “facade.” The overlap is thin. The question and the answer use fundamentally different vocabulary.

But if you generated the question “When was the Temple of Hera I built?” from the document text and embedded that question, the query “When was this temple built?” would match almost perfectly. Question embeddings match question embeddings far better than question embeddings match declarative text embeddings.

Implementation

question_chain = (
    ChatPromptTemplate.from_template(
        "Generate 4 questions that this text could answer. "
        "Return one question per line, numbered.\n\n"
        "Text: {text}")
    | llm | StrOutputParser()
)

# For each chunk, generate and index questions
for chunk, doc_id in zip(chunks, doc_ids):
    questions_text = question_chain.invoke({"text": chunk.page_content})
    
    # Parse numbered questions
    questions = [q.strip().lstrip("1234. ") 
                 for q in questions_text.split("\n") 
                 if q.strip()]
    
    for question in questions:
        vectorstore.add_documents([Document(
            page_content=question,
            metadata={"doc_id": doc_id}
        )])

For a corpus of 250 chunks, this generates ~1,000 questions (4 per chunk) and creates 1,000 question embeddings in the vector store. Each question links back to its source chunk via doc_id.

The Cost-Quality Tradeoff

The quality improvement is often the largest of any technique: 25-40% retrieval accuracy improvement for user-facing Q&A applications. The vocabulary gap between natural user questions and formal document text is the single biggest retrieval obstacle, and hypothetical questions bridge it directly.

Technique 4: chunk expansion (surround with context)

The simplest technique. No LLM calls required. For each granular chunk, concatenate the previous and next chunks into an expanded version. Store the granular chunk’s embedding in the vector store (for precise search) and the expanded version in the document store (for rich context).

How It Works

Imagine your granular chunks are labeled 1 through 10. For chunk 5:

• Search embedding: Just chunk 5 (focused, precise)
• Returned context: Chunk 4 + Chunk 5 + Chunk 6 (3x the context)

The search remains precise because the embedding reflects only chunk 5’s content. But the LLM receives the surrounding context, making it possible to understand chunk 5 in its proper setting.

The Difference From Chunk Overlap

This is not the same as chunk_overlap from earlier chapters. Overlap creates shared content between adjacent chunks during splitting. Expansion creates a separate, larger document that combines three chunks, but only the middle chunk is used for search. The distinction matters:

Implementation

# Build expanded versions from granular chunks
for i, chunk in enumerate(granular_chunks):
    parts = []
    if i > 0:
        parts.append(granular_chunks[i-1].page_content)
    parts.append(chunk.page_content)
    if i < len(granular_chunks) - 1:
        parts.append(granular_chunks[i+1].page_content)
    
    expanded_text = "\n\n".join(parts)
    doc_id = str(uuid.uuid4())
    
    # Search by granular chunk embedding
    vectorstore.add_documents([Document(
        page_content=chunk.page_content,
        metadata={"doc_id": doc_id}
    )])
    
    # Return expanded context
    docstore.mset([(doc_id, Document(page_content=expanded_text))])

A Concrete Example

Granular chunk (searched): “The Cornwall Ranger ticket allows unlimited travel on most bus services for one day. It costs £14 for adults and £7 for under-16s.”

Expanded context (returned to LLM): “First buses operate most routes across Cornwall, connecting major towns and coastal villages. Services run from early morning to late evening, with reduced schedules on Sundays. The Cornwall Ranger ticket allows unlimited travel on most bus services for one day. It costs £14 for adults and £7 for under-16s. For longer stays, weekly bus passes offer better value. The ‘ride Cornwall’ app provides decision-time bus tracking and mobile ticketing.”

The search found the specific fact about the Cornwall Ranger. The LLM received the full bus transportation context. The expanded version makes it possible to answer both “How much is the Ranger?” and “What other bus options are available?” from the same retrieval.

Chunk expansion is the recommended starting point for any RAG improvement because it is free (no LLM calls), fast (string concatenation), and effective (typically 15-25% improvement in answer quality for context-dependent questions).

The unifying pattern: multivectorretriever

All four techniques share the same architectural pattern: the two-store architecture with a vector store for search and a document store for synthesis. The only thing that changes is what goes into each store:

LangChain’s MultiVectorRetriever implements this pattern generically. You configure what goes in the vector store, what goes in the document store, and the id_key that links them. This means you can combine techniques: store both child chunk embeddings and summary embeddings in the same vector store, linked to the same original documents.

# Combined approach: child chunks + summaries in the same retriever
combined_vectorstore = Chroma(
    embedding_function=embeddings, 
    collection_name="combined_index")
combined_docstore = InMemoryByteStore()

# Add child chunk embeddings
for child, doc_id in zip(child_chunks, doc_ids):
    combined_vectorstore.add_documents([Document(
        page_content=child.page_content,
        metadata={"doc_id": doc_id, "type": "child_chunk"}
    )])

# Add summary embeddings (same doc_ids, different content)
for summary, doc_id in zip(summaries, doc_ids):
    combined_vectorstore.add_documents([Document(
        page_content=summary,
        metadata={"doc_id": doc_id, "type": "summary"}
    )])

# Both types of embeddings point to the same original documents
combined_docstore.mset(list(zip(doc_ids, original_chunks)))

retriever = MultiVectorRetriever(
    vectorstore=combined_vectorstore,
    byte_store=combined_docstore,
    id_key="doc_id"
)

Now a query might match a child chunk embedding for a specific question or a summary embedding for a broad question. Both lead to the same original document. This belt-and-suspenders approach maximizes recall at the cost of ~2x vector storage.

Choosing your technique: the decision framework

The Decision Tree

Start with chunk expansion (free, immediate improvement) or ParentDocumentRetriever (most well-tested general-purpose technique). Add summary embeddings for dense technical content where the raw text does not explicitly convey the conceptual meaning. Add hypothetical questions for user-facing Q&A where the vocabulary gap between users and documents is significant.

Combining Techniques

The MultiVectorRetriever architecture supports multiple embedding types per chunk. A production system might use child chunk embeddings as the primary index and hypothetical question embeddings as a secondary index. During retrieval, both sets of embeddings are searched, and results are merged. This belt-and-suspenders approach maximizes recall at the cost of increased storage and ingestion time.

Ingestion Cost and Performance Comparison

For a corpus of 200 documents producing 1,000 chunks:

Search latency is identical for all techniques because the search operation is the same: vector similarity search followed by a document store lookup. You pay the cost once during ingestion, and every subsequent query benefits for free.

Common advanced indexing mistakes

Mistake 1: Mismatched Embedding Models Between Vector Store and Queries

The most dangerous mistake, inherited later in the edition. If you embedded child chunks with OpenAI during ingestion but query with Chroma’s default model, the search produces garbage. This is especially easy to trigger with ParentDocumentRetriever because the setup code is longer and the embedding model specification can be overlooked.

# BAD: No embedding specified when loading existing collection
retriever = ParentDocumentRetriever(
    vectorstore=Chroma(collection_name="children"),  # Uses default!
    docstore=docstore,
    child_splitter=child_splitter
)

# GOOD: Always specify the same embedding model
retriever = ParentDocumentRetriever(
    vectorstore=Chroma(
        collection_name="children",
        embedding_function=OpenAIEmbeddings()  # Same as ingestion
    ),
    docstore=docstore,
    child_splitter=child_splitter
)

Mistake 2: Parent Documents Too Large

If your parent chunks are 10,000 characters, they may exceed the context window when combined with the question and prompt template. Four parent documents at 10,000 characters each is 40,000 characters (~10,000 tokens), consuming most of the context window and leaving no room for conversation history.

Mistake 3: Too Many Child Chunks Per Parent

If your parent is 2,000 characters and your child is 50 characters, each parent produces 40 child chunks. With 1,000 parents, you have 40,000 child chunks in the vector store. The search is still fast (ANN handles millions of vectors), but the ingestion time and storage cost multiply by 40x. A child size of 200 characters (producing ~10 children per parent) is a better balance.

Mistake 4: Not Testing Before and After

The most common process mistake. Teams implement ParentDocumentRetriever, feel good about the architecture, and deploy without measuring the improvement. Sometimes the improvement is 30%. Sometimes it is 2%. Sometimes it is negative (the child chunks happen to split key sentences at bad boundaries).

Always measure: create a test set of 20-50 questions before implementing any technique. Run the test set on basic RAG. Run it again after implementing the technique. Compare retrieval precision and answer quality. If the technique does not improve your specific content and queries, do not deploy it just because it is “advanced.”

Mistake 5: Forgetting the Document Store Is In-Memory

InMemoryByteStore loses all data when Python exits. This is fine for development but catastrophic for an operating deployment. in an operating environment, use a persistent document store:

# Development: in-memory (data lost on restart)
docstore = InMemoryByteStore()

# Production: Redis (persistent, shared across instances)
from langchain_community.storage import RedisStore
docstore = RedisStore(redis_url="redis://localhost:6379")

# Alternative: local file system
from langchain.storage import LocalFileStore
docstore = LocalFileStore("./docstore")