A relevant chunk can still be the wrong result
An operations analyst asks a banking assistant, “Can this customer break the fixed-rate loan without an early repayment charge after the property was compulsorily acquired?” The hybrid retriever returns ten passages. Eight discuss early repayment. One contains the exact product code. One explains compulsory acquisition. Recall@10 is reported as perfect because the labelled answer passage appears somewhere in the set.
The packet can still fail. The product-code passage may belong to a retired tariff. The compulsory-acquisition clause may apply to a different jurisdiction. Five results may repeat the same paragraph from copied manuals. The one current governing clause may be ranked ninth and removed by the context selector. A table footnote may carry the exception while the parser indexed only the table heading. A contradictory local addendum may never appear.
Recall@k asks whether a labelled relevant item appears among the first k results. It is useful. It is not an acceptance test for decision-grade retrieval. It ignores where the item ranks, whether relevance is graded, whether the set covers every required proposition, whether evidence is authoritative and applicable, whether duplicates displace useful material, and whether the downstream model actually receives the item.
Evaluate the evidence set the decision needs, not merely the chunk the benchmark happens to label. Hybrid retrieval should be decomposed and measured at each boundary: filtering, candidate generation, fusion, reranking, evidence assembly and answer support.
This article uses a fictional “Eastgate Bank” case. Product names, documents, queries, labels, timings and thresholds are composite design artefacts. Published research and official documentation support the methods; no production result is claimed.
The eastgate case has more than one relevance label
The fictional customer has mortgage product HML-5F-2019, booked in Eastgate’s United Kingdom entity. A public authority has compulsorily acquired the property. The analyst needs to know whether an early repayment charge applies and who can approve an exception. The corpus contains a current product tariff, a retired tariff with the same product code, a general mortgage guide, a local procedure, a staff training page, a complaint decision, an acquisition notice and the customer’s signed facility letter.
The decision requires at least five propositions: product and contract identity; current tariff treatment; whether compulsory acquisition qualifies; any contract-specific term, and approval authority. No single chunk proves all five. The best evidence set mixes exact lookup, lexical matching, semantic retrieval and a structured relationship between customer and facility.
| Evidence item | Topical relevance | Authority | Applicability | Role in packet |
|---|---|---|---|---|
Current HML-5F-2019 tariff |
High | Governing product artefact | Correct entity and effective date | Supports charge rule |
Retired HML-5F-2019 tariff |
High | Formerly governing | Wrong time | Hard negative; historical comparison only |
| Local procedure | Medium–high | Approved procedure | Correct workflow | Supports approval route |
| General mortgage guide | High | Explanatory | Broad | Background, not decisive |
| Staff training page | High | Secondary | May lag policy | Useful wording, insufficient alone |
| Complaint decision | Semantically similar | External case-specific authority | Different facts | Counterexample, not direct rule |
| Signed facility letter | Exact customer evidence | Contractual source | Correct customer and facility | Supports contract-specific term |
| Acquisition notice | Low lexical overlap with “fee” | Authoritative case evidence | Correct property and event | Supports triggering fact |
Binary relevance turns most of these into either 1 or 0. A better judgement records evidence function. The retired tariff is topically relevant and should be retrieved in a historical query, but it is harmful if admitted to the current decision without a warning. The acquisition notice may look semantically distant yet is essential.
Start with a query taxonomy
Aggregate metrics conceal predictable failure modes. Partition evaluation by the information need that determines which retrieval signal should work.
Exact-identifier queries contain product codes, account types, policy numbers, clause identifiers or named forms. Lexical retrieval should dominate. Paraphrase queries use business language that differs from document wording; dense retrieval may help. Scope queries require entity, jurisdiction, product and date filters. Multi-proposition queries need several evidence items. Table queries depend on layout-aware parsing. Exception queries need rare clauses and contradictions. Multilingual queries may cross language or script.
Benchmark suites such as BEIR expose heterogeneous domains and showed in their reported experiments that performance varies across retrieval architectures and datasets. MTEB broadened embedding evaluation across tasks. MIRACL provides human relevance judgements across 18 languages. These are valuable external tests. Eastgate still needs an internal taxonomy reflecting its corpus, entitlements, terminology and decisions.
Define the intended production mix. If exact product-code searches form a material share of high-consequence traffic, a benchmark dominated by broad semantic questions will select the wrong system. Report every metric by slice and consequence tier, then compute any weighted summary transparently.
Separate corpus, query and judgement versions
A retrieval score is meaningful only for a fixed corpus snapshot, index configuration, query set and judgement version. Policies change. Parsers improve. Labels are corrected. If those dependencies move together, a higher score does not identify what improved.
The evaluation manifest names source versions, chunker, parser, embedding model, lexical analyser, index build, filter policy, fusion parameters, reranker, context selector and evaluation code. It also names the question and judgement releases. Runs write ranked lists before scoring so that metrics can be recalculated independently.
| Artefact | Version evidence | Why it matters |
|---|---|---|
| Source corpus | Immutable source IDs and content hashes | Prevents silent policy or document drift |
| Parser output | Parser build, layout settings and extracted spans | Distinguishes retrieval from extraction defects |
| Chunk map | Source offsets, hierarchy and overlap | Lets judgements survive or migrate across chunking changes |
| Sparse index | Analyser, stop words, synonyms and index snapshot | Exact matching depends on tokenisation |
| Dense index | Embedding model, dimensions, normalisation and ANN settings | Candidate recall depends on representation and search |
| Filters | Policy and metadata schema version | Conditional recall can fall before ranking |
| Fusion and reranker | Code, model and parameters | Determines final order and interactions |
| Query set | Stable query IDs, text and slice labels | Avoids accidental test-set replacement |
| Judgements | Assessor, guideline, grade and revision | Exposes label uncertainty and correction |
| Metrics | Implementation and cut-offs | Prevents formula drift |
The TREC programme exists to create evaluation infrastructure and shared test collections for information retrieval. NIST’s overview of TREC 2023 explains this evaluation mission. The transferable practice is disciplined separation of test collection, runs and scoring: not the assumption that one TREC task matches bank evidence retrieval.
Recall@k still has a job
Recall@k is valuable for candidate generation. If the sparse and dense union never includes a necessary item, no reranker can recover it. Measure recall at the item, source and proposition levels.
Item recall asks whether each labelled evidence item entered the candidates. Source recall treats any valid chunk from the source as a hit, useful when chunk boundaries differ. Proposition recall asks whether at least one admissible item supporting each required proposition entered. These measures answer different engineering questions.
Always condition recall on admissibility. If the governing document is filtered out because its entity metadata is missing, recall@100 after filtering can reveal the loss. A global corpus recall that ignores permission and scope says little about what the production user can receive.
Measure exact nearest-neighbour recall separately from evidence recall. Approximate search can be compared with an exhaustive vector search for the same embeddings. Azure AI Search documents an exhaustive KNN query option alongside approximate vector search concepts. The comparison diagnoses ANN configuration; it does not say whether the embedding represents the banking query correctly.
Graded gain captures rank and usefulness
When evidence has degrees of usefulness, normalized discounted cumulative gain is more informative than binary recall. The original cumulated gain work introduced graded relevance and rank discounting. nDCG compares achieved discounted gain with the ideal order for the judged set.
Eastgate can define a grade rubric tied to evidence use. Grade 4: authoritative, applicable and directly supports a required proposition. Grade 3: authoritative and useful but incomplete. Grade 2: explanatory or secondary evidence. Grade 1: topically related context. Grade 0: irrelevant. A separate harmful flag marks superseded, forbidden or misleading items; do not hide them inside a low positive grade.
| Grade | Meaning for eastgate | Example | Assessor question |
|---|---|---|---|
| 4 | Decisive evidence for a required proposition | Current tariff clause for correct product and date | Could an authorised reviewer rely on this for the proposition? |
| 3 | Strong evidence requiring another item | Acquisition notice proving event but not fee treatment | What proposition does it establish? |
| 2 | Approved explanation or procedure detail | Local procedure describing route | Is it useful but non-governing? |
| 1 | Related background | General article on early repayment charges | Would omission change the decision packet? |
| 0 | No material relevance | Unrelated savings tariff | Does it help any required proposition? |
| Harmful | Topically plausible but inapplicable or forbidden | Retired tariff presented as current | Could admission mislead or violate policy? |
Report nDCG at context-relevant cut-offs, not only a conventional 10. If the assembler sends six items, nDCG@6 matters. Report harmful-at-k separately because an ordered gain score can mask one dangerous item among several good ones.
Grading should not collapse authority, applicability and topicality into assessor intuition. Capture the component labels and derive the use grade under a published rule. This enables a policy change without relabelling every topical relationship.
Proposition coverage is the centre of the evaluation
A packet needs all material propositions, not many chunks about one proposition. Define coverage as the share of required propositions with at least one admissible supporting item in the final packet. Weight only where the decision owner has approved proposition criticality.
For Eastgate, product identity, current charge rule, acquisition status, contract term and approval authority are all required. Retrieving four variants of the tariff clause covers one proposition. A simple diversity metric may reward the different wording, but proposition coverage remains one fifth until other needs are met.
Coverage needs negative and conditional propositions. A question may require checking whether an exclusion applies. The absence of retrieved exclusion language is not proof that no exclusion exists. The evidence contract should identify the authoritative section or completeness boundary.
Measure coverage_before_rerank, coverage_after_rerank and coverage_in_context. A drop between union and rerank points to ranking. A drop between rerank and context points to deduplication or token budgeting. This decomposition prevents teams from retraining embeddings for an assembler defect.
Evidence precision is not ordinary precision
Ordinary precision counts relevant retrieved items divided by retrieved items. Evidence precision asks how much of the final packet materially supports, contradicts or qualifies required propositions. Related background that consumes scarce context is a false positive for this purpose even if a search assessor calls it relevant.
Use the final packet as denominator. Report authoritative-support precision, secondary-evidence share, unsupported-context share and harmful share. Preserve a small amount of approved explanatory context if it improves comprehension, but make the budget explicit.
Token-weighted precision can expose one long irrelevant item that item precision hides. Compute the share of context tokens attached to an evidence role. Do not reward repeated copies. An overlap-aware calculation assigns gain once to substantially identical source spans.
A packet of ten relevant chunks can have low evidence precision if nine restate one proposition.
Redundancy and novelty need source-aware measures
Hybrid systems often return duplicates because the same manual appears in HTML, PDF, training pages and copied knowledge articles. Dense and lexical channels can also retrieve overlapping chunks from one source. Duplicates make recall and precision look better while displacing independent evidence.
Build a source-family graph during ingestion. Exact hashes catch identical text. Near-duplicate detection catches formatting and minor edits. Provenance links identify authorised copies. At evaluation time, compute unique-source coverage, unique-proposition gain and duplicate-token ratio.
Novelty is not the number of different sources. Two independent authoritative items supporting different propositions are valuable. Two blogs paraphrasing the same policy are not. Define novelty by new proposition support, contradiction or qualification after earlier ranked items.
The evidence assembler can use maximal marginal relevance or a custom submodular objective, but evaluate the result against proposition roles. Semantic dissimilarity alone can select diverse irrelevance.
Contradiction retrieval is a first-class objective
A system optimised only for supporting evidence may omit exceptions and local amendments. Create queries whose gold packet includes both a general rule and a contradiction, exception or narrower clause. Measure contradiction recall and contradiction rank.
In Eastgate’s case, the general tariff says early repayment charges apply during the fixed period. A compulsory-acquisition clause creates an exception, subject to documentation and approval. The retriever should return both. A model cannot faithfully describe the exception if the general rule consumes all top ranks.
Hard negatives should be plausible: retired versions, adjacent jurisdictions, similar product codes, general rules without exceptions, draft policies and copied guidance. Random irrelevant documents make evaluation easy and unrepresentative.
| Hard-negative family | Why it ranks | Required behaviour |
|---|---|---|
| Superseded near-copy | Shares most words and product code | Exclude for current frame; retain for historical tests |
| Adjacent jurisdiction | Same policy topic and terminology | Scope filter or applicability rerank rejects |
| Similar identifier | One-character or suffix difference | Exact identity prevents substitution |
| General rule | High semantic overlap | Rank with, but not above, decisive exception for exception query |
| Draft or consultation | New wording and recent date | Authority status blocks governing use |
| Training derivative | User-friendly paraphrase | Mark secondary and cite primary source |
| Table header without row | Repeats query concepts | Retrieve row and footnote evidence instead |
A benchmark without realistic near-miss documents measures search on easy negatives, not control-grade retrieval.
Diagnose the hybrid union before tuning fusion
Hybrid retrieval combines channels with different strengths. Measure each channel alone, the union, fusion and rerank. For every gold item, record which channels retrieved it and at what rank. This creates an evidence-attribution table.
Lexical retrieval should win exact identifiers and rare terms. Dense retrieval should add paraphrases. Learned sparse methods such as SPLADE v2 aim to combine sparse retrieval with learned term expansion. Late-interaction systems such as ColBERTv2 retain token-level interactions with compression. External papers report performance on their tasks; Eastgate must test each method under its corpus and latency constraints.
Report marginal contribution: proposition coverage added by dense beyond lexical, by lexical beyond dense, and by graph beyond both. If dense adds no unique evidence on exact-ID slices but dominates latency, reduce its use there. If lexical misses paraphrase exceptions, route those queries differently.
The union candidate budget matters. Giving each channel 100 candidates is not equivalent to a shared budget. Track candidates and compute by channel, and test whether one channel floods fusion with near-duplicates.
Fusion scores are not comparable by default
BM25 scores, cosine-derived scores and model logits have different ranges. Naively adding them makes a tuning artefact look like relevance. Reciprocal rank fusion avoids raw-score comparability by using ranks. The original RRF paper reported strong results on its evaluated collections. Azure AI Search documents its current hybrid RRF scoring for parallel text and vector queries.
RRF is a robust baseline, not a universal optimum. It cannot know that an exact product-code match should dominate a broad semantic match. Weighted fusion or learning-to-rank can use query class and evidence features, but increases evaluation and governance needs.
Run fusion ablations. Compare best single channel, union oracle, RRF, weighted RRF, score normalisation and reranking. The union oracle orders candidates by gold grades and shows the ceiling available to fusion. A large gap between oracle and fused score means the candidates exist but ordering is poor.
| Diagnostic | Question answered | Engineering response |
|---|---|---|
| Best-channel baseline | Is hybrid better than the strongest simple path? | Reject complexity without material slice gain |
| Union recall | Did any channel find required evidence? | Improve candidates if low |
| Union oracle nDCG | Could perfect ordering solve the run? | If high, focus fusion/rerank |
| RRF versus weighted fusion | Does query-aware weighting help? | Validate by slice and stability |
| Fusion unique gain | Which propositions appear only after combination? | Preserve productive channels |
| Duplicate amplification | Does fusion reward items present in mirrored indexes? | Collapse source families before gain |
Rerankers need stability and role evaluation
A cross-encoder reranker can improve topical order and still demote rare authoritative evidence. Evaluate nDCG, proposition coverage, harmful-at-k and role preservation before and after reranking. Record pairwise flips involving governing and secondary sources.
Create metamorphic tests. Add irrelevant tail documents; top evidence should remain stable. Duplicate a secondary article; it should not displace the governing clause. Paraphrase a query without changing intent; required evidence should remain. Change the product code; the old product’s items should disappear. Switch the temporal frame; current and retired versions should exchange eligibility in a controlled way.
Reranker scores are not calibrated probabilities of relevance. Thresholds should be tuned to an explicit abstention or context-budget objective. A fixed threshold can behave differently across query types and corpus releases.
Evaluate latency in batches and tails. A reranker that times out on the longest, most complex questions may systematically degrade the high-risk slice. The fallback ordering and its quality must be part of the test.
Context assembly is a retrieval stage
The ranked list is not the model context. The assembler may deduplicate, expand neighbouring chunks, include titles, trim tokens, reorder by source or group propositions. Each operation can change evidence coverage.
Measure source-boundary completeness: did the chosen chunk include the qualifier, table header, row and footnote needed to interpret it? Measure context proposition coverage and token-weighted evidence precision. Record items retrieved but excluded, with reason.
The order passed to the model can matter even when the set is identical. Test authoritative-first, proposition-grouped and original-rank orders. Do not optimise only for answer score; check citation mapping and contradiction preservation.
Retrieval evaluation and answer evaluation must stay separable
An answer can be correct from model memory despite poor retrieval. It can be wrong despite perfect evidence because generation failed. Score retrieval against judgements before looking at the answer. Then score claim support against the actual packet.
Frameworks such as RAGAS propose automated measures for retrieval-augmented generation, while RAGChecker separates retrieval and generation diagnostics in its published work. These methods can accelerate analysis. Model-based judges require validation against domain reviewers, version controls and bias checks.
For each answer claim, classify it as supported, contradicted, unsupported, or not requiring corpus evidence. Link support to exact evidence spans. Compute claim coverage and citation precision. A retrieval run with full proposition coverage but low claim support points to generation or citation. Low retrieval coverage with a correct answer may indicate memorisation or leakage.
Build judgements around evidence roles
High-quality evaluation data is an operating asset. A subject-matter expert should not receive 100 chunks and one question asking, “Relevant?” The guideline should define the decision, propositions, evidence roles, authority hierarchy, applicability facts, and harmful conditions. The tool should show source context and version metadata.
Use at least two stages. A query author drafts the information need and proposition set without seeing system results. Assessors then judge pooled candidates. An adjudicator resolves material disagreements and can correct the query if the gold set was incomplete. Keep every judgement revision with its reason.
Pooling borrows a durable IR practice: combine top results from diverse systems, remove duplicates and judge the pool. This reduces dependence on one baseline. It still misses relevant items no submitted system retrieved. Add source-directed search for required propositions and sample unjudged tail items.
Assessors need an unjudgeable state for missing context or expertise. Forcing a grade produces confident label noise. Record uncertainty and disagreement by slice. Low agreement on authority or applicability may reveal a policy problem, not assessor failure.
Measure agreement on components rather than only the final grade. Experts may agree that an item is topically relevant but disagree about whether it governs. That distinction helps policy owners target the ambiguity. Weighted agreement is appropriate for ordered grades; role labels may need class-specific agreement.
A gold set is a governed interpretation with provenance, not revealed truth. It should improve when new evidence or policy clarification appears, without rewriting prior benchmark results silently.
Keep query authors away from system vocabulary
Evaluation queries can be accidentally easy. If authors copy headings or keywords from the target document, lexical systems receive an advantage. If a model generates questions from a chunk, the question may echo the chunk’s wording and make retrieval circular.
Collect queries from work artefacts where permitted: analyst searches, case categories, service questions and policy-owner FAQs. Remove sensitive data through an approved process. Ask domain experts to express how a user would pose the problem before showing documents. Include abbreviations, misspellings and business language found in real workflows.
Synthetic queries remain useful for coverage. Generate paraphrases, scope changes and adversarial near misses, then have humans validate intent and difficulty. Keep synthetic and observed slices separate. A gain on generated questions may not transfer to live traffic.
Benchmark contamination also matters. A public corpus and public questions may appear in an embedding or reranker’s training data. External leaderboard strength should be treated as prior evidence, not proof of internal generalisation. A held-out, private evaluation set with controlled access is needed for final selection.
Avoid repeatedly tuning on the same test set. Maintain development, validation and locked acceptance sets. Rotate acceptance questions when policies change or leakage is suspected. Record every run submitted against the locked set.
Evaluate parsing before retrieval
Bank documents contain tables, footnotes, headers, scanned pages, forms and multi-column layouts. If the parser loses the decisive row, retrieval cannot find it. Create a parsing benchmark independent of search.
The Eastgate tariff contains an exception in a table row and a footnote defining compulsory acquisition. The parsing gold artefact should link the row cells, header hierarchy and footnote marker. Evaluate text fidelity, reading order, table-cell association, heading hierarchy, page/source offsets and OCR confidence.
| Parsing measure | Unit | Example failure | Retrieval consequence |
|---|---|---|---|
| Character or token fidelity | Gold span | OCR changes product code HML-5F |
Exact search misses or matches wrong code |
| Reading order | Ordered blocks | Two columns interleave | Sentence semantics become false |
| Table structure | Cell, row and header relation | Exception value loses header | Retrieved number lacks meaning |
| Footnote attachment | Marker to note | Qualifier becomes detached | General rule appears unconditional |
| Hierarchy | Heading ancestry | Local exception loses section context | Chunk looks globally applicable |
| Source offsets | Page and bounding region | Citation points to wrong page | Reviewer cannot verify evidence |
Parser metrics should be weighted by decision importance. A minor punctuation loss in narrative text is different from a changed decimal or identifier. Maintain a challenge set of documents that have previously failed.
When parsing changes, rebuild the retrieval index and score both layers. A retrieval gain may come from restored evidence rather than a better embedding. That is a valid improvement, but it needs the correct owner and regression test.
Evaluation, assurance and counterevidence
Filters need their own confusion matrix
Metadata and permission filters can improve precision while destroying recall. Evaluate them before candidate generation. For each query, define the admissible document set under the same identity, purpose, entity and time used in production. Compare filter output with expected admission.
False admission is a security or applicability failure. False denial is a utility failure and can systematically affect particular entities or older records. Report them separately by label, source and connector. A high overall accuracy is meaningless when sensitive false admissions are rare but severe.
Conditional retrieval metrics use the correctly admissible set as denominator. Also report pre-filter availability: whether the required evidence exists and has complete metadata. If the source is missing a legal-entity tag, the filter is behaving correctly by denying it while ingestion is defective.
Test filter interactions. A document can satisfy entity and product but fail effective time. Nested groups can change identity scope. A current document can be denied by an information barrier. Mutation tests should remove each predicate and prove the result set widens only as expected.
Approximate search has an independent recall budget
Dense candidate generation often uses approximate nearest-neighbour search. Two failures are possible: the embedding does not place the relevant item near the query, or the ANN index fails to return an item that is near under the embedding metric. Compare approximate results with exhaustive vector results on the same corpus and queries.
The HNSW paper describes a graph-based approximate nearest-neighbour method. The FAISS paper describes efficient similarity search and clustering at scale. Their reported results are not Eastgate performance. They establish algorithms whose operating points still require corpus-specific measurement.
Sweep search parameters and plot ANN recall against latency, memory and cost. Include filtered queries because filter selectivity can interact with ANN execution. Test updates and deletions; an index can perform differently after incremental churn than after a clean build.
Do not call ANN recall “retrieval recall” in dashboards. Name the denominator: exhaustive top-k neighbours under the chosen embedding. The exhaustive neighbours themselves may be poor evidence. Evidence recall compares with human evidence judgements.
Use a metric stack, not a composite mystery score
Leadership may ask for one score. A composite can help compare releases only if its components, weights and vetoes are visible. Never average away a security breach or complete loss on a critical proposition.
Create three layers. Control gates are pass/fail: no forbidden admissions in deterministic fixtures, no wrong governing version in critical cases, and full retention of hard-stop contradictions. Quality measures include coverage, nDCG, evidence precision and citation support. Operational measures include latency, availability and cost.
| Layer | Measures | Aggregation rule | Release effect |
|---|---|---|---|
| Control | False admission, applicability, critical proposition coverage, harmful-at-k | Veto or explicit exception | Failure blocks promotion |
| Retrieval quality | Proposition recall, nDCG, evidence precision, redundancy, contradiction recall | Slice-weighted with floors | Must improve or stay within approved non-regression |
| Generation linkage | Claim support, citation precision, abstention correctness | Reported after retrieval | Diagnoses downstream value |
| Operations | p50/p95/p99 latency, timeout quality, cost, index freshness | Efficiency frontier and service objectives | Must fit operating envelope |
| Equity and access | Language, document type, channel and entity slices | Minimum slice floors | Prevents average from hiding exclusion |
If a summary is unavoidable, publish a scorecard with red/amber/green components rather than a single decimal. Use a veto for forbidden admission and critical policy-version errors. Keep weights stable across compared runs.
The eastgate evaluation, stage by stage
Consider three candidate systems. System L uses BM25 and filters. System D uses a dense retriever. System H fuses both with RRF and uses a cross-encoder reranker. We need no invented score to compare the design; the evaluation plan specifies what evidence would decide.
First run the filter fixture for the UK entity, product and current effective date. The retired tariff and foreign policy should be excluded from the current decision pool but retained in a historical challenge. Second score each candidate generator. Lexical should retrieve the exact product and facility code. Dense should add the paraphrased acquisition clause. Third inspect union proposition recall.
Fourth run fusion and reranking. Record whether the current tariff, acquisition notice, facility letter and procedure survive and at which ranks. Fifth assemble a six-item context and compute proposition coverage, evidence precision, redundancy and harmful share. Sixth produce an answer and map each claim to evidence.
Suppose the hybrid union contains all evidence but the final packet lacks approval authority. The dense model does not need retraining. The diagnosis lies in reranking or context budgeting. Suppose the acquisition notice never enters any candidate list because the query and document share little language. Query decomposition or graph retrieval may be needed. Suppose the current tariff was filtered out because its effective date was parsed incorrectly. Fix ingestion.
This staged attribution is the main advantage over an end-to-end score: every loss has a boundary, an owner and a reproducible run.
Multi-hop retrieval needs path evaluation
Some evidence cannot be found with one query. The customer maps to a facility, the facility to a product version, the product to a tariff, and the event to an exception. An agent may perform several retrieval steps. Evaluate the path, not only the final documents.
Define a gold evidence graph with nodes and allowed relation types. Measure node recall, edge validity, path length, unnecessary branch count and recovery from a wrong intermediate hypothesis. A correct final document reached through an unauthorised or spurious relation is not a valid success.
Separate query-planning errors from retrieval errors. If the agent never asks for approval authority, the evidence contract or planner failed. If it asks correctly but receives the wrong procedure, retrieval failed. If it retrieves the procedure but omits it from the answer, generation failed.
Cap exploration. Measure tool calls, repeated queries and evidence gain per call. A plan that achieves full coverage in four bounded calls is preferable to one that searches twenty times and happens upon the answer.
Evaluate abstention and clarification
Retrieval sometimes cannot satisfy the evidence contract. The system should ask a bounded clarification or return unknown. A benchmark with only answerable questions rewards guessing.
Include unanswerable queries, ambiguous product names, missing documents, conflicting governing versions and insufficient permissions. Label the correct disposition: clarify entity, identify evidence gap, route to policy owner, or refuse unauthorised scope.
Measure false answer rate on unanswerable cases, unnecessary abstention on answerable cases and clarification efficiency. A clarification is useful if the answer resolves a material ambiguity without soliciting sensitive or irrelevant data.
Calibration curves can relate an exposed evidence-completeness score to actual proposition coverage. Do not use maximum vector similarity as confidence. Similarity can be high for a retired near-copy.
Cost and latency form an efficiency frontier
More candidates and larger rerankers can improve metrics while missing the operating window. Compare systems on an efficiency frontier rather than optimising quality without constraints.
Break latency into filter, lexical, dense, graph, fusion, rerank, source expansion and context assembly. Report p50, p95 and p99 by query slice and corpus size. Include cold starts, index updates and dependency timeouts. Compute quality under fallback because timeouts often route to a simpler path.
Cost should include embedding queries, index storage, reranker inference, document expansion, evaluation and human judgement. A system that adds a small average nDCG gain but no critical proposition coverage may not justify its complexity. A channel that uniquely retrieves rare exceptions can justify higher cost on routed queries.
Use conditional routing. Exact-code queries may take lexical plus structured lookup. Paraphrase exceptions may use hybrid and reranking. Multi-hop cases may use graph expansion. Evaluate the router’s classification and the end-to-end routed result.
The right operating point is the least expensive configuration that clears control gates and quality floors for each decision slice.
Report uncertainty, not just point estimates
Retrieval benchmarks are samples of queries and judgements. A difference in mean nDCG can be caused by a few queries, assessor choices or an easy slice. Report confidence intervals and paired differences at the query level. Use resampling methods appropriate to the metric and test design; do not assume independent documents when systems are compared on the same queries.
Statistical significance is not operational materiality. A tiny, stable gain across thousands of low-risk queries may be statistically clear and commercially irrelevant. A gain on ten high-consequence exception queries may be operationally important but uncertain. Report effect size, slice, consequence and uncertainty together.
Inspect the distribution of per-query changes. Count large wins, large losses and ties. Review the largest regressions. A mean improvement can conceal loss of exact identifiers while gaining on broad semantics. Set non-regression floors on critical slices before optimising the average.
Judgement uncertainty should flow into analysis. Score strict and permissive variants where assessors disagree. If system order changes under plausible labels, the selection is not robust. Acquire more expert judgement rather than declaring a winner from noisy decimals.
External benchmarks are transfer tests, not procurement scores
Public benchmarks help expose architecture strengths. BEIR tests heterogeneous zero-shot retrieval. MTEB spans embedding tasks. MIRACL tests multilingual retrieval. MS MARCO provides large-scale ranking data based on real anonymised queries; Microsoft’s dataset page also notes the relationship between its leaderboards and TREC Deep Learning evaluation.
Eastgate should run selected external tasks to catch broad regression and compare with published baselines. It should not rank vendors solely by a leaderboard. Public corpora differ from bank policies, scanned contracts, effective dates and exact codes. Training-data overlap may inflate scores. Licensing and reproducibility can limit direct use.
Finance-specific benchmarks are closer but still partial. FinanceBench evaluates open-book financial question answering over public company material with question, answer and evidence data. It is useful for evidence retrieval and numerical/document reasoning. It does not represent customer confidentiality, internal policy applicability or live bank workflows.
Use a benchmark portfolio:
| Portfolio layer | Purpose | Selection evidence | Claim permitted |
|---|---|---|---|
| Algorithmic sanity | Compare with established IR tasks | Public scripts and published protocols | Method behaves plausibly on known tasks |
| Domain transfer | Test finance, legal, tables or multilingual content | Relevant external benchmark | Generalises to some adjacent material |
| Internal corpus | Test Eastgate sources, terminology and filters | Versioned private benchmark | Works on sampled institutional information needs |
| Workflow simulation | Test evidence contracts and actions | Case-like scenarios and human review | Supports defined operating process under test |
| Shadow production | Test live distribution without user reliance | Approved logs and outcomes | Meets monitored pre-release criteria |
No layer substitutes for the next. A leaderboard is evidence about a task, not a certificate for a bank decision.
Implementation and operating detail
Online signals need causal restraint
Clicks, answer acceptance and task completion can complement offline evaluation. They are biased by rank, presentation, trust, time pressure and the absence of visible alternatives. A user may accept a wrong answer because it is fluent. A careful answer may take longer and be more reliable.
Instrument evidence interactions: source opened, citation inspected, correction made, answer abandoned, human escalation and downstream decision. Keep the event semantics stable and respect workforce privacy. Do not turn individual analyst behaviour into a covert performance score.
Randomised interleaving can compare result orders with less traffic than full A/B tests, but it may be inappropriate where a weaker result can affect a consequential decision. Use it on low-risk search tasks or in a sandbox. For high-risk workflows, shadow the candidate system and have reviewers compare evidence packets without letting the candidate drive action.
An online gain requires a mechanism. If task time falls, check whether proposition coverage and correction rates remain stable. If citation opens fall, determine whether evidence is clearer or users stopped verifying. Pair behavioural signals with audited outcome samples.
Monitor retrieval drift by boundary
Production retrieval changes even when models do not. The corpus grows, policies change, document formats shift, access metadata evolves, language mix changes and query patterns move. Build sentinel queries and live distributions for each boundary.
Corpus monitoring tracks source counts, parse failures, missing metadata, duplicate families and version closure. Query monitoring tracks identifiers, languages, length, ambiguity and decision types without retaining unnecessary sensitive text. Index monitoring tracks update lag, deleted-item persistence and ANN recall samples. Packet monitoring tracks proposition coverage on reviewed samples.
Trigger evaluation on change, not only calendar. A new parser, embedding model, synonym list, policy taxonomy, source connector or context budget can change retrieval. Run the affected slice plus the locked acceptance suite before promotion.
Drift alerts should identify hypotheses. A drop in exact-code recall after an analyser update belongs to search engineering. A rise in filter false denial after a taxonomy migration belongs to data governance and connector owners. A fall from reranked coverage to context coverage belongs to assembly.
Multilingual and transliteration slices matter in banking
Banking corpora and queries mix languages, scripts, transliterated names, local product terms and English policy. A global average can hide near-zero quality for one language. Define slices by query language, source language, cross-language need, script, OCR quality and code-switching.
Monolingual retrieval asks in one language over the same language. Cross-language retrieval asks in one language for evidence in another. Translation can occur before query, during indexing or after retrieval. Evaluate each path and retain exact identifiers unchanged.
MIRACL offers a strong research resource for multilingual ad hoc retrieval, but Eastgate needs financial terminology and local documents. Native-speaking domain reviewers should create and judge queries. Automatic translation can seed variants, not define the final gold set.
Measure proposition coverage and harmful-at-k per language. Test whether transliteration expands candidate sets with similar customer names and creates false matches. Test tokenisers on product codes and mixed scripts. Include right-to-left rendering and citation verification in user evaluation where relevant.
Multilingual support is not demonstrated by one multilingual embedding model. It is demonstrated by slice-specific evidence performance.
Table and numeric retrieval need structured judgements
A query may ask for a threshold, rate, fee or date. Retrieving a paragraph that mentions the concept is insufficient. The gold evidence should identify the table, row, column, unit, period and footnote. A chunk is useful only if those relationships survive.
For the Eastgate tariff, the fee can vary by product year and repayment circumstance. The relevant cell without its row label can be misleading. Context assembly may need a structured table representation and a page image link for verification.
Evaluate numeric exactness separately from retrieval. Check that the packet contains the correct value and unit, that the answer associates it with the right product and period, and that computations use cited inputs. A model may copy a real number from the wrong row; claim support must be relational, not substring-based.
Create hard negatives with adjacent rows, prior-year tables, percentage versus currency units, rounded summaries and footnote exceptions. Report row-selection accuracy and evidence completeness, not just answer exact match.
Security evaluation belongs inside retrieval evaluation
Corpus poisoning and indirect prompt injection can exploit highly ranked evidence. Add malicious and malformed documents to a controlled test partition. Some contain instructions to ignore the task, request secrets or cite the poisoned document. Others copy a policy and change one threshold. Some use hidden text or metadata to influence retrieval.
The filter should deny unauthorised sources. Provenance and authority features should demote unapproved copies. The context interface should mark retrieved content as data. The model and action layer should ignore embedded instructions. Each boundary receives its own verdict.
Do not call a poisoned document “relevant” merely because it repeats query terms. Give it a harmful role. Track harmful-at-candidate, harmful-at-rerank and harmful-in-context. A system that filters the answer after exposing sensitive text to a model has already failed the context boundary.
OWASP’s prompt-injection prevention guidance discusses indirect attacks through external content and layered mitigations. It does not prescribe bank retrieval metrics. The evaluation inference is clear: security has to be tested at ingestion, retrieval, context and action: not inferred from final prose.
Failure taxonomy turns metrics into work
Every failed query should receive one primary failure boundary and optional contributing factors. Without a taxonomy, teams create a new prompt or embedding model for every defect.
| Failure boundary | Diagnostic evidence | Typical owner | Wrong reflex |
|---|---|---|---|
| Source absence | Required artefact not present in governed corpus | Knowledge/data owner | Increase k |
| Parsing | Gold span missing or structurally wrong | Document platform | Retrain reranker |
| Metadata/filter | Gold source wrongly denied or wrong scope admitted | Connector/policy owner | Remove filters |
| Candidate generation | Gold evidence admissible but absent from union | Search engineering | Tune answer prompt |
| Fusion/rerank | Evidence in union but ordered below budget | Ranking engineering | Re-embed entire corpus immediately |
| Assembly | Evidence ranked but excluded, truncated or detached | Agent platform | Increase model context blindly |
| Generation | Packet complete but claim unsupported or mis-cited | Model/application team | Add more documents |
| Decision schema | Required proposition never requested | Process/product owner | Blame search recall |
| Label defect | Gold evidence wrong, incomplete or ambiguous | Evaluation governance | Optimise to noisy label |
Review failures weekly by impact and frequency. Fix representative root causes and add regression cases. A benchmark becomes valuable when it shortens diagnosis, not when it produces a dashboard with more decimals.
Release gates should be slice-specific
A single improvement threshold is unsafe. Define control gates and quality floors by decision and query class. The numbers below are examples of gate forms, not recommended universal values or measured Eastgate results.
For a critical policy question, require no false admissions in the deterministic fixture, complete coverage of mandatory propositions in the locked set, no current/retired version swaps, and zero harmful items in the final packet. For a low-risk knowledge search, allow lower proposition coverage and focus on nDCG and user usefulness.
Non-regression can permit trade-offs only with named approval. A dense model may improve paraphrases while slightly reducing exact-code rank. If exact codes are critical, route those queries lexically or reject the change. Do not average the loss away.
The release manifest states corpus, policy, models, parameters, query set and judgements. Promotion is reproducible. Rollback restores the whole compatible set rather than one model against a changed index.
Govern evaluation like a production system
The evaluation programme needs owners. Decision owners define propositions and consequence. Source owners confirm authority. Search engineering owns runs and diagnostics. Independent assessors govern labels. Security owns admission and poisoning fixtures. Model governance challenges automated judges. Operations owns service metrics.
Protect the locked set from repeated informal runs. Log access and submissions. Keep personal and customer data out unless a defined purpose and controls justify it. Prefer composite cases and approved de-identification for broad development. A small restricted set can validate residual realism.
Automated LLM judges can expand coverage, compare explanations and triage disagreements. Validate them against domain human judgements per task and language. Freeze judge model and prompt versions for comparisons. Audit position, verbosity and self-preference biases. Never let an unvalidated judge overrule a regulatory or contract authority label.
Evaluation code should have unit tests against hand-calculated examples. nDCG, recall and deduplication can differ subtly across libraries. Publish metric definitions, cut-offs, gain mappings and treatment of unjudged items. Recalculate old runs when a metric implementation is corrected, retaining the earlier report.
Counterevidence and limits
Role-rich judgements are expensive. They can slow iteration and still encode institutional blind spots. A compact binary set may be enough for a public FAQ retriever. Full proposition graphs are justified where missing or wrong evidence changes a material decision.
Offline metrics cannot reproduce every live query or document. Assessors can disagree. Pooling leaves unjudged relevant items. Synthetic hard negatives can overfit engineering effort. Online behaviour is biased. Automated judges can drift. Statistical tests do not repair sampling error.
Hybrid complexity can also be unnecessary. BM25 remains a strong baseline in many exact and domain-specific tasks. BEIR’s reported analysis found robust BM25 performance alongside trade-offs among dense, sparse, late-interaction and reranking methods. If lexical plus structured lookup clears Eastgate’s gates, adding dense retrieval needs unique evidence gain, not fashion.
Proposition coverage depends on a good decision schema. If the schema omits a decisive exception, the system can achieve 100 percent coverage and still fail. Authority labels can be wrong. Temporal metadata can be missing. Evaluation therefore challenges the evidence model as well as the retriever.
No metric stack proves regulatory compliance, truth or safe action. It provides bounded evidence about retrieval behaviour under defined tests.
Testable claims
This design makes falsifiable claims. Proposition coverage should diagnose incomplete packets that recall@k labels successful. Harmful-at-k should expose retired or wrong-scope near copies hidden by positive relevance averages. Stage metrics should localise whether evidence was lost at filtering, candidates, reranking or assembly. Source-family deduplication should reduce redundant context without reducing unique proposition support. Channel attribution should show where hybrid retrieval adds evidence over the strongest single path. Action and answer metrics should separate retrieval from generation failures.
Falsification is straightforward. If proposition coverage does not correlate with expert packet sufficiency, revise the schema or metric. If harmful labels do not predict reviewer error, examine interface and judgement assumptions. If stage diagnosis repeatedly assigns the wrong owner, improve instrumentation. If dense retrieval adds no unique evidence on held-out slices, remove or route it. If automated claim support disagrees materially with humans, it is not an acceptance measure.
Report these outcomes without promotional inference. A better offline packet metric is not a claim of reduced loss or faster operations. Those need separate study.
A practical ninety-day evaluation build
The first month defines three decision types, the query taxonomy, source snapshots and evidence-role guidelines. Build a small adjudicated set with hard negatives. Establish BM25, dense and hybrid baselines. Instrument ranked lists and manifests before tuning.
The second month adds parser challenges, filter fixtures, proposition coverage, nDCG, redundancy, contradiction and harmful metrics. Run ablations. Fix source and metadata defects. Freeze a locked acceptance set after independent review.
The third month adds context and claim-support evaluation, shadow packets, latency/cost frontiers and release gates. Promote one bounded use only if control gates pass. Continue human review and collect new failure cases without immediately adding them to the locked set.
| Period | Deliverable | Acceptance evidence |
|---|---|---|
| Days 1–15 | Decision schemas and query taxonomy | Named owners and proposition coverage review |
| Days 16–30 | Versioned corpus, judgements and baseline runs | Reproducible manifests and assessor agreement report |
| Days 31–45 | Stage and filter diagnostics | Known false-admission/denial fixtures pass |
| Days 46–60 | Hybrid ablations and hard negatives | Unique channel gain and regression analysis |
| Days 61–75 | Context and answer linkage | Claim support and citation mapping on reviewed sample |
| Days 76–90 | Shadow release and operating gates | Slice floors, latency/cost envelope and rollback rehearsal |
The timeline is an illustrative sequence, not a delivery promise. Corpus access, expert availability and source quality can dominate duration.
Reference architecture for retrieval evaluation
The evaluation platform sits beside production, not inside a spreadsheet. It snapshots corpus and configurations, executes every stage, stores ranked runs, applies versioned judgements, and publishes slice scorecards with drill-down to evidence.
For Eastgate, the platform does not ask whether hybrid search has a high recall score. It asks whether the current tariff, facility letter, acquisition evidence and approval procedure survive the entire path; whether the retired near-copy is excluded; whether each claim cites the right evidence, and whether the path meets the operating envelope.
Recall@k measures whether a labelled item entered a ranked prefix. Decision-grade evaluation measures whether the system assembled the authorised, applicable, non-redundant and sufficient evidence set: and whether it knows when it did not.
Route queries only when the router is measured
Hybrid does not need to mean every query uses every channel. A router can send identifier searches to lexical and structured retrieval, conceptual questions to hybrid, and relationship questions to graph expansion. Routing saves cost and can reduce noisy candidates. It also creates another failure boundary.
Build a router gold set from the query taxonomy. Measure class accuracy, but also score the evidence packet after the chosen route. A classification error matters only through its retrieval consequence. Some queries can safely take several routes when confidence is low; others need a deterministic identifier path.
Track route confusion by proposition loss. If exact identifiers are frequently sent to dense-only search, product identity will suffer. If paraphrase exceptions remain lexical-only, contradiction recall may fall. Set fallbacks from evidence coverage, not router confidence alone.
Router changes require the full locked suite because they alter traffic mix seen by downstream systems. Latency reports should include classification and fallback. A cheaper route is not a saving if it increases manual rework.
Measure query rewriting as an intervention
Rewriting can expand abbreviations, add product terms, decompose propositions or translate. It can also change intent. Preserve the original query and score the rewrite against a typed intent representation.
Evaluate three things: semantic fidelity, evidence gain and scope preservation. Semantic fidelity asks whether required propositions and constraints remain. Evidence gain measures unique admissible support added. Scope preservation ensures the rewrite did not change customer, entity, jurisdiction, time or action.
Run ablations with and without rewriting by query slice. Inspect rewrites that cause large rank changes. A rewrite that improves general nDCG but removes a negation or exception is harmful. Generated product codes should be prohibited unless resolved from an authoritative mapping.
Multi-query expansion needs a budget and deduplication. Record which rewrite retrieved each item. If one rewrite adds no unique proposition support, remove it or route it only where evidence shows benefit.
Evaluate deletion and update behaviour
Retrieval quality is also whether removed or corrected content stops appearing. Add lifecycle tests: ingest a document, update it, supersede it, revoke permission and delete it. Query after each event and at stated consistency cut-offs.
Measure update propagation, stale candidate rate, deleted-item persistence and source-family replacement. A new policy should enter the current index only when its effective scope activates. The old policy should remain available only to historical frames. A deleted customer artefact should disappear from candidates, caches, context expansions and evaluation traces under the applicable deletion policy.
Incremental updates can fragment ANN indexes or leave duplicate vectors. Compare quality before and after a representative churn sequence, not only after a clean rebuild. Reconciliation should compare index IDs and versions with the source registry.
A retriever that scores well immediately after a clean build but serves stale content under normal change is not production-ready.
The reviewer study needs specific questions
Ask reviewers whether the packet contains the evidence needed to decide, whether any item is misleading, which proposition remains unsupported, and whether the evidence roles are clear. Avoid “Was this helpful?” as the only outcome.
Measure decision agreement, time to identify governing evidence, correction rate, missed contradiction and confidence calibration. Keep the human’s authority unchanged. Reviewers should be able to open the original source and reject system labels.
Use a crossover design where feasible: the same reviewer sees different systems on matched cases in random order, with washout and enough separation to reduce recall. Where repeated exposure is unsafe, compare independent groups and document the limitation. Do not infer causal improvement from a before/after study that also changed policy and training.
Qualitative comments should map to the failure taxonomy. “Too many similar results” becomes redundancy. “I cannot tell which policy is current” becomes applicability presentation. “The right number has the wrong row” becomes parsing and relational support.
Prevent benchmark work from becoming theatre
An evaluation programme can produce impressive charts without changing release decisions. Tie every metric to a decision, owner and response. A failed filter fixture blocks. A low parser score opens a source-class restriction. A retrieval regression limits routing. An uncertain gain triggers more judgement rather than a launch announcement.
Publish negative results internally. If hybrid adds no value over BM25 on a slice, keep the simpler route. If a larger reranker improves nDCG but loses rare exceptions, reject or constrain it. If label agreement is weak, improve the policy and rubric.
Keep a one-page release card: intended use, excluded use, corpus, query distribution, gates, slice floors, known failures, latency/cost and rollback. Link the detailed run artefacts. Executives need the decision and limits; engineers need the evidence drill-down.
Minimum viable evaluation without compromise
For a small first release, choose fewer queries rather than weaker semantics. Fifty carefully adjudicated decision queries with proposition roles, hard negatives and stage instrumentation can be more useful than thousands of synthetic binary pairs. Include the critical slices and state uncertainty.
The minimum stack is a lexical baseline, dense baseline, hybrid run, filter fixture, proposition coverage, nDCG, harmful-at-k, redundancy, final-context coverage, claim support, latency and failure review. Add parser and temporal challenges for the source types actually in scope. Lock a portion for acceptance.
The programme grows through observed failures and source expansion. It does not need every public benchmark on day one. It does need reproducibility, role-aware labels and the ability to say why a release passed.
Final decision record for eastgate
Eastgate should choose a retrieval configuration only after recording which channels add unique evidence, which query slices clear proposition floors, whether filters and versions are sound, how many harmful items reach context, and what the latency/cost envelope is. The decision record should also name excluded languages, document types and decision classes.
If the hybrid system wins broad semantic questions but loses exact product identity, deploy query routing or do not deploy it for that slice. If a cross-encoder improves ranking but times out on complex cases, retain a tested fallback and disclose its quality. If current/retired policy swaps occur, block production regardless of average recall.
The result is a bounded engineering judgement, not a claim that hybrid retrieval is universally superior. The strongest system is the one that assembles the right evidence for the specified decision distribution with visible failure, acceptable resource use and reproducible controls.
Preserve evaluation evidence for challenge
Every promoted run should retain its ranked lists, filter decisions, source and chunk mappings, metric outputs, configuration manifest and judgement release. Store enough information to recalculate metrics without calling the production retriever again. Keep sensitive content in governed sources and use stable references in the run package.
When a label changes, score the old run against both judgement releases and explain the difference. When a source is corrected, preserve the old corpus snapshot for the historical release decision while blocking it from current production use. When evaluation code changes, retain checksums and hand-calculated unit cases.
An independent reviewer should be able to select a failed or passed query and follow it from admissible corpus through channel candidates, fusion, reranking, context and answer claims. If the evidence disappears into a dashboard aggregate, the programme cannot support challenge.
This retention is not indefinite by default. Evaluation artefacts inherit source access, privacy and records policy. A de-identified diagnostic can outlive restricted excerpts where approved. Deletion events propagate to run stores while preserving a non-sensitive record of the metric limitation.
The discipline closes the loop: a retrieval release is not supported by the score shown on launch day, but by the evidence that lets another expert reproduce and contest that score later.
Finally, keep the baseline runnable. A hybrid stack can accumulate enough dependencies that nobody can tell whether it still beats the lexical system it replaced. Re-run the simple baseline on every corpus and judgement release. Compare not only averages but critical slices, harmful results, cost and operational failure. Simplicity is a continuing control option, not a historical footnote.
Keep that comparison visible.