Part I · Define the match before choosing the mechanism

Four kinds of match

A retriever receives a query, evaluates candidates and returns an ordering. Hidden inside that ordinary description is a strong assumption: the scoring function knows what “relevant” means. It does not. Relevance belongs to a task, user and decision. The score can only express a proxy learned or designed in advance.

Consider four match contracts. A lexical contract asks whether a rare string or phrase is present. A semantic contract asks whether two expressions have sufficiently similar intent. An entity contract asks whether two names refer to the same object, even through aliases. A relational contract asks whether specified objects stand in a typed relation, perhaps at a specified time. These contracts overlap, but none reduces cleanly to another.

A retriever can therefore be correct at one layer and wrong at the next. It may identify the right topic but the wrong instrument, the right entity but the wrong period, or the right clause but an inapplicable version. This is why a single binary relevance label is often too coarse for consequential search. A useful judgement record separates at least topical relevance, proposition support, source authority, temporal applicability and user entitlement. Candidate generation usually optimises the first one or two. The final evidence decision needs all five.

The distinction also changes query decomposition. “Show the current limit for facility AB-204 and the officer authorised to approve an excess” is not one similarity request. It contains an entity key, a time condition, a quantitative proposition and an authority relation. The system may search text for the limit, resolve the facility through a canonical identifier, traverse an approval relation and reject expired records. Treating the whole sentence as one vector silently asks a learned geometry to perform identity resolution, temporal reasoning and access control without an explicit contract.

A match contract names the feature that must survive, the acceptable invariances and the dangerous confusions. For an identifier, case or punctuation may be ignorable while character order is decisive. For a semantic policy question, grammatical form may be ignorable while negation and modality are decisive. For an entity, spelling variation may be ignorable while legal identity is decisive. For a relation, surface wording may be ignorable while edge direction and valid time are decisive. That specification is more durable than a preferred model family.

These experiments vary one feature at a time. They expose why architecture by slogan fails. A query may contain both an identifier and a paraphrase, and the corpus may contain near-duplicate records with conflicting dates. The system then needs more than one channel plus a rule for how the channels interact.

A third distinction is more severe. Suppose two policies have the same title and near-identical body, but one was superseded yesterday. Textual similarity remains high under every reasonable representation. Applicability changes because valid time changes. Similarity retrieves candidates; it does not confer authority, currency or permission. Metadata filters, temporal logic and entitlement checks are preconditions, not extra relevance features to sprinkle into a final score.

Part II · Follow the geometry

flowchart LR
  A["One character changes the answer"] --> B["Four kinds of match"]
  B --> C["The representation bargain"]
  C --> D["Hybrid is a complementarity claim"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The representation bargain

Every retrieval representation makes a bargain. It preserves some neighbourhoods cheaply by collapsing other distinctions. Sparse and dense systems differ in where they place that compression, yet the boundary has become porous.

What sparse search preserves

Classical sparse retrieval represents a document in a vocabulary-sized space. A coordinate corresponds to a term or feature, and most coordinates are zero. An inverted index stores the postings for active terms, so a query can reach documents containing a rare token without comparing against every document.

BM25 is the durable example. Its score combines term rarity, term-frequency saturation and document-length normalisation. One common form is:

Here q is the query, d a document, t a query term, f(t,d) its frequency in the document, |d| document length and avgdl average document length. The parameter k₁ controls how quickly repeated occurrences stop adding much evidence; b controls length normalisation. The inverse-document-frequency term gives uncommon terms more discriminative weight. Karen Spärck Jones supplied the foundational account of term specificity, while Robertson and Zaragoza place BM25 in the probabilistic relevance framework.[1][2]

The mechanism explains BM25’s strengths. Product codes, clause numbers, chemical names, error strings and uncommon people’s names can dominate a ranking because they are rare and exact. The score is also inspectable: an engineer can see which terms matched and how corpus statistics contributed. This is valuable during incident diagnosis.

It also explains the weakness. If query and evidence use different words, the relevant coordinate may never activate. Stemming, synonyms and query expansion reduce that mismatch, but they introduce assumptions about equivalence. A rule that expands “termination” to “closure” can improve recall and still be wrong in a domain where those terms carry distinct legal effects.

Modern sparse retrieval is not limited to literal BM25. SPLADE uses a language model to produce sparse vocabulary activations and expansions while retaining inverted-index efficiency and explicit term dimensions.[6] This matters because the usual binary is false: sparse describes the representation’s occupancy, not the absence of learned semantics.

Sparse retrieval begins before BM25 scores anything. The analyser decides how text becomes terms: case folding, punctuation, stemming, stop-word removal, synonym expansion and field boundaries all alter the coordinates available to the scorer. Splitting HML-5F-2019 into common fragments can destroy the very identifier signal the index was chosen to preserve. Conversely, indexing a title, body, identifier and jurisdiction as undifferentiated text lets frequent body terms overwhelm a decisive field. Exact subfields, phrase queries and field boosts are part of the representation, not cosmetic tuning.

Corpus statistics also bind BM25 to an index snapshot. A term’s inverse-document-frequency weight changes when the corpus changes, even if the query and candidate text do not. This is normally desirable because rarity is corpus-relative, but it means rankings can move after a large ingestion or archival exercise. Reproducible evaluation should record analyser configuration, field schema, corpus version and BM25 parameters alongside the code release. Lucene’s implementation makes these choices explicit through its similarity and indexing machinery.[16]

The strongest sparse baseline is not “run defaults over raw chunks”. It preserves identifiers as identifiers, phrases as phrases, fields as fields and language-specific morphology where justified. Only after that baseline is competent does a dense arm face a meaningful comparison. A weak sparse setup can make hybrid retrieval appear necessary by manufacturing lexical failures that an appropriate analyser or structured lookup would have prevented.

What embeddings preserve

A dense dual encoder maps a query and each passage to fixed-length vectors. Retrieval ranks passages by a similarity such as dot product or cosine:

The document vectors can be computed before the query arrives. At query time, approximate nearest-neighbour search finds vectors close to e_q. Training pulls relevant query-passage pairs together and pushes selected negatives apart. Dense Passage Retrieval showed that this simple dual-encoder design could outperform a strong BM25 baseline substantially on several open-domain question-answering datasets used in that study.[3]

The gain comes from a different invariance. The model can learn that “terminate a relationship” and “close an account” often serve similar information needs even when exact overlap is weak. It can also carry contextual information into a token’s representation, so “bank” near “river” differs from “bank” near “loan”.

Dense retrieval likewise begins before nearest-neighbour search. The training pairs define what should become close, while negative examples define which plausible neighbours must remain apart. If negatives are mostly random, the model may learn broad topical separation yet fail on near-identical policies whose decisive difference is may versus must not. Hard negatives can sharpen that boundary, but only for distinctions represented in the data and loss. The embedding geometry inherits the supervision’s notion of relevance.

The index then adds a second approximation. A passage is chunked, encoded and placed in a vector store; approximate nearest-neighbour search explores only part of that space under latency and memory limits. Failure may therefore come from the encoder, the chunk boundary, vector quantisation, an approximate-search setting or the chosen candidate depth. Calling all of these an “embedding miss” prevents useful diagnosis. The release manifest should keep model, pooling strategy, chunker, vector dimension, distance function, index parameters and corpus snapshot together.

Single-vector encoders make one further bargain: every token interaction must be compressed into one document vector before the query arrives. Late-interaction systems such as ColBERT retain token-level vectors and defer part of the matching operation, often preserving finer evidence at additional storage and compute cost.[10] The relevant continuum is not simply sparse to dense. It runs from exact field lookup through weighted lexical and learned sparse retrieval to single-vector, multi-vector and joint query-document scoring, with increasing interaction and usually increasing cost.

Compression creates its own blind spots. EntityQuestions found that dense retrievers generalised poorly to simple questions about tail entities compared with sparse retrieval, especially when entities or question patterns were uncommon in training.[5] A later analysis projected dense representations back into vocabulary space and connected failures on tail entities with “token amnesia”, where decisive entity tokens received little representational weight.[7]

BEIR made the broader lesson difficult to ignore. Across heterogeneous zero-shot retrieval tasks, BM25 remained a robust baseline, while late-interaction and reranking methods achieved strong average performance at higher computational cost.[4] MTEB likewise found no single embedding method dominating every evaluated task.[11] Model choice therefore depends on the corpus, query mix and label definition, not an embedding leaderboard alone.

The categories continue to blur. ColBERT retains token-level dense vectors and applies late interaction rather than compressing each text into one vector.[10] BGE-M3 exposes dense, sparse and multi-vector retrieval functions from one model family.[12] These systems reinforce the central point: assess what information survives the scoring path. The marketing label on the model is secondary.

Part III · Demand marginal evidence

Hybrid is a complementarity claim

Running BM25 and an embedding retriever in parallel creates two ranked lists. It does not yet create a better system. Improvement requires at least one arm to retrieve relevant candidates the other misses, without introducing so many hard negatives that fusion or reranking cannot recover the order.

Let R_s(q,k) be the relevant documents found by the sparse retriever for query q within depth k, and R_d(q,k) the dense equivalent. The oracle union is R_s ∪ R_d. Its gain over the better single arm gives a ceiling on what any fusion method can extract from those candidate sets. If the union adds nothing, fusion has no missing evidence to rescue.

This is stricter than asking whether two rankings differ. They may disagree only about irrelevant documents. Useful complementarity concerns relevant evidence and decision-changing hard negatives. Lee and colleagues formalised the need to optimise fuller complementarity rather than treating dense retrieval as a residual patch over sparse retrieval.[8]

A recent Dutch retrieval preprint makes the marginal-value test concrete. Once BM25 and a transformer embedding retriever were present, two additional static embedding retrievers received no positive weight in the study’s cross-validated fusion selections. The conclusion is bounded to its models and five datasets, but the method is widely useful: standalone quality does not establish incremental value inside an existing ensemble.[13]

The negative control matters because “hybrid by default” is otherwise unfalsifiable. Any weak result can be blamed on fusion weights, embedding choice, chunking or the reranker. A proper comparison fixes the corpus and query set, then measures each arm, the oracle union, simple fusion and any learned policy. If the second arm has negligible unique-hit rate on critical slices, remove it or reserve it for a route where it demonstrably helps.

A proper hybrid experiment therefore starts with candidate-set accounting, not end-to-end answer fluency. For every query and depth k, record whether each arm retrieved at least one supporting evidence unit, which relevant units were unique to each arm and which hard negatives each arm introduced. Then compute the oracle union. This is an intentionally optimistic ceiling: it assumes a perfect selector can choose every relevant item from the combined candidates. If the oracle union barely improves on the best arm, no fusion algorithm can create substantial recall from those lists.

Aggregate recall is not enough. Slice queries by the distinction that decides relevance: exact identifiers, rare entities, paraphrases, multilingual variants, numeric conditions, negation, tables, relations and time-sensitive rules. Report denominators and confidence intervals for each slice. A one-point average gain can conceal a severe regression on identifier queries, or a large gain can come entirely from an easy, high-volume slice that does not affect consequential decisions. The acceptance rule should attach weights to decision consequence, not merely query frequency.

The hard-negative ledger is equally important. For each relevant item, construct or collect a plausible rival: one-character identifier twins, same-name entities, semantically similar opposite rules, superseded versions, wrong jurisdictions and passages that mention every query term without supporting the requested proposition. These controls reveal whether a retriever preserves the decisive feature rather than merely the topic. They also make reranker evaluation meaningful because the candidate list contains the confusions the reranker is expected to resolve.

Finally, measure the system cost created by complementarity. A second candidate generator adds indexing, storage, synchronisation, latency, release coupling and incident paths. Report median and tail latency, index size, ingest time and the proportion of queries invoking each arm. Compare simple routing, equal RRF, weighted RRF and reranking against the best single arm under the same candidate budget. The winning design is the smallest one whose decision-weighted gain survives held-out queries, bootstrap uncertainty and realistic operating limits.

Fusion combines rankings, not evidence semantics

Raw BM25 scores and cosine similarities do not share a natural scale. Min-max normalisation can become unstable when score distributions shift across query types. Reciprocal rank fusion avoids direct score comparison. For document d, it sums reciprocal rank contributions:

The constant c reduces the gap between adjacent ranks. Cormack, Clarke and Büttcher introduced RRF as a simple rank-combination method, and current search platforms expose it because it works without calibrating heterogeneous scores.[9][17]

RRF is a strong baseline, not a law. Equal weighting can let a systematically wrong arm pull the best candidate down. Weighted RRF needs tuning and revalidation. A learned fusion model can exploit richer features but creates label, drift and interpretability obligations. A deterministic router can send obvious identifiers to exact search and broad paraphrases to both arms, yet routing itself needs a labelled test set.

A 2026 preprint on financial filings found that retrieval-unit fairness materially changed the comparison and that tested query-adaptive routers did not establish a reliable improvement over simple training-free fusion under its evaluation design.[14] The lesson is not to abandon routing. It is to keep a simple fusion baseline and require statistically credible marginal gain.

Reranking comes later. A cross-encoder or late-interaction scorer can read the query and candidate together, resolving subtler interactions than independent encoders. It can improve order among retrieved candidates. A reranker cannot recover evidence that candidate generation omitted, and it must never receive evidence the caller was not entitled to retrieve.

Part IV · Make the claim executable

stateDiagram-v2
  [*] --> Claim
  state "Lexical identity" as Claim
  state "Semantic equivalence" as Test
  state "Entity identity" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

A controlled BM25, dense and hybrid lab

The following laboratory uses six synthetic policy passages and four query shapes. The BM25 arm is implemented from the formula above. The dense arm is deliberately transparent: synonyms map to shared concept dimensions, while identifiers and negation receive no dimension. This is not a substitute for a trained embedding model. It is an intervention device that makes compression visible.

The four cases ask for an exact identifier, a lexical-mismatch paraphrase, a rare schedule and a negated rule. Equal-weight RRF fuses the two rankings. A small routed policy gives more weight to sparse evidence when an identifier or negation is present and more weight to dense evidence for the remaining paraphrase-shaped query.

These numbers are generated by the supplied synthetic programme. They establish no external performance claim. Their value is causal. The semantic encoder fixes the paraphrase and loses negation; BM25 preserves the identifier and negation; equal RRF can perform worse than the better arm because fusion adds ranks without understanding why they were produced. The routed result succeeds because the query classes were designed into the experiment.

That last sentence is also the warning. The router has seen the relevant distinctions by construction. On a real corpus, query categories overlap, tokenisers split identifiers unexpectedly, users misspell names and labels contain uncertainty. The policy must be evaluated on held-out queries and compared with equal fusion. The toy result supports a test design, not a universal routing rule.

#!/usr/bin/env python3
"""Controlled BM25, dense and hybrid retrieval laboratory.

The dense arm is deliberately a transparent semantic encoder rather than a
pretrained model. It maps synonyms to shared concept dimensions and ignores
identifiers and negation. That makes the representation bargain visible and
keeps the experiment runnable with the Python standard library.

Expected result: no single retriever wins every query. Equal-weight RRF can be
worse than its best input; a simple, explicit routing policy recovers the four
synthetic gold passages. This is a mechanism demonstration, not a benchmark.
"""

from __future__ import annotations

import math
import re
from collections import Counter, defaultdict
from dataclasses import dataclass
from typing import Iterable, Mapping, Sequence

TOKEN_RE = re.compile(r"[a-z0-9]+(?:-[a-z0-9]+)*")
IDENTIFIER_RE = re.compile(r"\b[A-Z]{2,5}-\d+[A-Z]?\b")
NEGATION = {"not", "never", "without", "exclude", "excluding"}

DOCUMENTS: dict[str, str] = {
    "D1": "Policy XR-17: dormant accounts may be closed after 90 days.",
    "D2": (
        "Procedure LM-204: an inactive consumer association can be "
        "terminated after three months."
    ),
    "D3": (
        "Policy XR-17A: vulnerable-customer accounts must not be closed "
        "without specialist review."
    ),
    "D4": (
        "Guide GN-8: ending an unused banking relationship requires an "
        "authorised request."
    ),
    "D5": (
        "Schedule QZ-48291: an early repayment charge is waived after "
        "compulsory acquisition."
    ),
    "D6": (
        "Schedule QZ-48219: an early repayment charge applies after "
        "voluntary sale."
    ),
}

CASES: dict[str, tuple[str, str]] = {
    "exact identifier": ("XR-17", "D1"),
    "semantic paraphrase": (
        "May a dormant retail relationship be ended after a quarter?",
        "D2",
    ),
    "rare entity": ("QZ-48291 compulsory acquisition charge", "D5"),
    "negated rule": ("Which rule says accounts must not be closed?", "D3"),
}


def tokenise(text: str) -> list[str]:
    """Lower-case tokenisation that preserves hyphenated identifiers."""

    return TOKEN_RE.findall(text.lower())


@dataclass(frozen=True)
class ScoredDocument:
    document_id: str
    score: float


class BM25:
    """Small BM25 implementation using the common Robertson-style IDF."""

    def __init__(
        self,
        documents: Mapping[str, str],
        *,
        k1: float = 1.2,
        b: float = 0.75,
    ) -> None:
        if k1 < 0:
            raise ValueError("k1 must be non-negative")
        if not 0 <= b <= 1:
            raise ValueError("b must be in [0, 1]")

        self.documents = dict(documents)
        self.k1 = k1
        self.b = b
        self.tokens = {key: tokenise(text) for key, text in documents.items()}
        self.document_count = len(self.tokens)
        self.average_length = (
            sum(len(tokens) for tokens in self.tokens.values())
            / self.document_count
        )
        self.document_frequency: Counter[str] = Counter()
        for tokens in self.tokens.values():
            self.document_frequency.update(set(tokens))

    def score(self, query: str, document_id: str) -> float:
        query_terms = tokenise(query)
        document_terms = self.tokens[document_id]
        term_frequency = Counter(document_terms)
        document_length = len(document_terms)
        total = 0.0

        for term in query_terms:
            frequency = term_frequency[term]
            if frequency == 0:
                continue
            containing_documents = self.document_frequency[term]
            idf = math.log(
                1
                + (
                    self.document_count
                    - containing_documents
                    + 0.5
                )
                / (containing_documents + 0.5)
            )
            numerator = frequency * (self.k1 + 1)
            denominator = frequency + self.k1 * (
                1
                - self.b
                + self.b * document_length / self.average_length
            )
            total += idf * numerator / denominator

        return total

    def rank(self, query: str, *, limit: int = 6) -> list[ScoredDocument]:
        scored = [
            ScoredDocument(document_id, self.score(query, document_id))
            for document_id in self.documents
        ]
        # A lexical index does not normally return zero-match documents.
        positive = [item for item in scored if item.score > 0]
        return sorted(
            positive,
            key=lambda item: (-item.score, item.document_id),
        )[:limit]


class TransparentDenseEncoder:
    """A visible semantic compression model for controlled experiments.

    Synonyms share a concept coordinate. Identifiers and negation have no
    coordinate, reproducing two common failure shapes without downloading a
    model or hiding behaviour inside learned weights.
    """

    dimensions = (
        "closure",
        "inactivity",
        "account",
        "duration",
        "fee",
        "involuntary event",
        "vulnerability",
        "review",
    )

    lexicon: dict[str, str] = {
        "close": "closure",
        "closed": "closure",
        "closing": "closure",
        "terminate": "closure",
        "terminated": "closure",
        "ending": "closure",
        "ended": "closure",
        "end": "closure",
        "inactive": "inactivity",
        "dormant": "inactivity",
        "unused": "inactivity",
        "account": "account",
        "accounts": "account",
        "association": "account",
        "relationship": "account",
        "profile": "account",
        "consumer": "account",
        "retail": "account",
        "banking": "account",
        "90": "duration",
        "days": "duration",
        "three": "duration",
        "months": "duration",
        "quarter": "duration",
        "charge": "fee",
        "repayment": "fee",
        "waived": "fee",
        "applies": "fee",
        "compulsory": "involuntary event",
        "acquisition": "involuntary event",
        "vulnerable-customer": "vulnerability",
        "vulnerable": "vulnerability",
        "specialist": "review",
        "review": "review",
        "authorised": "review",
        "request": "review",
    }

    def __init__(self, documents: Mapping[str, str]) -> None:
        self.documents = dict(documents)
        self.document_vectors = {
            key: self.encode(text) for key, text in documents.items()
        }

    def encode(self, text: str) -> tuple[float, ...]:
        vector = [0.0] * len(self.dimensions)
        position = {name: index for index, name in enumerate(self.dimensions)}
        for token in tokenise(text):
            dimension = self.lexicon.get(token)
            if dimension is not None:
                vector[position[dimension]] += 1.0
        length = math.sqrt(sum(value * value for value in vector))
        if length == 0:
            return tuple(vector)
        return tuple(value / length for value in vector)

    @staticmethod
    def cosine(left: Sequence[float], right: Sequence[float]) -> float:
        return sum(a * b for a, b in zip(left, right, strict=True))

    def rank(self, query: str, *, limit: int = 6) -> list[ScoredDocument]:
        query_vector = self.encode(query)
        scored = [
            ScoredDocument(
                document_id,
                self.cosine(query_vector, document_vector),
            )
            for document_id, document_vector in self.document_vectors.items()
        ]
        positive = [item for item in scored if item.score > 0]
        return sorted(
            positive,
            key=lambda item: (-item.score, item.document_id),
        )[:limit]


def reciprocal_rank_fusion(
    rankings: Iterable[tuple[float, Sequence[ScoredDocument]]],
    *,
    rank_constant: int = 60,
) -> list[ScoredDocument]:
    """Fuse ranked lists without pretending their raw scores are comparable."""

    if rank_constant < 1:
        raise ValueError("rank_constant must be positive")
    fused: defaultdict[str, float] = defaultdict(float)
    for weight, ranking in rankings:
        if weight < 0:
            raise ValueError("weights must be non-negative")
        for rank, item in enumerate(ranking, start=1):
            fused[item.document_id] += weight / (rank_constant + rank)
    return sorted(
        (ScoredDocument(document_id, score) for document_id, score in fused.items()),
        key=lambda item: (-item.score, item.document_id),
    )


def route_weights(query: str) -> tuple[float, float, str]:
    """Return sparse weight, dense weight and an inspectable reason."""

    terms = set(tokenise(query))
    if IDENTIFIER_RE.search(query):
        return 2.0, 0.5, "identifier present"
    if terms & NEGATION:
        return 2.0, 0.5, "negation must survive"
    return 0.5, 2.0, "paraphrase-oriented default"


def top_id(ranking: Sequence[ScoredDocument]) -> str:
    return ranking[0].document_id if ranking else "ABSTAIN"


def run() -> None:
    sparse = BM25(DOCUMENTS)
    dense = TransparentDenseEncoder(DOCUMENTS)
    rows: list[tuple[str, str, str, str, str, str, str]] = []

    for case_name, (query, gold) in CASES.items():
        sparse_ranking = sparse.rank(query)
        dense_ranking = dense.rank(query)
        equal_hybrid = reciprocal_rank_fusion(
            ((1.0, sparse_ranking), (1.0, dense_ranking))
        )
        sparse_weight, dense_weight, reason = route_weights(query)
        routed_hybrid = reciprocal_rank_fusion(
            (
                (sparse_weight, sparse_ranking),
                (dense_weight, dense_ranking),
            )
        )
        rows.append(
            (
                case_name,
                gold,
                top_id(sparse_ranking),
                top_id(dense_ranking),
                top_id(equal_hybrid),
                top_id(routed_hybrid),
                reason,
            )
        )

    headers = (
        "case",
        "gold",
        "BM25",
        "dense",
        "equal RRF",
        "routed",
        "route reason",
    )
    widths = [
        max(len(headers[index]), *(len(row[index]) for row in rows))
        for index in range(len(headers))
    ]

    def format_row(values: Sequence[str]) -> str:
        return " | ".join(
            value.ljust(widths[index]) for index, value in enumerate(values)
        ).rstrip()

    print(format_row(headers))
    print("-+-".join("-" * width for width in widths))
    for row in rows:
        print(format_row(row))

    methods = {"BM25": 2, "dense": 3, "equal RRF": 4, "routed": 5}
    print()
    for method, column in methods.items():
        correct = sum(row[column] == row[1] for row in rows)
        print(f"{method:10s} top-1 correct: {correct}/{len(rows)}")

    # Positive and negative checks keep the demonstration honest.
    assert rows[0][2] == "D1", "positive sparse case failed"
    assert rows[1][3] == "D2", "positive dense case failed"
    assert rows[1][4] != "D2", "negative fusion case unexpectedly disappeared"
    assert rows[3][3] != "D3", "negation failure was not exposed"
    assert all(row[5] == row[1] for row in rows), "routed policy missed a gold item"


if __name__ == "__main__":
    run()

case                | gold | BM25 | dense   | equal RRF | routed | route reason
--------------------+------+------+---------+-----------+--------+----------------------------
exact identifier    | D1   | D1   | ABSTAIN | D1        | D1     | identifier present
semantic paraphrase | D2   | D1   | D2      | D1        | D2     | paraphrase-oriented default
rare entity         | D5   | D5   | D5      | D5        | D5     | identifier present
negated rule        | D3   | D3   | D2      | D2        | D3     | negation must survive

BM25       top-1 correct: 3/4
dense      top-1 correct: 2/4
equal RRF  top-1 correct: 2/4
routed     top-1 correct: 4/4

Worked example: a decision-shaped policy query

A relationship manager asks: “For product HML-5F-2019, is the early repayment charge waived after compulsory acquisition, and who may approve the exception?” The corpus contains a current tariff, a retired tariff with the same product code, a general mortgage guide, a local approval procedure, a customer facility letter and an acquisition notice.

No single retriever should own this request. The product code calls for exact or entity lookup. “Compulsory acquisition” may appear under a formal synonym, so semantic expansion can help. The current-versus-retired distinction requires effective-date filtering before ranking. The approval question may require a procedure record linked to product and jurisdiction. The customer letter is governed by entitlement. The acquisition notice is case evidence, not a semantically similar policy.

Work the policy query as an evidence plan. First, parse HML-5F-2019 into an exact product key and preserve the original string for audit. Second, bind “after compulsory acquisition” to the charge-waiver proposition rather than treating those words as a loose topic. Third, interpret “who may approve” as an authority relation, with jurisdiction and decision date as required arguments. The output of this step is a typed request, not a rewritten search phrase.

Next, apply preconditions. Remove records the caller cannot access, records outside the product’s jurisdiction and documents whose valid-time interval excludes the decision date. Do not retrieve broadly and hide unauthorised passages later. Pre-filtering changes the candidate universe; post-filtering can leak existence, scores or snippets and can allow an unauthorised item to displace an authorised one before the cut-off.

Candidate generation can now be plural without becoming vague. Exact lookup and BM25 should recover the product tariff and schedule. Dense or learned sparse retrieval may bridge “compulsory acquisition” to a formal phrase such as “statutory purchase”. Entity or graph lookup can locate the approving role and its delegation record. Fusion may order textual candidates, but it should not combine a policy passage and an authority edge into one undifferentiated relevance score. The evidence packet keeps their types and provenance separate.

Use a serious negative control. Insert a retired tariff with the exact product code, identical waiver wording and a stronger lexical score than the current tariff. A system that returns it has excellent textual retrieval and an invalid decision result. Then insert a current general guide that is semantically close but does not authorise the waiver. Passing both controls requires temporal eligibility and proposition-level support beyond similarity. This is the boundary the production design must demonstrate before answer generation is evaluated.

The trace should retain query interpretation, entitlement scope, corpus and index versions, candidates from each arm, fused rank, reranker version, selected passages and rejection reasons. Without arm-level provenance, an incident team cannot distinguish a parse defect from a sparse miss, a dense miss, a fusion error or a context-selection error.

Evaluation must also use the right unit. If BM25 searches entire documents while a dense encoder sees truncated chunks, the comparison confounds representation with evidence exposure. The 2026 financial-filings preprint cited earlier calls this evidence-unit fairness. Chunk maps, table extraction and parent-child links are part of the retriever, not neutral plumbing.

A recent preprint benchmarking mixed text-and-table financial documents reported a strong BM25 first-stage baseline and gains from hybrid retrieval followed by neural reranking in its evaluated corpus.[15] Treat that as current, domain-specific evidence rather than a portable architecture law. The acceptance test still belongs to the target corpus.

The match contract

The architecture decision can be made explicit before model selection. Complete one contract for each consequential query slice, not one for the whole application.

Failure boundaries

Hybrid retrieval ceases to help when its arms are redundant, jointly blind or evaluated against the wrong target. Redundancy means the second arm adds no unique relevant evidence. Joint blindness means both representations discard the deciding feature, as with an unmodelled temporal relation or table structure. A wrong target means the benchmark rewards topical similarity while the real decision needs authority, proposition coverage or contradiction discovery.

It also fails operationally when index versions drift apart, access controls are applied only after candidate generation, or raw scores are exposed as calibrated confidence. Embedding upgrades require compatible re-encoding and cutover. Sparse analyser changes alter token identity. Fusion parameters bind those releases together. Rollback must restore the full manifest, not one index in isolation.

Finally, retrieval cannot answer every information need. Multi-hop relations, calculations, exhaustive aggregation and current operational state may require graph traversal, database queries, event reconstruction or tools. Sending those tasks through ever more elaborate similarity search hides a mechanism mismatch.

Evidence ledger

Sources and boundaries

The sources below support mechanisms and bounded empirical findings. The policy examples, figures and laboratory are synthetic. The match-contract worksheet and governed retrieval path are practitioner synthesis.

1. PrimaryKaren Spärck Jones, “A statistical interpretation of term specificity and its application in retrieval”, 1972.
2. PrimaryStephen Robertson and Hugo Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond”, 2009.
3. PrimaryVladimir Karpukhin et al., “Dense Passage Retrieval for Open-Domain Question Answering”, EMNLP 2020.
4. PrimaryNandan Thakur et al., “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models”, 2021.
5. PrimaryChristopher Sciavolino et al., “Simple Entity-Centric Questions Challenge Dense Retrievers”, EMNLP 2021.
6. PrimaryThibault Formal, Benjamin Piwowarski and Stéphane Clinchant, “SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking”, SIGIR 2021.
7. PrimaryOri Ram et al., “What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary”, 2023 version.
8. PrimaryDohyeon Lee et al., “On Complementarity Objectives for Hybrid Retrieval”, ACL 2023.
9. PrimaryGordon Cormack, Charles Clarke and Stefan Büttcher, “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods”, SIGIR 2009.
10. PrimaryOmar Khattab and Matei Zaharia, “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT”, SIGIR 2020.
11. PrimaryNiklas Muennighoff et al., “MTEB: Massive Text Embedding Benchmark”, EACL 2023.
12. PrimaryJianlv Chen et al., “M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings”, revised 2025.
13. PreprintAntónio Pereira Barata, “Do Static Embeddings Add Value to Hybrid Dutch Retrieval?”, 2026. Findings are limited to the evaluated Dutch tasks and retrievers.
14. PreprintChenyu Wu and You Lin, “Evidence-Unit Fairness and the Limits of Query-Adaptive Sparse-Dense Fusion in Financial Document Retrieval”, 2026.
15. PreprintMeftun Akarsu, Recep Kaan Karaman and Christopher Mierbach, “From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documents”, 2026.
16. OfficialApache Lucene, Similarity and BM25Similarity documentation.
17. OfficialElastic, Reciprocal rank fusion reference. Product behaviour should be rechecked at implementation time.