Part IThe error before the score

Two-stage retrieval exists because the operations needed for broad search and fine discrimination have different economics. A first-stage retriever must compare a query with a large collection. It therefore uses an indexable signal such as lexical matching, sparse learned weights, dense vectors or late interaction. A second-stage model can spend more computation on tens or hundreds of candidates because the search space has already collapsed.

This division is useful, but it creates a hidden contract. The retriever promises to return a pool that contains the evidence worth comparing. The reranker promises to order that pool according to a richer relevance judgement. When evaluation reports only the final top ten, a broken first promise can be misdiagnosed as a weak second promise.

A thought experiment: the sealed tray

Imagine a librarian who must answer a question from a million books. An assistant places twenty pages on a sealed tray. A brilliant scholar may read all twenty jointly, compare clauses and detect subtle entailment. The scholar still cannot cite page twenty-one. Replacing the scholar with a smarter one changes ordering accuracy inside the tray. It does not change the tray’s contents.

Now vary one causal feature. In world A, the controlling page is on the tray but sits underneath nineteen distractors. Better comparison should help. In world B, the controlling page remains in the library. The same comparison mechanism cannot help, regardless of its parameter count. This is the simplest negative test for a reranking proposal: hold the reranker perfect and ask whether the candidate pool still permits the right answer.

The ceiling in one inequality

Let U be the corpus, R(q) the relevant items for query q, and C_K(q) the top K candidates produced by the first stage. A pure reranker returns a permutation of C_K. For any final cutoff m no larger than K, its relevant output must come from the intersection R(q) ∩ C_K(q).

Reranking can improve recall at the smaller final cutoff by moving relevant items upward. It cannot improve recall of the complete candidate pool without another retrieval step. Work on cross-encoder retrieval states this candidate-recall upper bound directly, while ReFIT adds a second retrieval operation precisely because ordinary reranking cannot alter Recall@K.[5][8]

Two losses, two owners

Candidate omission and ordering error can coexist. A query may retrieve one of three relevant passages and bury that one at rank forty. Improving the reranker helps the visible passage but leaves two missing. The useful decomposition is therefore not “retriever versus reranker” as competing products. It is the amount of final error attributable to each stage.

Define the candidate oracle as the best possible ordering of the existing pool under trusted relevance labels. Its score is a counterfactual ceiling, not a deployable model. The gap between the corpus ideal and the candidate oracle is omission loss. The gap between the candidate oracle and the current first-stage order is ordering headroom. A reranker earns its place only by capturing enough of that headroom to justify latency, cost and new failure modes.

The unit of omission

Calling a document “retrieved” can conceal the evidence needed from it. Suppose the controlling rule spans a heading, a paragraph and a footnote. Document-level recall is one if any fragment from that document enters the pool. Passage-level recall may also be one if a topically related paragraph is present. Yet the final context still lacks the condition in the footnote. Candidate sufficiency must be measured at the unit the decision consumes, not at whichever unit makes retrieval look successful.

There are at least four useful units. A document is enough when the downstream reader can inspect the whole source. A passage is enough when one bounded span contains the material proposition. A proposition is enough when the claim and its qualifiers can be reconstructed from one or more spans. An evidence bundle is needed when the answer depends on several items, such as a general rule, an exception and a case-specific fact. These units produce different oracle ceilings from the same ranked list.

This distinction changes chunking evaluation. A chunk can be semantically close yet structurally incomplete because its heading, table labels, effective date or cross-reference was detached. A reranker may correctly prefer that chunk over weaker candidates and still support the wrong answer. The missing object is not always a document. It may be a relationship between fragments that the retrieval representation erased.

For multi-evidence questions, ordinary Recall@K can also be too forgiving. Finding one of three required items is not a sufficient candidate set. Record both item coverage and bundle completion. Item coverage measures how much relevant material appeared. Bundle completion asks whether at least one admissible combination can support the decision. The latter is stricter, but it matches the operational question the reranker is meant to serve.

flowchart TD
  A["The authoritative paragraph<br/>that no model could rank"] --> B["Part I The error before the score"]
  B --> C["Part II What interaction buys"]
  C --> D["Part III The viable operating<br/>region"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part IIWhat interaction buys

The attraction of reranking is real. Fast retrievers usually compress a query and a document into representations that can be indexed or matched cheaply. That compression limits how much query-specific interaction can occur. A cross-encoder instead presents the query and candidate together, allowing attention to compare terms, qualifiers and relations before producing a relevance score.

Nogueira and Cho’s BERT reranker demonstrated the power of this early interaction on passage ranking.[1] monoT5 reformulated relevance scoring as generation of target tokens and reported strong transfer across several collections.[2] Later work explored pairwise and listwise objectives, including RankT5 and language-model permutation ranking.[6][7] These mechanisms can make distinctions that one vector dot product misses.

Pointwise, pairwise and listwise are different questions

A pointwise reranker asks, “How relevant is this candidate to the query?” Each candidate receives a score independently. A pairwise model asks, “Which of these two candidates should rank higher?” A listwise model considers several candidates together and can reason about relative order, redundancy or coverage. The distinction matters because the deployment objective is a list, while much training data labels individual pairs.

Listwise context is not automatically superior. It introduces order sensitivity, context limits and a harder inference path. RankGPT used instructional permutation generation and a sliding-window strategy to rank longer lists, then explored distillation into smaller models.[7] RankT5 reported that ranking losses could improve text ranking and that listwise training appeared to transfer better than classification losses in its evaluated settings.[6] These are useful mechanisms, not permission to skip candidate diagnostics.

A minimal worked example

Suppose the corpus contains two relevant passages, graded 3 and 2. The first stage retrieves both, but places them at ranks five and three. At a final cutoff of three, only the grade-2 passage contributes. A competent reranker moves the grade-3 passage to rank one and the grade-2 passage to rank two. Here candidate sufficiency is complete, the candidate oracle is perfect, and the ordering headroom is large. This is the clean case for reranking.

Change one fact: remove the grade-3 policy passage from the candidate set. The candidate oracle now falls below the corpus ideal. A better reranker may still surface the grade-2 guide, but it cannot produce a complete evidence set. This counterfactual matters more than comparing two model leaderboards because it tells the team whether any pure reranker can solve the observed failure.

Interaction is not authority

A relevance model estimates fit to an information need. It does not establish that a document is current, authoritative, permitted or true. In regulated retrieval, the scoring function may need typed features for source authority, effective time, jurisdiction, entitlement and claim coverage. Some of these should be hard filters rather than learned preferences. A fluent reranker can otherwise place a superseded FAQ above an operative policy because semantic resemblance is not legal applicability.

What the scoring target makes visible

A reranker does not discover relevance in the abstract. It learns a preference induced by labels, negatives, query form and loss. If training marks every passage from an answer-bearing document as positive, the model can learn document topic rather than evidential support. If negatives are sampled only from obviously unrelated documents, a model may look excellent until it meets near-duplicate policies or superseded versions. The richer interaction mechanism raises capacity; it does not guarantee that the capacity is pointed at the right distinction.

Pointwise scores are especially easy to misread as probabilities. Their numeric scale may drift across queries, domains or model versions even when within-query order remains useful. A threshold such as “accept above 0.8” therefore needs separate calibration evidence. Ranking asks which candidate should come first. Abstention asks whether any candidate is good enough. Those are related but different measurement problems.

Pairwise and listwise models introduce their own target choices. A pairwise model can learn local preferences that form cycles: A over B, B over C and C over A. A listwise model can use the surrounding candidates to reduce redundancy, but its decision may change when irrelevant items are inserted or the input order changes. These are testable properties. Shuffle the same list, inject duplicates and add a clearly irrelevant control. A robust ranking should preserve the important order unless list composition legitimately changes the coverage objective.

The query is also part of the target. “Password reset policy” and “Password reset policy excluding contractors after the latest control change” are not the same information need. FollowIR was designed to test whether retrieval systems respond appropriately to detailed instructions rather than treating them as decorative text.[9] In a governed system, the instruction should state the required source type, exclusions, effective time and evidence role. The reranker should then be tested by changing one instruction at a time and checking whether the order changes in the expected direction.

Interaction buys conditional discrimination, not semantic omniscience. Its value is highest when candidate pairs differ through qualifiers or relations that cheap matching compressed away, and when the labels reward those distinctions. It is lowest when the task can already be solved by an exact identifier, a deterministic rule or a first-stage score with little remaining headroom.

Part IIIThe viable operating region

The usual reranking experiment fixes a candidate depth, applies a model and reports a final ranking metric. That can demonstrate a gain, but it does not reveal why the gain occurred or whether a cheaper intervention would do the same. A viable evaluation sweeps candidate depth, measures the candidate oracle, retains the original first-stage order, and records latency and downstream utility.

Candidate depth changes three things at once

Increasing K can raise candidate recall because the pool admits more items. It also raises reranking cost, commonly close to linear in the number of query-candidate evaluations for a pointwise cross-encoder. Finally, it changes the negative distribution presented to the reranker. More candidates can include harder useful negatives, but also remote documents unlike the training pool.

It is tempting to assume that quality rises monotonically with K. The “Drowning in Documents” study challenged that assumption by evaluating rerankers as progressively more documents were scored. In its tested settings, gains diminished and quality could degrade beyond a point, with high scores assigned to irrelevant documents.[11] A broader comparison of cross-encoders and language-model rerankers also found that model choice and reranking depth mattered more in out-of-domain evaluation than in the familiar MS MARCO setting.[10]

Depth is a resource curve, not a safety knob

The right candidate depth is the smallest one that reaches the required sufficiency with stable ranking quality. Finding it requires a sweep, not a convention. For each query slice, evaluate K at several depths, construct the candidate oracle, apply the reranker and record the full latency distribution. The useful plot has four curves: candidate coverage, oracle quality, realised quality and cost. A larger K is justified only while the extra oracle headroom is both recoverable and worth its operational price.

Mean latency is not enough. Pointwise cross-encoders score each query-candidate pair, so workload grows with candidates, token length and traffic. Batching can improve device utilisation, but queues, long documents and bursty queries create tail latency. Listwise models may reduce the number of calls while increasing prompt length and sensitivity to context limits. The architecture decision must therefore use p95 or p99 service time, timeout behaviour and fallback quality, not a notebook average.

Depth can also be conditional. Exact identifiers may need ten candidates and no reranker. Ambiguous paraphrases may justify one hundred candidates and a cross-encoder. Multi-evidence questions may require iterative retrieval because a first passage reveals the entity or exception needed for the next query. ReFIT illustrates this distinction by feeding relevance information back into another retrieval step rather than merely permuting the original pool.[8] Once the system widens or changes the pool, it is no longer a pure reranker, and its additional retrieval behaviour must be evaluated separately.

A cascade can stop early when evidence is decisive. A deterministic identifier match may bypass semantic ranking. A small reranker may handle easy candidates, with a larger model reserved for close margins or required bundle coverage. The escalation signal should be validated against error, not chosen because a score “looks uncertain”. Margin size, disagreement between rankers, missing evidence roles and candidate-oracle estimates are possible inputs, but each needs calibration on held-out queries.

The viable region is bounded on both sides. Too little depth omits evidence. Too much depth consumes budget and may expose the model to a noisier distribution than it can rank reliably. The optimum belongs to a workload and a failure tolerance, not to the reranker name.

A second thought experiment: equal final scores, unequal futures

Consider two systems with the same nDCG@10 today. System X retrieves almost all judged-relevant items into its top one hundred but orders them poorly before reranking. System Y retrieves only half of them, while its reranker perfectly orders what survives. Their final score may coincide on a small test set. They are not equally improvable.

Vary only the reranker. System X has headroom; System Y does not. Vary only the retriever. System Y may improve sharply; System X may not. A single end metric hides this difference. The candidate oracle reveals it by asking what the final metric would be under perfect ordering of the current pool.

Three negative controls

A serious reranking evaluation needs conditions under which reranking should not appear necessary. First, an oracle-order baseline shows the maximum available gain. Second, an already-correct first-stage order tests whether the reranker adds noise where no headroom exists. Third, an omitted-answer case tests whether the evaluation mistakenly credits fluent but non-authoritative substitutes.

Evaluation labels are part of the system

The candidate oracle is only as trustworthy as its judgements. MS MARCO enabled large-scale ranking research with real queries and human-generated answers, while TREC Deep Learning added rigorous reusable judgements for selected test queries.[12][13] BEIR then showed why a model that excels in one homogeneous regime may not generalise across domains and tasks.[4]

In an enterprise corpus, relevance is often graded and decision-specific. A passage may be topically similar yet insufficient to support a claim. An older document may be semantically exact but superseded. A chunk may contain one half of a rule. Judging only “looks related” trains the system to prefer plausible context, not decision-sufficient evidence. The evaluation set should therefore label at least relevance, authority, temporal applicability and whether the passage contains the material proposition.

FollowIR adds another warning: retrieval models can struggle to use detailed instructions rather than short keyword-like queries.[9] When the information need includes exclusions, jurisdiction or required evidence type, the reranker’s input contract must carry those instructions and the evaluation must test whether changing them changes order in the right direction.

stateDiagram-v2
  [*] --> Claim
  state "A thought experiment, the<br/>sealed tray" as Claim
  state "The ceiling in one inequality" as Test
  state "Two losses, two owners" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Part IVTest the stage, not the fashion

A production-shaped experiment begins with a decision, not a reranker catalogue. Consider a synthetic internal policy assistant that must return the operative paragraph, one corroborating procedure and any applicable exception. The final context budget allows four passages. The corpus contains policies, procedures, FAQs, archived notices and meeting notes.

The first-stage system uses hybrid lexical and dense retrieval, fused into one hundred candidates. A cross-encoder then scores each query-candidate pair. Before comparing models, the team samples queries across exact identifiers, paraphrases, exclusions, temporal questions and multi-evidence questions. Reviewers judge the top one hundred deeply enough to estimate candidate sufficiency and construct a candidate oracle.

Worked production example

For exact policy identifiers, lexical retrieval already places the operative document near the top. Ordering headroom is small, so a reranker adds latency without changing the evidence set. For natural-language paraphrases, relevant passages appear within the top one hundred but are scattered; headroom is large and cross-encoder interaction helps. For exception questions, the appendix fragments are absent; reranking gains on topical FAQs do not improve decision correctness.

The architecture response is mixed. Keep the simple baseline for identifier queries. Route paraphrase queries through the reranker. Repair exception retrieval by changing chunk boundaries, adding section lineage and issuing a second structured query for appendices. This is better than one global “reranking on” switch because each path addresses the loss it can actually change.

Recent shared-task systems still use this staged pattern. A 2026 SemEval multi-turn retrieval entry combined query rewriting, hybrid retrieval and cross-encoder reranking, reporting its result against an official task benchmark.[15] The important lesson is architectural rather than numerical: stages can complement one another, but each stage needs its own evidence and failure tests.

Build judgement from the decision backwards

The evaluation set should begin with what a reviewer must be able to establish, then identify evidence that would support or defeat it. This reverses the common process of sampling convenient query-document pairs and calling similarity “relevance”. For the policy assistant, a judgement record can name the operative claim, required evidence roles, acceptable sources, effective-time condition and any disqualifying source state. Reviewers can then label whether each candidate is merely topical, individually supportive, contradictory or part of a complete bundle.

Some questions have several valid evidence bundles. One procedure may restate the policy accurately; another path may combine the policy with a current control notice. The oracle should accept all defensible bundles rather than force one canonical paragraph. At the same time, near-duplicates should not earn repeated credit if the final task requires coverage. A diversity-aware metric or bundle-completion measure can distinguish four copies of one rule from a rule, its exception and the relevant procedure.

Judgement depth matters because an apparently low candidate ceiling may be an annotation artefact. If only the top results from an old system were judged, a new retriever can surface unjudged relevant material that the oracle treats as irrelevant. TREC-style pooling makes reusable evaluation practical, but no finite pool proves that the rest of the corpus is irrelevant.[13] Audit a sample of unjudged high-ranked candidates, record assessor disagreement and mark uncertain labels instead of silently converting them to zero.

Use temporal splits where documents and policies evolve. Randomly splitting near-duplicate passages can leak wording and version families into both train and test. A temporal test asks whether the system ranks the operative version after a change, suppresses superseded text and handles novel phrasing. A source-held-out or domain-held-out slice tests whether the reranker learned transferable relevance rather than one collection’s annotation style, a concern reinforced by heterogeneous evaluations such as BEIR.[4]

A candidate oracle is an instrument, not ground truth delivered by nature. Its labels, pooling process and evidence unit determine the headroom it reports. Publish those choices with the score so that a weak ceiling is not mistaken for a hard property of the retrieval system.

The reranking viability test

The following instrument turns that reasoning into a release gate. Apply it per query slice, not only as an overall mean.

1. Define the final evidence requirement. State the number and types of passages the downstream decision needs. A single-answer question and a coverage task require different relevance labels.
2. Measure candidate sufficiency. At several depths, estimate how often all material evidence enters the pool. Record omission causes such as vocabulary mismatch, chunking, access filtering, freshness or query underspecification.
3. Build the candidate oracle. Sort the current pool by trusted graded labels. The oracle’s final score is the best a pure reranker can achieve without new retrieval.
4. Calculate ordering headroom. Subtract the current first-stage final metric from the candidate-oracle metric. Low headroom is evidence against adding a reranker.
5. Measure headroom capture and regressions. Compare the reranked score with both the baseline and oracle. Inspect slices where the reranker degrades an already-correct order.
6. Price the gain. Record p50 and tail latency, compute, throughput, memory, operational dependencies and cost per additional successful query.
7. Verify downstream effect. Confirm that better ranking changes answer support, human disposition or another business outcome. A ranking gain that does not survive the consumer is not yet value.

Metrics that preserve the diagnosis

Use Recall@K or graded candidate coverage to measure whether the pool contains the necessary evidence. Use nDCG, MRR or task-specific ranking measures at the final cutoff to measure order. Report the candidate-oracle score beside both. Then add operational measures: candidates scored per query, tokens processed, latency percentiles, model failures and cost.

For answer-generation systems, keep retrieval evaluation separate from generation evaluation. A generator may answer correctly from weak evidence through prior knowledge, masking retrieval failure. It may also fail despite strong evidence. The cleanest retrieval test asks whether the required evidence appears, whether it is high enough in the final context, and whether an independent reader can support the claim from that context.

Do not report reranker lift only on queries whose answer was already retrievable. That conditional analysis is useful for mechanism, but it overstates system-wide value. Publish both the conditional gain when evidence is present and the unconditional gain across the full workload.

Executable viability audit

The artefact below implements the decomposition on three synthetic cases: an answer present but buried, an authoritative exception absent from the pool, and a first-stage order that is already correct. It uses only the Python standard library. The positive case must improve; the omission case must remain impossible; the already-correct case exposes a regression.

Run it with python reranking_viability.py. Expected output includes candidate recall, baseline and reranked nDCG@3, the candidate ceiling, omission gap, ordering headroom and realised gain. Replace the synthetic cases with judged query pools from an evaluation set.

#!/usr/bin/env python3
"""A dependency-free audit for deciding whether a reranker can repair a retrieval failure.

The audit separates:
1. candidate omission, which no pure reranker can repair; and
2. ordering headroom, which a reranker may capture.

All examples are synthetic. Replace the cases with judged query pools from your own
retrieval evaluation. Run with: python reranking_viability.py
"""

from __future__ import annotations

from dataclasses import dataclass
from math import log2
from typing import Iterable, Sequence


@dataclass(frozen=True)
class Candidate:
    """One retrieved item and its trusted relevance grade."""

    candidate_id: str
    grade: int
    first_stage_score: float
    reranker_score: float


@dataclass(frozen=True)
class QueryCase:
    """A synthetic query, its candidate pool, and all corpus-relevant grades."""

    name: str
    candidates: tuple[Candidate, ...]
    corpus_relevant_grades: tuple[int, ...]
    final_cutoff: int = 3


def dcg(grades: Sequence[int], cutoff: int) -> float:
    """Discounted cumulative gain using gain 2**grade - 1."""

    return sum((2**grade - 1) / log2(rank + 2) for rank, grade in enumerate(grades[:cutoff]))


def ndcg(grades: Sequence[int], ideal_grades: Sequence[int], cutoff: int) -> float:
    """Normalised DCG at cutoff, returning zero when no relevant item exists."""

    ideal = dcg(sorted(ideal_grades, reverse=True), cutoff)
    return 0.0 if ideal == 0.0 else dcg(grades, cutoff) / ideal


def order(candidates: Iterable[Candidate], score_name: str) -> list[Candidate]:
    """Return candidates in descending order for a named score field."""

    if score_name not in {"first_stage_score", "reranker_score", "grade"}:
        raise ValueError(f"Unsupported score field: {score_name}")
    return sorted(candidates, key=lambda item: getattr(item, score_name), reverse=True)


def grades(candidates: Sequence[Candidate]) -> list[int]:
    return [item.grade for item in candidates]


def analyse(case: QueryCase) -> dict[str, float]:
    """Compute pool sufficiency, baseline, reranked and candidate-oracle quality."""

    baseline_order = order(case.candidates, "first_stage_score")
    reranked_order = order(case.candidates, "reranker_score")
    oracle_order = order(case.candidates, "grade")

    relevant_total = sum(1 for grade in case.corpus_relevant_grades if grade > 0)
    relevant_in_pool = sum(1 for item in case.candidates if item.grade > 0)
    candidate_recall = 1.0 if relevant_total == 0 else relevant_in_pool / relevant_total

    baseline = ndcg(grades(baseline_order), case.corpus_relevant_grades, case.final_cutoff)
    reranked = ndcg(grades(reranked_order), case.corpus_relevant_grades, case.final_cutoff)
    ceiling = ndcg(grades(oracle_order), case.corpus_relevant_grades, case.final_cutoff)

    omission_gap = max(0.0, 1.0 - ceiling)
    ordering_headroom = max(0.0, ceiling - baseline)
    realised_gain = reranked - baseline
    headroom_capture = realised_gain / ordering_headroom if ordering_headroom > 1e-12 else float("nan")

    return {
        "candidate_recall": candidate_recall,
        "baseline": baseline,
        "reranked": reranked,
        "ceiling": ceiling,
        "omission_gap": omission_gap,
        "ordering_headroom": ordering_headroom,
        "realised_gain": realised_gain,
        "headroom_capture": headroom_capture,
    }


def decision(metrics: dict[str, float]) -> str:
    """A deliberately simple decision rule for the synthetic demonstration."""

    if metrics["candidate_recall"] < 0.95 or metrics["omission_gap"] > 0.05:
        return "FIX RETRIEVAL: the pool omits too much relevant evidence"
    if metrics["ordering_headroom"] < 0.02:
        return "SKIP OR REDUCE RERANKING: little recoverable ordering error"
    if metrics["realised_gain"] <= 0:
        return "REJECT THIS RERANKER: it failed to capture available headroom"
    return "RERANKING IS VIABLE: validate latency, slices and downstream outcome"


def synthetic_cases() -> tuple[QueryCase, ...]:
    """Positive, omission and already-correct controls."""

    return (
        QueryCase(
            name="A. Answer is present but buried",
            corpus_relevant_grades=(3, 2),
            candidates=(
                Candidate("distractor-a", 0, 0.99, 0.10),
                Candidate("distractor-b", 0, 0.96, 0.08),
                Candidate("guide", 2, 0.72, 0.91),
                Candidate("distractor-c", 0, 0.68, 0.12),
                Candidate("operative-policy", 3, 0.64, 0.98),
            ),
        ),
        QueryCase(
            name="B. Authoritative exception is absent",
            corpus_relevant_grades=(3,),
            candidates=(
                Candidate("topical-faq", 0, 0.97, 0.99),
                Candidate("archived-note", 0, 0.84, 0.73),
                Candidate("meeting-summary", 0, 0.78, 0.65),
            ),
        ),
        QueryCase(
            name="C. First stage is already correct",
            corpus_relevant_grades=(3, 2),
            candidates=(
                Candidate("operative-policy", 3, 0.99, 0.70),
                Candidate("guide", 2, 0.95, 0.98),
                Candidate("distractor-a", 0, 0.60, 0.96),
            ),
        ),
    )


def pct(value: float) -> str:
    return "   n/a" if value != value else f"{value * 100:6.1f}%"


def main() -> None:
    header = (
        f"{'Case':44} {'Recall':>8} {'Base':>8} {'Rerank':>8} "
        f"{'Ceiling':>8} {'Omit':>8} {'Headroom':>10} {'Gain':>8}"
    )
    print(header)
    print("-" * len(header))

    for case in synthetic_cases():
        result = analyse(case)
        print(
            f"{case.name:44} {pct(result['candidate_recall'])} {pct(result['baseline'])} "
            f"{pct(result['reranked'])} {pct(result['ceiling'])} {pct(result['omission_gap'])} "
            f"{pct(result['ordering_headroom'])} {pct(result['realised_gain'])}"
        )
        print(f"  -> {decision(result)}")

    # Properties that must hold for every pure reranker in this audit.
    for case in synthetic_cases():
        result = analyse(case)
        assert result["reranked"] <= result["ceiling"] + 1e-12
        assert result["ceiling"] <= 1.0 + 1e-12


if __name__ == "__main__":
    main()

Release gates

A reranker should enter service only when candidate sufficiency is high enough for the decision, ordering headroom is material, the candidate model captures that headroom on relevant slices, and the downstream outcome improves within the latency budget. It should remain off for slices with negligible headroom. It should trigger a retrieval repair when omission dominates.

Set the thresholds from consequence. A research browser can tolerate an occasional weak order because the reader can inspect more results. A system that supplies four passages to an automated decision needs stricter bundle completion, source authority and fallback behaviour. Report the residual omission rate in the same units as the decision risk. “Recall improved by three points” is less useful than “the operative exception remained absent in six of two hundred high-consequence cases”.

Operationally, log the candidate-set identifier, query representation, first-stage scores, reranker version, final order and evidence selected. These records make regressions replayable. They also prevent a common observability mistake: recording only the final context, which makes it impossible to tell whether a missing passage was never retrieved, was filtered as ineligible or was retrieved and then demoted.

Canary a new reranker against frozen candidate sets before changing retrieval at the same time. This isolates ordering behaviour. Then repeat the end-to-end test with live candidate generation. Monitor by slice because an overall gain can hide damage to exact identifiers, long documents, rare entities or instruction-heavy questions. A rollback must restore the previous model, preprocessing and score interpretation together.

Current libraries make cross-encoder experimentation accessible. Sentence Transformers, for example, documents retrieve-and-rerank workflows and cross-encoder APIs.[14] Convenience should not collapse evaluation. A few lines of inference code do not answer whether the workload sits in the reranking quadrant of the two-error plane.

Compact glossary

Candidate set

The items admitted by first-stage retrieval for later comparison.

Candidate sufficiency

The graded coverage of required relevant evidence present in that set.

Candidate oracle

The best possible order of the existing candidates under trusted labels.

Ordering headroom

The metric gap between the current order and the candidate oracle.

Headroom capture

The fraction of available ordering improvement realised by the reranker.

Pure reranker

A component that only reorders existing candidates and performs no new retrieval.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The authoritative paragraph that no model could rank
  M->>E: A thought experiment, the sealed tray
  E-->>M: compare rival explanations
  E->>D: Fix the stage that lost the evidence
  D-->>Q: revise scope or act

Fix the stage that lost the evidence

Reranking is often described as the last mile of retrieval. The phrase is accurate only if the road already reaches the destination. A cross-encoder, pairwise model or listwise language model can spend richer computation on evidence that broad search has admitted. It cannot order an absent document, restore a dropped table or override a legitimate entitlement boundary.

The changed architecture decision is to gate reranking on measured candidate sufficiency and ordering headroom. When the pool is sufficient and the order is weak, rerank. When the pool is insufficient, repair retrieval, representation or query formation. When both fail, repair both. When neither fails, keep the simpler baseline.

The changed research experiment is equally concrete: report the first-stage order, candidate oracle, reranked order, omission gap, headroom capture and cost across candidate depths and query slices. This turns a model comparison into a causal diagnosis. It also creates a result that can fail honestly. A stronger reranker may still be the right answer, but only after the system proves that ranking is the error it actually has.