The retrieval problem, restated
Dense retrieval often looks strong against a curated pilot corpus. A real estate contains mixed formats, inconsistent terminology and identifiers with little semantic meaning. Performance can fall when evaluation finally includes this material. This article uses worked figures to show how each stage should be measured.
The weakness is structural. Dense embeddings connect “close an account” with “terminate a banking relationship,” but they are less dependable for account numbers, ISINs, case references and clause numbers. A query for REF-2024-88213 needs an exact match, not a plausible neighbour. Sparse retrieval handles lexical identity directly. A larger embedding model does not remove that requirement.
The architecture pairs Vector Search for dense retrieval with AlloyDB for sparse search, structured filters and metadata. Reciprocal rank fusion combines the lists, then a cross-encoder reranks a small candidate set before generation.
Why dense retrieval alone fails in practice
Dense retrieval embeds the query and document chunks in a shared space, then ranks them by distance. It is strong at synonym and paraphrase matching. Its weaknesses cluster around three common enterprise patterns.
The first weakness is identifier retrieval. The second is tabular precision: narrative commentary may outrank the requested row. The third is negation. An excluded concept can remain semantically close to the query.
Dense retrieval should not be dropped. It remains effective when staff use plain language and documents use formal terms. Dense retrieval needs a sparse partner, not a replacement.
Keeping sparse retrieval alive
Sparse retrieval, in the form of BM25 or a similar term-frequency and inverse-document-frequency ranking function, has been treated as legacy technology in much of the RAG literature, superseded by dense embeddings. In enterprise deployments the opposite pattern holds: sparse retrieval is frequently the single highest-value addition to a dense-only pipeline, because it directly fixes the identifier and exact-match failure mode described above, and it does so cheaply.
On Google Cloud, the practical home for sparse retrieval alongside a dense Vector Search index is AlloyDB, using its full-text search capabilities (built on PostgreSQL's tsvector and tsquery machinery, with AlloyDB's performance improvements over stock PostgreSQL making this viable at enterprise query volumes). Storing the same document chunks in AlloyDB with a generated tsvector column, alongside the structured metadata (document type, business unit, effective date, access control tags), gives a query path that can combine an exact lexical match with structured filters in a single SQL statement, something a pure vector index cannot express natively.
A worked compliance-retrieval illustration assumes a hand-labelled set of 500 analyst queries. Adding a BM25-style sparse path raises recall at ten from 71 percent for dense retrieval alone to 89 percent before reranking. The modeled gain comes mainly from product codes, identifiers and regulatory citations, the lexical signals dense retrieval often handles poorly. A real programme should establish its own baseline and interval rather than import these figures.
Vector search on scann, the dense half
Vector Search, formerly Vertex AI Vector Search, is Google Cloud's managed approximate-nearest-neighbour service. Its ScaNN lineage uses anisotropic vector quantisation to preserve inner-product search quality under compression. The resulting memory, recall and latency trade-off must still be measured on the intended corpus rather than inferred from the algorithm alone.
The main tuning decision is recall versus latency. num_leaves_to_search controls how much of the index is explored, while the neighbour count determines how many candidates reach later stages. Do not copy a leaf count from another corpus. Sweep the parameters against a brute-force baseline and choose the smallest search that meets the measured recall target within the p95 budget.
Freshness is easy to underestimate. A corpus with new case notes and revised product terms needs an incremental update path from the start. Measure the extra serving cost against the business freshness requirement rather than quoting a universal percentage premium.
Alloydb as the sparse and structured half
AlloyDB contributes more than sparse text search. It can apply entitlement, business-unit, effective-date and jurisdiction filters alongside retrieval. Its pgvector support may also be sufficient for a smaller corpus, avoiding a separate dense index until scale or latency proves the need.
One AlloyDB query can perform the sparse match and mandatory structured filters in the same pass. The result is then fused with dense candidates. Post-filtering is weak: removing unauthorized rows after retrieval can leave too few usable results and spends work on documents the user should never have retrieved.
PostgreSQL row-level security provides database enforcement beneath application filtering. An audit function can inspect that policy instead of trusting every retrieval path to apply entitlement correctly.
Reciprocal rank fusion
Once you have two ranked lists, dense results from Vector Search and sparse or filtered results from AlloyDB, you need a principled way to combine them into a single ranking, rather than arbitrarily interleaving or picking one list over the other. Reciprocal rank fusion (RRF) is the most dependable default for this: for each document, its score is the sum, across every list it appears in, of one divided by a constant plus its rank. A document ranked third in the dense list and seventh in the sparse list, with a constant of 60, contributes 1/(60+3) + 1/(60+7) to its fused score.
RRF avoids normalizing scores from incompatible systems. Cosine similarity and a BM25-style score do not share a meaningful scale, but rank positions can be combined. This usually makes RRF less brittle than a weighted sum when the document mix shifts.
The worked comparison uses k=60 and sends the top twenty fused candidates to reranking. Under its assumptions, top-ten recall moves from 89 percent for a naive union to 93 percent with RRF. Treat the table as an evaluation template, not a portable benchmark.
| Retrieval configuration | Top-10 recall | P95 query latency |
|---|---|---|
| Dense only (Vector Search) | 71% | 55 ms |
| Sparse only (AlloyDB text search) | 68% | 30 ms |
| Dense + sparse, naive union | 89% | 85 ms |
| Dense + sparse, RRF fusion | 93% | 90 ms |
| RRF fusion + cross-encoder rerank | 97% | 240 ms |
Cross-encoder reranking, the expensive last mile
RRF fusion gets you a good candidate set quickly and cheaply. But both the dense and sparse retrieval stages score query and document independently (bi-encoder style), meaning neither ever actually reads the query and the candidate document together. A cross-encoder reranker does exactly that: it takes the query and each candidate document as a single joint input and produces a relevance score conditioned on both, which is considerably more accurate but far more computationally expensive, because it cannot be precomputed or indexed.
This is why cross-encoder reranking is applied only to a small candidate set, typically the top 20 to 50 documents from the fusion stage, never to the full corpus.
The reranker reads the query and each of the top twenty candidates jointly, then returns the best five to eight. In the worked table, this moves recall from 93 to 97 percent and improves the first result. The required model size and candidate count must be chosen against the latency budget.
Reranking has a visible cost and latency increment. Model it from candidate count, query volume and the chosen serving endpoint. Compare that spend with the error and rework avoided. The answer may differ for a staff research tool, a real-time client interaction and a high-volume low-stakes search.
Latency and cost budget across the full pipeline
Measure latency at every stage: query embedding, parallel dense and sparse retrieval, fusion, reranking and generation. Keep retrieval and generation budgets separate so the team knows which stage moved. The worked design targets a few hundred milliseconds before generation, but region, index size, model endpoint and concurrency determine the observed distribution.
Build the cost ledger from embedding, vector serving, database compute, reranking and generation. Attribute shared infrastructure explicitly and date every unit price. In many designs generation dominates, but that is a measurement to establish rather than a premise. Do not remove reranking until an ablation shows that its saved cost exceeds the quality loss.
Query understanding before retrieval even starts
Enterprise queries are often fragments, pasted email paragraphs, misspelled product names or internal shorthand. A query-understanding step should sit before both retrieval paths. Do not assume the embedding model will repair every malformed request.
A lightweight pass can expand a maintained glossary, extract identifiers and generate one or two paraphrases for a terse query. Route extracted references directly to exact match. Evaluate the pass through zero-result rate, recall and added latency. A rewrite that changes the user's meaning should be treated as a failure, even when retrieval returns something plausible.
A second, related discipline is query classification: routing a query to a different retrieval configuration depending on its apparent type. An identifier-only query (matching a regular expression for known reference number formats) skips dense retrieval entirely and goes straight to an AlloyDB exact lookup, saving both latency and the risk of a plausible-but-wrong dense result outranking the correct exact match.
A long, narrative query (several sentences describing a situation rather than naming a document) is weighted more heavily toward the dense path, because narrative queries are precisely where semantic similarity outperforms lexical overlap. This routing logic is a handful of rules, not a trained classifier, and it is worth resisting the temptation to over-engineer it. The rules that mattered in practice covered perhaps six query shapes and captured the overwhelming majority of traffic.
Alternatives considered and set aside
Late-interaction retrieval such as ColBERT deserves consideration in a greenfield design. It represents query and document with token-level vectors and can approach reranker quality at lower online cost. The trade is operational: the team may need to run a separate index and take it through security, resilience and support review. Revisit the choice against current managed services rather than freezing an old platform assumption.
A unified hybrid embedding can reduce the number of indexes. It also couples semantic and lexical behaviour to one model version. A two-index design is easier to diagnose because dense, sparse, fusion and reranking outputs can be inspected independently. The evaluation should decide whether that observability is worth the extra component.
The final alternative is to skip reranking and pass many candidates to the generator. Test that through an ablation: compare top-six reranked context with a much larger unranked set. Measure groundedness, latency and token cost. Larger context is not free relevance filtering, and it can expose the generator to more distracting evidence.
Failure modes
The first failure mode is uniform chunking. Narrative policy, tabular finance data and short FAQ entries need different boundaries. Tables require row-level extraction with repeated header context or a structured path. A token window that cuts a table mid-row destroys its meaning. Compare chunking strategies on a format-balanced evaluation set.
The second failure mode is embedding model and index drift after a model version change. Switching embedding models requires a full corpus re-embedding and reindex, because vectors from two different embedding models are not comparable. A partial reindex, where old and new vectors coexist in the same index, silently corrupts every similarity comparison between an old-vector document and a new-vector query. This has to be run as an atomic cutover (build the new index fully, validate it against a frozen evaluation set, then switch traffic), never as an incremental migration of a live index.
The third is application-only access filtering, a single point of failure around sensitive documents. The fourth is displaying a reranker score as “94 percent match” without calibration. Reranker scores order candidates; they are not probabilities. Presenting them as confidence gives a wrong top result false certainty.
Worked example, relationship manager document retrieval
The worked scenario serves relationship managers across client agreements, term sheets, KYC records and case notes. It assumes 1.4 million source documents producing about nine million chunks. The hard requirements are zero retrieval outside the assigned client book and a visible citation from every passage to its source and page.
Format-aware ingestion chunks narrative text with overlap, extracts tables row by row and preserves each case note as a unit. The output is written to Vector Search and to AlloyDB with full-text representation and metadata. Each query resolves the caller's client book, enforces it before retrieval, takes forty candidates from each path, fuses the lists, reranks twenty and passes six to generation with citations. Those counts are starting parameters for evaluation, not constants.
The acceptance pack should report top-six recall on labelled queries, entitlement tests, citation validity and end-to-end resolution time. Segment recall by source age and format. A migrated archive with shorter legacy notes can otherwise look healthy in the blended metric while remaining underrepresented. Re-chunk and re-evaluate that segment before release.
Make query routing an explicit, testable policy
Hybrid retrieval performs best when it does not send every question through an identical path. An identifier, a broad policy question and a table lookup have different evidence shapes. A small deterministic router can recognise strong lexical signals before a model-based classifier handles the ambiguous remainder. The router's output is a policy decision that belongs in the trace.
The deterministic stage should inspect patterns such as policy codes, International Securities Identification Numbers, account references and quoted phrases. It should not infer business meaning. The classifier selects among approved routes and can abstain. An abstention may cost more because it runs both retrievers, but it is safer than confidently choosing the wrong one.
Routing quality needs its own labelled set. A query can produce the right answer despite a poor route because the corpus is forgiving. That accidental success disappears after corpus growth. Measure route selection, candidate recall and final answer separately. A retrieval score cannot diagnose a routing defect when all stages are collapsed into one metric.
| Query class | Initial route | Mandatory filter | Reranking policy | Primary acceptance measure |
|---|---|---|---|---|
| exact identifier | sparse first | entitlement plus identifier type | only for collisions | exact-match recall |
| policy explanation | dense and sparse | jurisdiction and effective date | cross-encoder by default | labelled passage recall |
| table or schedule | layout-aware structured route | product and client scope | row-context comparison | cell-to-header correctness |
| exploratory research | dense and sparse with expansion | approved corpus classes | budget-dependent | source diversity and recall |
| ambiguous request | both paths, conservative depth | full caller entitlement | rerank then abstain if weak | calibrated abstention |
Apply entitlement before both candidate generators
Sparse and dense stores often encode access scope differently. AlloyDB may use row policies and joins. Vector Search uses metadata restrictions attached to datapoints. Those mechanisms must derive from one entitlement vocabulary. A nightly comparison should prove that every indexed document has the same client, jurisdiction, confidentiality and purpose attributes as its source record.
Do not retrieve broadly and remove forbidden results after fusion. The dense service may expose a snippet, score or document identifier before that filter runs. It may also crowd authorised evidence out of the top candidate window. Pre-filtering is therefore a confidentiality control and a relevance requirement.
The test pack should create near-identical authorised and unauthorised documents. A query must return the authorised item without revealing the restricted twin. Repeat the test for direct queries, paraphrases and malicious instructions embedded in permitted content. Confirm the denial in both database and retrieval telemetry.
Operate corpus and model changes as paired releases
Retrieval behaviour changes when documents change, even if the code and embedding model remain fixed. A new policy can introduce lexical collisions. A document migration can remove headings that chunking relied upon. Treat corpus releases with the same discipline as model releases.
The release manifest should include source snapshot, parser, chunker, embedding model, sparse-index configuration, RRF constant, candidate depths and reranker. That set is the retrieval system. Recording only the embedding model leaves most behaviour unreconstructable.
Shadow evaluation catches distribution effects that a frozen set misses. Run sampled live queries against the current and proposed indexes without changing the user answer. Compare candidate overlap, critical-slice recall, latency and entitlement denials. Investigate large shifts even when the proposed system's aggregate score improves.
Rollback must restore a complete compatible set. Reverting the dense index while retaining new sparse tokenisation can produce an untested fusion. Keep the prior manifest and serve all referenced artifacts until the rollback window closes. A retrieval release is atomic at the manifest level, not at the individual index level.
Diagnose retrieval incidents from the candidate set outward
When an answer is wrong, start with the labelled evidence and work forward. If the correct chunk never entered either candidate set, inspect parsing, chunking, access filters and query routing. If it entered but ranked too low, inspect RRF depth and reranking. If it ranked high but the answer ignored it, the failure belongs to generation or prompting.
This sequence prevents expensive model changes from masking a data defect. It also assigns incidents to an accountable owner. Corpus teams own parse and metadata failures. Retrieval teams own candidate and ranking failures. Agent teams own evidence use and response behaviour. The shared trace connects the stages.
Retain query, route, entitlement scope, candidate identifiers and scores, fused rank, reranker version, selected passages and citation outcome. Sensitive passage text can follow a stricter retention policy than identifiers and hashes. The evidence must still allow a reviewer to reproduce the ranking against the retained index version.
Notes for practitioners
Do not evaluate a retrieval system on a hand-picked document set and extrapolate to production accuracy. Build the evaluation set from real queries against the real corpus, including its messiest formats, before committing to an architecture. Keep sparse retrieval in the pipeline permanently rather than treating it as a stopgap superseded by better embeddings, because the identifier and exact-match failure mode it solves is structural to dense retrieval, not a symptom of an under-trained model. Use reciprocal rank fusion rather than a hand-tuned weighted score combination when merging dense and sparse rankings, because it is less sensitive to corpus and query drift and requires no ongoing recalibration.
Include cross-encoder reranking in the design comparison whenever a wrong top result carries compliance or client-trust cost. Keep it only if the ablation shows that the gain justifies its serving and latency budget. Enforce access control at the database layer rather than only after retrieval, and verify it through logs and adversarial entitlement tests. Treat embedding upgrades as full-corpus cutovers behind a frozen evaluation gate. Design chunking by document format instead of forcing one token window across a heterogeneous corpus.
Retrieval is accepted by slices, not by an average
| Evaluation slice | Dense signal | Sparse signal | Reranker role | Failure to watch |
|---|---|---|---|---|
| natural-language paraphrase | primary | supporting | resolve semantic near-ties | plausible but wrong policy |
| identifier or clause number | supporting | primary | usually unnecessary | exact token lost in chunking |
| table value | weak without layout-aware chunks | useful for headers and labels | compare row context | narrative passage outranks row |
| negated request | candidate generation only | candidate generation only | test exclusion explicitly | prohibited concept returns |
| access-controlled corpus | only after filtering | only after filtering | never repairs leakage | filtering applied after retrieval |
Measure recall before generation. Score identifiers separately from prose queries. Treat access filtering as a precondition, not a relevance feature. A reranker can improve order. It cannot repair a candidate that was never retrieved. It must never see a document the caller cannot access.
The final gate should compare configurations against a frozen query set and a live shadow sample. Promote only when every critical slice stays inside its error budget. The BEIR benchmark provides useful background. An institution still needs its own corpus and query distribution.
Platform and method references
- Google Cloud, Vector Search overview.
- Google Cloud, Vector search with AlloyDB.
- PostgreSQL, Full Text Search and Row Security Policies.
- Cormack, Clarke and Buettcher, Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods, SIGIR 2009.
Service names, supported index modes and pricing can change. Confirm them against the linked product documentation before committing a production design.