Hybrid retrieval made my results worse
Everyone recommends fusing lexical and dense retrieval, then reranking. I measured it on a hand-labelled set and the default recommendation finished third of six configurations — behind dense retrieval on its own.
The received wisdom for production RAG is three steps: retrieve with BM25 and embeddings, fuse the two rankings, then rerank the survivors with a cross-encoder. It is in every architecture diagram.
I built a harness to measure it, expecting to confirm it. On my corpus, fusion scored below dense retrieval on its own, and the cross-encoder bought 1.8 points of recall for ten times the latency while nDCG went down.
The numbers
60 documents, 28 hand-labelled queries, k=5, CPU only.
| configuration | recall@5 | mrr@5 | ndcg@5 | latency |
|---|---|---|---|---|
| BM25 | 0.679 | 0.632 | 0.589 | 0.2 ms |
| TF-IDF | 0.643 | 0.634 | 0.587 | 8.8 ms |
| Dense (MiniLM) | 0.839 | 0.832 | 0.778 | 39.9 ms |
| Fusion (RRF) | 0.768 | 0.759 | 0.712 | 49 ms |
| Fusion + rerank | 0.821 | 0.817 | 0.751 | 516 ms |
| Dense + rerank | 0.857 | 0.827 | 0.774 | 397 ms |
0.839
dense alone
0.768
after fusing
−8.5%
relative recall
Why fusion lost
Reciprocal Rank Fusion gives every document a score of 1 / (60 + rank) from each ranking it appears in, then adds them up. It rewards documents that several retrievers agree on.
That is exactly right when your retrievers have complementary failures — when BM25 finds things the embeddings miss and vice versa. It is exactly wrong when two of your three retrievers are simply worse than the third, because then they outvote it.
RRF has no way to know one of its inputs is better. It is a democracy, and I had put two weak voters in the room.
The fix is not to weight the inputs — that reintroduces exactly the per-query normalisation problem RRF exists to avoid. The fix is to check whether the weak retrievers earn their place at all, and here they did not.
The premise I built the eval set around was wrong
I tagged every query lexical (phrased in the corpus's own vocabulary — "security definer bypasses row level security") or paraphrase (phrased as a user would — "can another customer see my uploaded files").
The expectation: BM25 wins the lexical ones, dense wins the paraphrases, fusion collects both.
| lexical | paraphrase | |
|---|---|---|
| BM25 | 0.850 | 0.583 |
| TF-IDF | 0.750 | 0.583 |
| Dense | 0.950 | 0.778 |
| Fusion | 0.900 | 0.694 |
Dense beat BM25 on the lexical queries too, 0.950 against 0.850. A modern sentence embedding has no trouble with exact term matches — it just also handles the other case. There was no complementary failure to collect, which is why fusion had nothing to offer.
The gap that did survive is dense's own: 0.950 on lexical against 0.778 on paraphrase. Questions phrased in the user's words are still the hard ones for everybody, and that is where retrieval work should go.
The reranker was a rounding error
A cross-encoder reads the query and the document together, so attention runs across both. It genuinely knows more than a bi-encoder, which embedded the document before it knew what would be asked. That is a real architectural advantage and it is why the two-stage pattern exists.
Applied to the strongest first stage:
- Dense alone: 0.839 recall, 0.778 nDCG, 40 ms
- Dense + rerank: 0.857 recall, 0.774 nDCG, 397 ms
Recall up 1.8 points. nDCG down — and nDCG is the only metric here sensitive to the order of several relevant results, which is precisely what a reranker is supposed to improve.
The likely reason is domain. ms-marco-MiniLM-L-6-v2 is a 22M-parameter model trained on web search passages, being asked about MCP internals and Postgres row-level security. On a domain it has never seen, "knows more" and "knows better" come apart.
Where it clearly did help: recovering from the fusion damage, 0.768 → 0.821. Which is an argument against the fusion, not for the reranker.
Running the reranker on two first stages is what made this legible
The measurement I nearly did not do. Reranking only the fused list tells you the pair's combined score and nothing about which half contributed:
# Reranking the fused list alone cannot tell you whether the reranker
# helped or merely undid damage the fusion did, so the strongest single
# retriever gets the same treatment and the two are compared directly.
stages = {
"fusion + rerank": fused,
"dense + rerank": {qid: [s.doc_id for s in hits]
for qid, hits in raw["dense"].items()},
}Two rows instead of one, and the conclusion inverts: fusion + rerank (0.821) still trails dense + rerank (0.857), so the reranker cannot fully undo what fusion cost.
If you are evaluating a pipeline stage, evaluate it on more than one input. A stage measured against a single predecessor is measuring the pair.
What I am not claiming
Sixty documents is not a corpus and this is not a benchmark. A different domain — legal text with terms of art, product SKUs, code identifiers — could easily flip every row here, and lexical retrieval's advantage on rare exact tokens is real even though it did not show up on my prose.
The transferable part is the method:
- Hand-label 25–30 queries against your own documents. It is an afternoon, and it is the only thing that turns an architecture argument into a table.
- Include queries phrased the way users phrase them, not the way your documents are written. That split was the most informative column.
- Measure each stage against more than one predecessor, or you cannot attribute the change.
- Report the configuration you expected to win, especially when it did not.
The whole harness is rag-reranker-lab. BM25 and the four metrics are written out rather than imported, because the point was to be able to read the scoring — pytest -q runs 36 tests with no network and no model download.
This started as a follow-up to deleting a similarity threshold, which was the last time a default I had accepted turned out to be the bug.