← All writing
RAGRetrieval

Hybrid retrieval with BM25 and RRF

Why keyword search still earns its place next to vectors when every answer has to cite a legal article.

The Kuwaiti law agent has to answer with the exact article and line it relied on. That requirement shapes the retrieval more than anything else, and it is the reason I did not stop at vector search.

A lawyer does not want a passage that is about the right thing. They want Article 47, line 3. Retrieval for that job has to be good at two different kinds of question at once.

Two retrievers, two kinds of wrong

BM25 matches words. It is precise on exact terms such as article numbers and defined legal phrases, and it is blind to paraphrase. Vector search matches meaning. It handles loose, natural phrasing, and it can drift on exact terms, returning something related when the question named something specific.

Their failures barely overlap. A question the keyword search misses is often one the vectors catch, and the reverse. That is the whole argument for running both.

BM25exact terms, article numbers
Vector searchmeaning, paraphrase
RRF → one listrank-based, no score tuning
Both retrievers run on the pre-filtered corpus. Fusion only looks at ranks.

Narrow first

Before either retriever runs, a pre-filter narrows the corpus. With 77 laws and 7,399 articles, searching everything for every question wastes work and invites near-miss matches from the wrong law. Both retrievers then run on the narrowed set.

Fusing ranks, not scores

Now there are two ranked lists, and they have to become one. The obvious move is to add the scores. It does not work cleanly, because a BM25 score and a cosine similarity live on different scales, so any sum needs weights, and the weights need tuning.

Reciprocal Rank Fusion skips the problem by ignoring scores entirely. Each document earns points for where it ranks in each list: a high rank is worth more, and appearing in both lists adds up.

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

That is the whole algorithm. The constant k softens the gap between the top few ranks so that no single list dominates.

What comes next

The fused list goes to the answer step. A second, dedicated LLM call then extracts the verbatim lines to cite, which is what makes the line- and article-level citations possible. That split deserves its own post.

Hussam ElnemrAI & Backend Engineer in Cairo, open to remote roles in the EU and US.
See the work
Next postWhy I pick LangChain over LangGraph for multi-stage agents