Isn't hybrid search (keyword + vector) always better than either alone?

Not automatically. A naive weighted combination of BM25 and vector scores can be dominated by whichever signal has more extreme score spread for a given query — a strong keyword match can outrank a much more semantically relevant chunk. Hybrid search needs a fusion method that's robust to that, or a reranking pass afterward; simply averaging two scores is not enough on its own.

Why does a cross-encoder rerank better than the retrieval step that already ran?

A bi-encoder (used for vector search) embeds the query and each document separately, then compares vectors — fast enough to search millions of documents, but it never lets the query and document text directly interact. A cross-encoder processes the query and one candidate document together in a single forward pass, so it can model actual query-document relevance far more precisely — at the cost of only being fast enough to run on a small candidate set, not a whole corpus.

What is query rewriting and when does it help?

Reformulating the user's original query into one or more alternative phrasings before retrieval — expanding abbreviations, adding synonyms, or generating several different-angle queries (multi-query). It helps most when the user's literal wording doesn't closely match the vocabulary used in the source documents, which is common for conversational or vague queries.

What is reciprocal rank fusion (RRF)?

A way to combine multiple ranked lists (e.g. from BM25 and vector search) using only each document's *rank* in each list, not its raw score — score(d) = sum of 1/(k + rank_i(d)) across each ranking i, with k a constant (60 is a common default). RRF avoids the raw-score-scale problem naive hybrid search hits, since ranks are already comparable across very different scoring systems.

Do I need all of hybrid search, reranking, and multi-query in every RAG system?

No — start with whichever fixes the failure you're actually seeing. Reranking has the best cost/benefit ratio for most systems since it only touches a small candidate set. Hybrid search matters most when queries contain exact terms (product codes, error messages) vector search alone tends to miss. Multi-query costs an extra query (or several) per request, so it's worth it mainly for genuinely ambiguous or underspecified user queries.

RAG: Advanced Retrieval and Reranking

The basic RAG pipeline retrieves once and generates once. Production RAG systems usually retrieve a wider candidate set, then narrow it down more carefully — because the fast retrieval step and the accurate ranking step turn out to be different problems, best solved by different techniques.

Setup

Same shape of policy documents as the RAG lesson, chunked the same way, plus a fifth document (a password reset procedure) that overlaps in topic with the account security policy — on purpose, so retrieval has to actually discriminate between similar documents.

DOCUMENTS = [
    {"id": "policy_001", "title": "Refund Policy",
     "content": "Customers may request a full refund within 30 days of purchase. "
                 "After 30 days and up to 90 days, a 50% store credit is issued. "
                 "After 90 days, no refunds are available. Digital products are non-refundable "
                 "once downloaded. Damaged items qualify for full refund at any time with photo evidence."},
    {"id": "policy_002", "title": "Shipping Policy",
     "content": "Standard shipping takes 5-7 business days. Express shipping (2-3 days) "
                 "costs $15. Overnight shipping costs $35. Free standard shipping on orders over $50. "
                 "International shipping is available to 45 countries, taking 10-21 business days. "
                 "Orders are processed within 24 hours on business days."},
    {"id": "policy_003", "title": "Account Security Policy",
     "content": "Passwords must be at least 12 characters with uppercase, lowercase, "
                 "numbers, and symbols. Two-factor authentication is required for accounts with "
                 "more than $500 in transactions. After 5 failed login attempts, the account is "
                 "locked for 30 minutes. Security questions are deprecated; use TOTP apps instead."},
    {"id": "policy_004", "title": "Warranty Policy",
     "content": "Electronics carry a 1-year manufacturer warranty. Extended warranty "
                 "(3 years) is available at 15% of product price. Warranty covers defects in "
                 "materials and workmanship. Physical damage, water damage, and unauthorized "
                 "modifications void the warranty. Warranty service is provided via mail-in only."},
    {"id": "policy_005", "title": "Password Reset Procedure",
     "content": "To reset a forgotten password, click 'Forgot Password' on the login screen. "
                 "A reset link is emailed and expires in 15 minutes. The new password must meet "
                 "the standard complexity requirements. Support cannot reset passwords manually "
                 "for security reasons; only the automated email flow is available."},
]

def chunk_text(text, chunk_size=25, overlap=5):
    words = text.split()
    chunks, start = [], 0
    while start < len(words):
        end = min(start + chunk_size, len(words))
        chunks.append(" ".join(words[start:end]))
        start += chunk_size - overlap
    return chunks

CHUNKS = []
for doc in DOCUMENTS:
    for i, c in enumerate(chunk_text(doc["content"])):
        CHUNKS.append({"chunk_id": f"{doc['id']}_{i}", "doc_id": doc["id"], "title": doc["title"], "content": c})

print(f"Total chunks: {len(CHUNKS)}")

$ python rag_setup.py
Total chunks: 15

Save this file as rag_setup.py — every example below imports CHUNKS from it, the same way you’d split a real project into a setup module and separate scripts that use it.

Naive Hybrid Search Can Rank the Wrong Document First

from rag_setup import CHUNKS
import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
from rank_bm25 import BM25Okapi

chunk_texts = [c["content"] for c in CHUNKS]
embed_model = SentenceTransformer("all-MiniLM-L6-v2")
chunk_embeddings = embed_model.encode(chunk_texts)
bm25 = BM25Okapi([c.lower().split() for c in chunk_texts])

def hybrid_search(query, k=3, alpha=0.5):
    q_emb = embed_model.encode([query])
    vec_sims = cosine_similarity(q_emb, chunk_embeddings)[0]
    bm25_scores = np.array(bm25.get_scores(query.lower().split()))
    vec_norm = (vec_sims - vec_sims.min()) / (vec_sims.max() - vec_sims.min() + 1e-9)
    bm25_norm = (bm25_scores - bm25_scores.min()) / (bm25_scores.max() - bm25_scores.min() + 1e-9)
    combined = alpha * vec_norm + (1 - alpha) * bm25_norm
    top = np.argsort(combined)[::-1][:k]
    return [(CHUNKS[i]["chunk_id"], combined[i]) for i in top]

query = "How do I get my money back if a product breaks?"
print(f"Query: {query!r}\n")
for cid, score in hybrid_search(query):
    print(f"  {score:.3f}  {cid}")

$ python hybrid_search.py
Query: 'How do I get my money back if a product breaks?'

  0.836  policy_004_0
  0.676  policy_001_0
  0.500  policy_001_1

This is a genuinely wrong ranking. “Get my money back” is the Refund Policy’s territory (policy_001), but policy_004_0 — a Warranty Policy chunk — comes out on top, because it shares literal words (“product”) that give it a strong BM25 score, and min-max score normalization let that dominate the 50/50 blend. Pure vector search alone actually ranks policy_001 first for this exact query; naively averaging in a keyword score made the result worse, not better. This is the real failure mode hybrid search has to be designed around, not a hypothetical one.

Reranking Fixes It

from rag_setup import CHUNKS
from sentence_transformers import CrossEncoder

query = "How do I get my money back if a product breaks?"
# The candidate pool: whatever the first-stage retrieval (vector + BM25) returned
candidate_ids = ["policy_004_0", "policy_001_0", "policy_001_1", "policy_005_0", "policy_001_2"]
candidates = [c for c in CHUNKS if c["chunk_id"] in candidate_ids]

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, c["content"]) for c in candidates]
scores = reranker.predict(pairs)

ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
print(f"Query: {query!r}\n")
for c, score in ranked:
    print(f"  {score:.3f}  {c['chunk_id']}  ({c['title']})")

$ python rerank.py
Query: 'How do I get my money back if a product breaks?'

  -8.031  policy_001_1  (Refund Policy)
  -8.285  policy_001_0  (Refund Policy)
  -10.700  policy_004_0  (Warranty Policy)
  -10.798  policy_001_2  (Refund Policy)
  -11.282  policy_005_0  (Password Reset Procedure)

Feeding the query and each candidate together through a cross-encoder correctly promotes both Refund Policy chunks above the Warranty Policy chunk that the first-stage hybrid search got wrong. This is the standard production shape: a cheap, wide first-stage retrieval (vector and/or BM25, hundreds of candidates) followed by an accurate, narrow reranking pass (a cross-encoder, tens of candidates) — cross-encoders are too slow to run against an entire corpus, but plenty fast against the shortlist a first-stage search already narrowed down.

Multi-Query Retrieval Widens Coverage for Vague Queries

from rag_setup import CHUNKS
import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

embed_model = SentenceTransformer("all-MiniLM-L6-v2")
chunk_embeddings = embed_model.encode([c["content"] for c in CHUNKS])

def vector_search(query, k=3):
    q_emb = embed_model.encode([query])
    sims = cosine_similarity(q_emb, chunk_embeddings)[0]
    top = np.argsort(sims)[::-1][:k]
    return [CHUNKS[i]["chunk_id"] for i in top]

original_query = "I'm locked out of my account"
print(f"Original query: {original_query!r}")
print("Single-query retrieval:", vector_search(original_query, k=3))

# Rewritten variants — a real system generates these with an LLM call;
# hand-written here since this lesson has no live LLM API available
rewrites = [
    "account locked after failed login attempts",
    "how to reset a forgotten password",
    "two-factor authentication account access",
]
all_hits = []
print("\nQuery rewrites:")
for r in rewrites:
    hits = vector_search(r, k=2)
    print(f"  {r!r} -> {hits}")
    all_hits.extend(hits)

unique_hits = list(dict.fromkeys(all_hits))
print(f"\nMerged unique chunks from multi-query: {unique_hits}")

$ python multi_query.py
Original query: "I'm locked out of my account"
Single-query retrieval: ['policy_005_0', 'policy_003_1', 'policy_005_1']

Query rewrites:
  'account locked after failed login attempts' -> ['policy_003_1', 'policy_005_0']
  'how to reset a forgotten password' -> ['policy_005_0', 'policy_005_1']
  'two-factor authentication account access' -> ['policy_003_0', 'policy_003_1']

Merged unique chunks from multi-query: ['policy_003_1', 'policy_005_0', 'policy_005_1', 'policy_003_0']

The vague original query’s top-3 never surfaces policy_003_0 — the chunk that actually states the 2FA requirement. Three rewritten variants covering different angles of the same underlying need pull it in. Multi-query retrieval trades one extra retrieval pass (or three, here) for meaningfully wider coverage on exactly the kind of underspecified query real users type. These three techniques fix retrieval quality; when a RAG system is still giving wrong answers after applying them, RAG Failure Diagnosis covers the other angle — bad chunking, stale documents, and context overload.

Practice

1. Lower the hybrid search alpha to 0.8 (favor vector search much more) and see if it fixes the wrong ranking from the first example.

Appended to hybrid_search.py, reusing its embed_model, chunk_embeddings, and bm25:

def hybrid_search_alpha(query, alpha, k=3):
    q_emb = embed_model.encode([query])
    vec_sims = cosine_similarity(q_emb, chunk_embeddings)[0]
    bm25_scores = np.array(bm25.get_scores(query.lower().split()))
    vec_norm = (vec_sims - vec_sims.min()) / (vec_sims.max() - vec_sims.min() + 1e-9)
    bm25_norm = (bm25_scores - bm25_scores.min()) / (bm25_scores.max() - bm25_scores.min() + 1e-9)
    combined = alpha * vec_norm + (1 - alpha) * bm25_norm
    top = np.argsort(combined)[::-1][:k]
    return [(CHUNKS[i]["chunk_id"], combined[i]) for i in top]

query = "How do I get my money back if a product breaks?"
for cid, score in hybrid_search_alpha(query, alpha=0.8):
    print(f"  {score:.3f}  {cid}")

  0.800  policy_001_1
  0.760  policy_001_0
  0.737  policy_004_0

Yes — at alpha=0.8, both Refund Policy chunks (policy_001_1, policy_001_0) outrank the Warranty Policy chunk. Tuning the blend weight is a real lever, but it’s a global constant applied to every query; this fixes this query while potentially hurting a different query where the keyword signal was the one that mattered. Reranking fixes both without needing per-query tuning.

2. Test the cross-encoder reranker on a query where BM25 alone gets the right answer, and confirm reranking doesn't break it.

Appended to rerank.py, reusing its reranker and the imported CHUNKS:

query = "What is the shipping cost for overnight delivery?"
candidate_ids = ["policy_002_0", "policy_002_1", "policy_001_0", "policy_004_0"]
candidates = [c for c in CHUNKS if c["chunk_id"] in candidate_ids]
pairs = [(query, c["content"]) for c in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
for c, score in ranked:
    print(f"  {score:.3f}  {c['chunk_id']}  ({c['title']})")

  8.287  policy_002_0  (Shipping Policy)
  1.502  policy_002_1  (Shipping Policy)
  -10.791  policy_004_0  (Warranty Policy)
  -11.018  policy_001_0  (Refund Policy)

policy_002_0 (which literally contains “Overnight shipping costs $35”) wins by a wide margin — the reranker agrees with the obvious keyword match here rather than second-guessing it. Reranking isn’t biased against keyword matches; it just isn’t fooled by them the way naive score blending can be.

3. Add a fourth query rewrite and see if it changes which chunks get surfaced.

Appended to multi_query.py, reusing its vector_search function:

rewrites = [
    "account locked after failed login attempts",
    "how to reset a forgotten password",
    "two-factor authentication account access",
    "password complexity requirements",
]
all_hits = []
for r in rewrites:
    hits = vector_search(r, k=2)
    all_hits.extend(hits)
print(list(dict.fromkeys(all_hits)))

['policy_003_1', 'policy_005_0', 'policy_005_1', 'policy_003_0']

Same four chunks as with three rewrites — this particular addition didn’t surface anything new because “password complexity requirements” retrieves chunks already covered by the other rewrites. Multi-query retrieval has diminishing returns once the rewrite set already spans the query’s real angles; adding more rewrites mostly adds retrieval cost, not new coverage.

Setup

Naive Hybrid Search Can Rank the Wrong Document First

Reranking Fixes It

Multi-Query Retrieval Widens Coverage for Vague Queries

Practice

Frequently Asked Questions