Skip to main content
LLM Engineering · lesson 14 of 27

Beginner

8

Intermediate

11

Advanced

8

Also part of LLM Engineering

4
Open the LLM Engineering page →
LLM Engineering intermediate Lesson 14 of 27

RAG Failure Diagnosis

Why is my RAG system giving the wrong answer? Three real failure modes — bad chunking, stale documents, context overload — broken, then diagnosed.

By SkillByExample · Published

“Build a RAG system” tutorials show the pipeline working. Production RAG systems spend most of their debugging time on the pipeline not working — a real question, a confident wrong answer, and no error message anywhere. Three genuine failure modes, each reproduced and measured rather than described.

Failure 1: Bad Chunking Splits the Answer in Half

def chunk_text_bad(text, chunk_size=8, overlap=0):
    words = text.split()
    chunks, start = [], 0
    while start < len(words):
        end = min(start + chunk_size, len(words))
        chunks.append(" ".join(words[start:end]))
        start += chunk_size - overlap
    return chunks

refund_policy = ("Customers may request a full refund within 30 days of purchase. "
                  "After 30 days and up to 90 days, a 50% store credit is issued. "
                  "After 90 days, no refunds are available.")

chunks = chunk_text_bad(refund_policy)
for i, c in enumerate(chunks):
    print(f"chunk {i}: {c!r}")
$ python bad_chunking.py
chunk 0: 'Customers may request a full refund within 30'
chunk 1: 'days of purchase. After 30 days and up'
chunk 2: 'to 90 days, a 50% store credit is'
chunk 3: 'issued. After 90 days, no refunds are available.'

A query about “what happens after 60 days” retrieves chunk 2 on its own — 'to 90 days, a 50% store credit is' — which is missing the word “issued” and the entire next sentence. If only this one chunk makes it into context, the model has the number (50%) but not confirmation of what it means, and has to guess or hedge. This isn’t a prompting problem; the information was cut in half before the model ever saw it. Fixing it means re-chunking with a larger chunk_size, more overlap, or splitting on sentence boundaries instead of a fixed word count — never a prompt change, since the missing text simply isn’t in the context to reason about.

Failure 2: A Stale Document Wins by a Hair

import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

DOCS = [
    {"id": "refund_2024", "year": 2024, "content": "Customers may request a full refund within 30 days of purchase."},
    {"id": "refund_2019", "year": 2019, "content": "Customers may request a full refund within 14 days of purchase."},
]

embed_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = embed_model.encode([d["content"] for d in DOCS])

query = "How many days do I have to get a refund?"
q_emb = embed_model.encode([query])
sims = cosine_similarity(q_emb, embeddings)[0]
for d, s in zip(DOCS, sims):
    print(f"[{s:.4f}] ({d['year']}) {d['content']}")

top = np.argmax(sims)
print(f"\nTop match: {DOCS[top]['content']!r} (from {DOCS[top]['year']}, similarity {sims[top]:.4f})")
print(f"Similarity gap: {abs(sims[0]-sims[1]):.4f}")
$ python stale_document.py
[0.7352] (2024) Customers may request a full refund within 30 days of purchase.
[0.7435] (2019) Customers may request a full refund within 14 days of purchase.

Top match: 'Customers may request a full refund within 14 days of purchase.' (from 2019, similarity 0.7435)
Similarity gap: 0.0084

The outdated 2019 policy (14 days) genuinely outranks the current 2024 policy (30 days) — by less than a percentage point of cosine similarity, but it wins. Vector similarity measures semantic closeness to the query, and both sentences are almost identically phrased — nothing about embedding similarity encodes “this document is current.” If a company’s old policy document never got deleted from the vector store, this is exactly how a customer gets told the wrong refund window with complete model confidence. The fix lives at ingestion time (delete or flag superseded documents, filter by a stored timestamp), not at query time.

Failure 3: Context Overload Dilutes the Signal

import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

DOCUMENTS = [
    {"id": "policy_001", "title": "Refund Policy",
     "content": "Customers may request a full refund within 30 days of purchase. "
                 "After 30 days and up to 90 days, a 50% store credit is issued. "
                 "After 90 days, no refunds are available."},
    {"id": "policy_002", "title": "Shipping Policy",
     "content": "Standard shipping takes 5-7 business days. Express shipping costs $15."},
    {"id": "policy_003", "title": "Account Security Policy",
     "content": "Passwords must be at least 12 characters. Two-factor authentication is required."},
    {"id": "policy_004", "title": "Warranty Policy",
     "content": "Electronics carry a 1-year manufacturer warranty covering defects."},
]

embed_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = embed_model.encode([d["content"] for d in DOCUMENTS])

query = "How long is my refund window?"
q_emb = embed_model.encode([query])
sims = cosine_similarity(q_emb, embeddings)[0]
ranked = sorted(zip(DOCUMENTS, sims), key=lambda x: -x[1])

for k in [1, 4]:
    print(f"\n=== top-k={k} ===")
    top_k = ranked[:k]
    relevant_frac = sum(1 for d, s in top_k if d["id"] == "policy_001") / k
    for d, s in top_k:
        print(f"  [{s:.3f}] {d['title']}")
    print(f"  Fraction of context actually about refunds: {relevant_frac:.0%}")
$ python context_overload.py

=== top-k=1 ===
  [0.525] Refund Policy
  Fraction of context actually about refunds: 100%

=== top-k=4 ===
  [0.525] Refund Policy
  [0.250] Shipping Policy
  [0.128] Warranty Policy
  [0.032] Account Security Policy
  Fraction of context actually about refunds: 25%

k=4 doesn’t retrieve 4 relevant documents — it retrieves the 1 relevant one plus 3 barely- related ones at similarity scores as low as 0.032, and hands the model a prompt where only a quarter of the provided context actually addresses the question. A model asked to “answer using only the provided context” now has three plausible-looking but wrong documents competing for its attention alongside the right one. Higher k looks like it should only add information, but past the point where relevant results run out, it adds noise instead.

Practice

1. Increase the chunk size in Failure 1 to 15 words and confirm the refund sentence no longer gets split. 
def chunk_text_bad(text, chunk_size=15, overlap=3):
    words = text.split()
    chunks, start = [], 0
    while start < len(words):
        end = min(start + chunk_size, len(words))
        chunks.append(" ".join(words[start:end]))
        start += chunk_size - overlap
    return chunks

refund_policy = ("Customers may request a full refund within 30 days of purchase. "
                  "After 30 days and up to 90 days, a 50% store credit is issued. "
                  "After 90 days, no refunds are available.")
for i, c in enumerate(chunk_text_bad(refund_policy)):
    print(f"chunk {i}: {c!r}")
chunk 0: 'Customers may request a full refund within 30 days of purchase. After 30 days and'
chunk 1: '30 days and up to 90 days, a 50% store credit is issued. After 90'
chunk 2: 'issued. After 90 days, no refunds are available.'

The full “50% store credit is issued” phrase now survives inside one chunk (chunk 1). Larger chunks reduce the chance of a critical detail landing exactly on a boundary — the trade-off is each chunk now also carries more unrelated text alongside the relevant sentence.

2. Add a third, even older refund policy (2015, 7 days) and see where it ranks. 
import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

DOCS = [
    {"id": "refund_2024", "year": 2024, "content": "Customers may request a full refund within 30 days of purchase."},
    {"id": "refund_2019", "year": 2019, "content": "Customers may request a full refund within 14 days of purchase."},
    {"id": "refund_2015", "year": 2015, "content": "Customers may request a full refund within 7 days of purchase."},
]
embed_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = embed_model.encode([d["content"] for d in DOCS])
q_emb = embed_model.encode(["How many days do I have to get a refund?"])
sims = cosine_similarity(q_emb, embeddings)[0]
for d, s in sorted(zip(DOCS, sims), key=lambda x: -x[1]):
    print(f"[{s:.4f}] ({d['year']}) {d['content']}")
[0.7435] (2019) Customers may request a full refund within 14 days of purchase.
[0.7382] (2015) Customers may request a full refund within 7 days of purchase.
[0.7352] (2024) Customers may request a full refund within 30 days of purchase.

The 2019 policy still wins, and the current 2024 policy comes in last of all three — not even the oldest one wins, which confirms there’s no recency signal in play at all here, just noise in how “30” vs “14” vs “7” happen to embed relative to the query’s wording. Every additional stale duplicate document makes this failure mode more likely to surface, not less; deleting superseded documents matters more as a document store grows, not less.

3. Add a reranker on top of the top-4 context-overload result and see if it fixes the dilution. 
from sentence_transformers import CrossEncoder

DOCUMENTS_BY_ID = {
    "policy_001": "Customers may request a full refund within 30 days of purchase.",
    "policy_002": "Standard shipping takes 5-7 business days. Express shipping costs $15.",
    "policy_003": "Passwords must be at least 12 characters. Two-factor authentication is required.",
    "policy_004": "Electronics carry a 1-year manufacturer warranty covering defects.",
}
query = "How long is my refund window?"
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, text) for text in DOCUMENTS_BY_ID.values()]
scores = reranker.predict(pairs)
for (doc_id, text), score in sorted(zip(DOCUMENTS_BY_ID.items(), scores), key=lambda x: -x[1]):
    print(f"[{score:.3f}] {doc_id}: {text}")
[2.536] policy_001: Customers may request a full refund within 30 days of purchase.
[-9.569] policy_004: Electronics carry a 1-year manufacturer warranty covering defects.
[-10.096] policy_002: Standard shipping takes 5-7 business days. Express shipping costs $15.
[-11.253] policy_003: Passwords must be at least 12 characters. Two-factor authentication is required.

The reranker separates the genuinely relevant document from the other three by a huge margin (2.536 vs. all below -9.5) — far more decisively than the retrieval-stage similarity scores did (0.525 vs. 0.250/0.128/0.032). A reranking pass after retrieval, before the prompt gets built, is a direct, measurable fix for context overload: keep only chunks above a score threshold instead of a fixed top-k.

Frequently Asked Questions

My RAG system answers confidently but wrongly. Where do I even start debugging?
Start at retrieval, not generation. Print exactly which chunks got retrieved for the failing query and read them yourself — if the correct answer isn't literally present in the retrieved text, the LLM never had a chance regardless of prompting. Only move on to prompt/generation debugging once you've confirmed the retrieved context actually contains the answer.
How do I know if my chunk size is too small?
Retrieve for a real failing query and check whether the answer spans a chunk boundary — a sentence, a number, or a qualifying clause (like '50% store credit') cut in half between two chunks. If the answer straddles a boundary, no amount of prompt engineering fixes it; only re-chunking (larger chunks, more overlap, or splitting on semantic boundaries instead of a fixed word count) does.
Why would an outdated document ever outrank the current one?
Vector similarity measures how semantically similar text is to the query, not how current it is. A 2019 policy stating '14 days' and a 2024 policy stating '30 days' are nearly identical sentences — both score similarly against 'how many days for a refund' — and the ranking between them can come down to noise. Without a timestamp filter, recency-preference boost, or deleting stale documents, similarity search has no signal that tells it which one is current.
Does retrieving more chunks (higher k) always help?
No — past a point, it actively hurts. Low-relevance chunks get included alongside the genuinely relevant one, diluting the context the model has to work from and giving it more opportunity to synthesize an answer from the wrong source. Measure what fraction of your retrieved context is actually relevant to a given query before assuming higher k is safer.
What's the actual fix once I've diagnosed which failure mode I have?
Bad chunking: increase chunk size/overlap or switch to semantic chunking. Stale/duplicate documents: add a recency filter or delete outdated documents at ingestion time, don't rely on similarity alone. Context overload: lower k, or add a reranking pass (see [RAG: Advanced Retrieval and Reranking](/llm/rag-advanced-retrieval-reranking/)) that filters low-relevance chunks before they reach the prompt.