RAG vs. Fine-Tuning vs. Prompting

Real-World Scenario

A support bot for a startup called Nimbusreach Cloud gets asked about the refund policy. The base model has never heard of this company — but a confidently wrong answer is worse than no answer. This lesson solves the exact same question two ways and shows, with real generated output, why only one of them is grounded in fact.

Prompting Alone: Confident and Wrong

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

def ask(prompt: str, system: str = "", max_tokens: int = 60) -> str:
    messages = ([{"role": "system", "content": system}] if system else []) + [{"role": "user", "content": prompt}]
    chat_input = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tok(chat_input, return_tensors="pt")
    torch.manual_seed(0)
    out = model.generate(**inputs, max_new_tokens=max_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    return tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()

query = "What is Nimbusreach Cloud's refund policy for annual plans?"
print(ask(query))

$ python prompt_only.py
Nimbusreach Cloud offers two types of annual plans: Basic and Premium. The refund policy for these plans varies depending on the plan type.

For Basic Plans:
- If you cancel your subscription within 30 days after the initial payment date, you will receive a full refund.
- If you cancel

Nimbusreach Cloud doesn’t exist — this lesson made it up. The model has no knowledge of it whatsoever, and produces a fluent, specific, entirely fabricated policy (a “Basic/Premium” plan structure, a “30 days” window) anyway, with no hedge or uncertainty. This is exactly the failure prompting alone cannot fix: no amount of clever phrasing gives the model facts it was never trained on.

RAG: Retrieve the Real Answer, Then Answer From It

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

DOCS = [
    "Nimbusreach Cloud annual plans can be cancelled any time; refunds are prorated to the exact "
    "unused number of days, minus a flat $20 processing fee.",
    "Nimbusreach Cloud monthly plans are billed on the 1st and can be cancelled with no refund for "
    "the current month, effective the following cycle.",
    "Nimbusreach Cloud offers a 99.95% uptime SLA on all Business-tier and above plans, with service "
    "credits for any shortfall.",
]

def retrieve(query: str, k: int = 1) -> list[tuple[str, float]]:
    query_emb = embedder.encode([query])
    doc_embs = embedder.encode(DOCS)
    sims = cosine_similarity(query_emb, doc_embs)[0]
    top_idx = sims.argsort()[::-1][:k]
    return [(DOCS[i], float(sims[i])) for i in top_idx]

results = retrieve(query)
for doc, score in results:
    print(f"[{score:.3f}] {doc}")

$ python rag_retrieve.py
[0.782] Nimbusreach Cloud annual plans can be cancelled any time; refunds are prorated to the exact unused number of days, minus a flat $20 processing fee.

Real embedding similarity correctly ranks the annual-plan document above the monthly-billing and SLA documents for an annual-refund query — no keyword match was needed, just semantic similarity.

context = results[0][0]
rag_prompt = f"Context: {context}\n\nQuestion: {query}\n\nAnswer using only the context above."
print(ask(rag_prompt, system="Answer only from the given context. Do not add information not in the context."))

$ python rag_answer.py
Refunds are prorated to the exact unused number of days, minus a flat $20 processing fee.

Same model, same weights, same temperature=0 — the only difference is that the prompt now contains the real policy text instead of nothing. The answer is now exactly correct, with none of the invented “Basic/Premium” plan structure from the ungrounded attempt.

Where Fine-Tuning Would Differ

Neither of the two examples above needed fine-tuning, because the problem was missing knowledge, and RAG solves missing knowledge directly. Fine-tuning (fine-tuning.md, lora-qlora-peft.md) would be the right tool for a different symptom on this same bot — for example, if the facts were all correct but every answer needed to follow a strict POLICY: ... / EXCEPTIONS: ... / CONTACT: ... format regardless of phrasing, or needed to consistently match a specific brand voice. That’s a behavior/format problem baked into how the model responds, not a knowledge problem — RAG can hand the model the right facts, but it can’t make a general-purpose instruction-tuned model reliably follow a custom format better than a well-designed prompt can, at scale, without repeating the format instructions (and their token cost) on every single call.

Need current/private knowledge the model wasn't trained on?  -> RAG
Need a specific output format/tone applied consistently?     -> fine-tuning
Need a one-off task the model can already mostly do?         -> prompting

Practice

1. Ask the ungrounded model about Nimbusreach Cloud’s monthly plan (not annual) and confirm it’s equally confident and equally wrong.

   Solution

   print(ask("What is Nimbusreach Cloud's refund policy for monthly plans?"))

   I'm sorry, but I don't have specific information about the refund policies of Nimbusreach Cloud as it's not mentioned in my training data or publicly available information. However, I can provide some general guidance on how to handle refunds:
   
   1. Check your account balance: Before you initiate any refund process

   This phrasing of the question got an honest hedge instead of a fabrication — worth noticing, because it means the failure mode from the main example isn’t guaranteed on every question. That’s actually the more important, less comfortable lesson: you can’t rely on the model to reliably hallucinate or to reliably hedge without grounding, so you can’t tell which one you’re getting without retrieval to check against. A wrong answer that sometimes looks uncertain and sometimes doesn’t is harder to catch in production than one that’s always wrong the same way.

2. Retrieve for the monthly-plan question instead, confirm the correct document (the monthly one, not annual) now ranks first, and get the grounded answer.

   Solution

   monthly_query = "What is Nimbusreach Cloud's refund policy for monthly plans?"
   results = retrieve(monthly_query)
   for doc, score in results:
       print(f"[{score:.3f}] {doc}")
   
   context = results[0][0]
   rag_prompt = f"Context: {context}\n\nQuestion: {monthly_query}\n\nAnswer using only the context above."
   print(ask(rag_prompt, system="Answer only from the given context. Do not add information not in the context."))

   [0.813] Nimbusreach Cloud monthly plans are billed on the 1st and can be cancelled with no refund for the current month, effective the following cycle.
   Nimbusreach Cloud does not offer a refund policy for monthly plans.

   The right document was retrieved again (0.813, clearly the top match) and the answer is factually consistent with it — worded more tersely than the retrieved text itself, but not contradicting it. Grounding fixes what facts the model has access to; it doesn’t guarantee the model restates them verbatim.

3. Ask a question with no matching document at all (e.g., about a feature never mentioned in DOCS) and observe what the RAG pipeline retrieves and answers — does grounding help when there’s nothing relevant to retrieve?

   Solution

   bad_query = "Does Nimbusreach Cloud support single sign-on (SSO)?"
   results = retrieve(bad_query)
   for doc, score in results:
       print(f"[{score:.3f}] {doc}")
   
   context = results[0][0]
   rag_prompt = f"Context: {context}\n\nQuestion: {bad_query}\n\nAnswer using only the context above."
   print(ask(rag_prompt, system="Answer only from the given context. If the context doesn't answer the question, say so."))

   [0.499] Nimbusreach Cloud offers a 99.95% uptime SLA on all Business-tier and above plans, with service credits for any shortfall.
   No

   The similarity score (0.499) is noticeably lower than the 0.78-0.81 range the correctly-matched queries scored — a real, measurable signal that nothing relevant was actually found, even though some document still comes back as “top match” (retrieval always returns something; low similarity doesn’t mean empty results). A production system should threshold on this score and refuse to answer below some cutoff, rather than always injecting the top-1 result regardless of how weak the match is — a terse, unhelpful “No” is still better than confidently inventing an SSO policy the way the ungrounded model did earlier in this lesson.

4. Compare retrieve()’s top match against the two other documents’ scores for the original annual-plan query, to see the actual separation between the right document and the wrong ones.

   Solution

   results = retrieve(query, k=3)
   for doc, score in results:
       print(f"[{score:.3f}] {doc[:60]}...")

   [0.784] Nimbusreach Cloud annual plans can be cancelled any time; ...
   [0.758] Nimbusreach Cloud monthly plans are billed on the 1st an...
   [0.513] Nimbusreach Cloud offers a 99.95% uptime SLA on all Busi...

   The right document still ranks first, but only barely (0.784 vs. 0.758) — the annual and monthly refund documents are semantically close to each other (both are about billing/cancellation), and a generic sentence embedder doesn’t separate them by much. This is the realistic case, not the earlier hallucination example: retrieval got the right answer, but with a thin enough margin that a slightly different phrasing of the query could plausibly flip the ranking. rag-advanced-retrieval-reranking.md covers what to do when top-k scores are this close and a single embedding lookup isn’t confident enough on its own.