TL;DR
Retrieval-augmented generation (RAG) answers a question by first retrieving the most relevant chunks from your documents, augmenting the prompt with them, and then letting the model generate an answer it can read off the facts in front of it. The model doesn't memorize your data — it reads it, fresh, every time.
The idea
An LLM is great at language, but it has never seen your documents — anything you uploaded last week is invisible to it (the knowledge cutoff from Module 1). RAG closes that gap in three moves:
- Retrieve. Embed the user's question and search your vector index (the one you built in Module 3) for the chunks closest in meaning.
- Augment. Stitch those chunks into the prompt the model sees, right alongside the question.
- Generate. The model composes its answer with the right facts in view.
RAG = retrieve the relevant chunks, augment the prompt with them, then generate a grounded answer. The vector index is what Module 3 built.
The model now has the right facts in front of it as it writes. It doesn't have to know your refund policy — it just has to read the chunk that contains it.
Analogy
RAG is an open-book exam. The model is a strong test-taker, but it was never taught your specific material. Retrieval is flipping to the right page and laying it open before the model answers — so it reasons over your facts instead of guessing from memory.
Why not just fine-tune the facts in?
Fine-tuning bakes facts into the model's weights. It sounds tidy, but it loses on exactly the things that matter for a knowledge agent:
| Fine-tuning (bake facts in) | RAG (retrieve at query time) | |
|---|---|---|
| Update a changed policy | Retrain the model | Edit the doc, re-ingest |
| Show which source a fact came from | Impossible | A citation to the exact chunk |
| Remove one wrong fact | Hard once baked in | Delete/fix the chunk |
| Keeps one source of truth | No — weights drift from docs | Yes — the documents stay canonical |
Retrieval keeps your documents as the single source of truth and lets you point at the exact chunk that produced each fact. (Fine-tuning still has its place — for behavior and style, not changing facts; see Module 3's deep dive.)
What "relevant" means here
Relevance is the meaning-similarity from Module 3. You embedded each chunk into a vector; now you embed the question the same way and find the chunks whose vectors are closest. That's vector search, and it's what makes retrieval work even when the user's wording differs from how the document is written ("how long for a refund?" finds "we refund within seven business days").
Common mistake
Thinking RAG makes the model "smarter." It doesn't change the model at all — it changes the model's input, putting the right facts in the prompt. Garbage retrieval in, garbage answer out. The quality of a RAG system is mostly the quality of its retrieval, which is what the rest of this module tunes.
Key takeaways
Key takeaways
- RAG = retrieve relevant chunks → augment the prompt → generate an answer.
- The model reads your facts at query time; it never memorizes them.
- RAG beats fine-tuning for facts: cheap updates, citations, one source of truth.
- "Relevant" = closest in meaning (vector search over the Module 3 index).
Go deeper: where RAG can still go wrong
RAG removes the memorization problem but introduces its own failure modes — worth knowing before you tune retrieval:
- Retrieval miss. If the right chunk never gets retrieved, the model answers from its general knowledge (or hallucinates). Top-K and hybrid search (this module) exist to reduce misses.
- Context dilution. Retrieve too many chunks and the signal drowns; the model loses the thread. More retrieved ≠ better.
- Stale index. RAG is only as current as your last ingest. Change the docs but forget to re-ingest, and the agent confidently cites the old policy.
- No grounding guarantee. Putting a chunk in the prompt doesn't force the model to use it. Citations (last lesson) let you verify it actually did.
Each of these maps to a lever you'll pull in this module: K, search mode, and citations. RAG is a system you tune, not a switch you flip.