TL;DR
A raw LLM has two hard limits: it only knows its training data (knowledge cutoff), and everything it sees has to fit — and be paid for — in the context window. A knowledge base fixes both by retrieving just the few relevant sentences from your documents at query time and slotting them into the prompt.
The two limits
An LLM by itself can't answer questions about your world, for two reasons:
- Knowledge cutoff. The model only knows what was in its training data. It has never seen your billing handbook, your runbooks, or yesterday's incident notes.
- Context window. Anything the model does see has to fit in the prompt. Even very large windows aren't free — every extra token costs latency and money, and dilutes the signal the model can focus on.
Analogy
The model is a brilliant new hire who has read the entire public internet but has never seen a single one of your internal documents. Retrieval is handing them the exact page they need, at the moment they need it — not making them memorize the whole filing cabinet.
Example: the refund-policy question
A customer asks: "What's your refund window for digital downloads?" The model has never seen your policy, so it has only two moves:
- Guess — often confidently, often wrong. This is what hallucination looks like in the wild: a plausible-sounding "30 days" you never actually wrote.
- Say "I don't know" — accurate, but useless to the customer.
Neither is what you want. The fix is to retrieve the relevant fragment of your documents at runtime and put it in front of the model right before it answers — so it reads your real policy instead of inventing one.
Find the few sentences in my documents most relevant to the question, paste them into the prompt, and let the model answer with them in view.
That's retrieval in one sentence. The interesting parts — what counts as "relevant" and how to split documents so the relevant bit is findable — are the next lessons.
Why not just paste the whole handbook?
This is the tempting shortcut, and it's the checkpoint question. It fails for both limits at once:
- The context window. Even a 100-page PDF is too big to send on every turn — and you'd pay for all of it on every turn, even when 99% is irrelevant to the current question.
- The relevant part gets buried. Drop 50 pages plus a one-sentence question into the same prompt and the model has to wade through all of it — the one detail that matters is lost in the noise, and answer quality drops with it.
Less is more — but only if you can pick the right "less." That picking is what the rest of this module builds.
Common mistake
Treating the system prompt as the place to store knowledge. The system prompt sets behavior ("be terse, cite sources"); it's the wrong home for a 100-page handbook. Knowledge belongs in a retrievable store you pull from per question — not pasted in once and paid for forever.
Key takeaways
Key takeaways
- A raw LLM is limited by its knowledge cutoff and its context window.
- A knowledge base retrieves the few relevant sentences at query time.
- Pasting whole documents wastes tokens and buries the relevant part in noise.
- This module gets your data in; Module 4 wires retrieval into the running agent.
Go deeper: retrieval vs. fine-tuning vs. bigger context
Three ways to get an agent to "know" your data — they solve different problems:
- Retrieval (RAG). Keep documents in a store; fetch the relevant bits per query. Cheap to update (edit the doc, re-ingest), and the model only ever sees what's relevant. This is what you'll build.
- Fine-tuning. Bake knowledge into the model's weights by training on it. Expensive, slow to update, and it teaches style/behavior far better than it teaches facts — a fine-tuned model still hallucinates specifics.
- Just use a bigger context window. Tempting as windows grow, but you still pay per token every turn, and the model gets worse at finding the relevant detail as you stuff more in — the needle gets lost in a bigger haystack. Big windows complement retrieval; they don't replace it.
Rule of thumb: facts that change → retrieval; behavior/format → prompt (or fine-tuning at scale). You'll see why retrieval wins for facts throughout this lab.