TL;DR
Once every chunk is ranked by similarity, you choose how many to send the model — that count is K. Too low and you miss the answer; too high and you bury it in noise and burn tokens. K = 3–5 is the workhorse default.
How many chunks?
Retrieval ranks every chunk by similarity to the question. K is where you draw the line — the number of top-ranked chunks you actually paste into the prompt. It's a precision/recall dial:
| K | Cost | Risk |
|---|---|---|
| 1 | Cheapest | If the top chunk is slightly off, the whole answer is off |
| 3–5 | Moderate | The sweet spot — enough to triangulate, not enough to bloat |
| > 10 | Expensive | Quality often drops — the model loses the thread in the noise |
The surprising part is the bottom row: more context can make answers worse. The model attends to everything you give it, so ten marginally-relevant chunks dilute the two that actually mattered. That's the checkpoint — going from K=4 to K=20 buried the signal.
Analogy
K is how many search results you'd hand a colleague before asking them to write the summary. One link is a gamble. The top three or four give them enough to cross-check. Twenty tabs and they spend all their effort just figuring out which ones to ignore.
See it for yourself
The widget runs top-K retrieval over a small fixture corpus. Slide K up and down to see which chunks fall in and out — and watch the similarity scores drop off as you go further down the ranking. The chunks past the first few are exactly the low-score ones that add noise without adding signal.
What goes into the prompt
After retrieval, the API prepends a Context: block to the agent's system prompt:
Context:
[1] <chunk content> (source: filename)
[2] <chunk content> (source: filename)
That [N] indexing isn't just formatting — it's a contract with the model. When
the model states a fact, we ask it to point back with [1] or [1,2] to the chunk
it used. Those markers become clickable citations (last lesson of this module).
Common mistake
Treating a low-quality answer as "need more chunks" and cranking K up. If the right chunk isn't in the top few, the problem is usually upstream — chunking, the embedding match, or the search mode — not K. Raising K past ~5 mostly adds noise. Fix retrieval quality first; only then tune K.
Key takeaways
Key takeaways
- K is how many top-ranked chunks you send to the model.
- K = 3–5 is the default; K = 1 is fragile; K > 10 usually hurts quality.
- More context isn't better — past a point, extra chunks bury the relevant one.
- Retrieved chunks enter the prompt as a numbered
Context:block the model cites with[N].
Go deeper: K interacts with chunk size and the context budget
K isn't an isolated knob — it multiplies with chunk size:
- Tokens in prompt ≈ K × chunk_size. Small chunks let you afford a larger K for the same token budget; large chunks force a smaller K. Tune them together, not separately.
- Small chunks + larger K is often the better combination: each chunk is on-topic (precision) and a slightly larger K recovers answers that span two adjacent chunks (recall) — the spillover problem from Module 3.
- A reranker raises the ceiling. Production systems retrieve a generous K (say
- cheaply, then a reranker model rescores those candidates and keeps the best
- You get wide recall without paying the dilution cost of sending all 20 to the generator. Out of scope for the lab, but it's how teams push past the K=3–5 plateau.
The mental model: K controls how much evidence reaches the model; chunk size controls how concentrated each piece of evidence is. Balance both against your token budget.