TL;DR
The unit of retrieval isn't the document — it's the chunk. You split documents into small pieces, embed each one, and retrieve only the pieces that match the question. Chunk size trades precision against context; overlap keeps a sentence that straddles a boundary intact in at least one chunk.
Why split at all?
A 200-page handbook is too large to embed as one vector — you'd blur the whole document into a single point and lose all information about where the relevant section is. So we split it into smaller pieces, embed each piece, and let retrieval pick the few pieces that match the question. The chunk is the smallest "answerable bite" of your document.
The size tradeoff
There's no perfect chunk size — it's a genuine tradeoff:
| Chunk size | Upside | Downside |
|---|---|---|
| Small (≈256 chars) | Tight precision — a hit is mostly about one topic | Answer may be split across several chunks; more chunks total |
| Large (≈1024 chars) | More context per hit; less likely to truncate a thought | Drags in irrelevant text that buries the part that matters |
256–1024 characters is a reasonable range. The lab lets you slide between them and feel the difference on a real document.
Fixed-size chunking with overlap
The simplest strategy is fixed-size with overlap:
- Slide a window of
chunk_sizecharacters across the document. - After each chunk, advance the window by
chunk_size − overlapcharacters.
The overlap is the safety net. Without it, a sentence that straddles a chunk boundary gets split — half in chunk N, half in N+1 — and neither chunk answers well on its own. With a small overlap (we default to 64 characters), each chunk reaches a little into its neighbors, so the boundary sentence appears whole in at least one of them.
Analogy
Think of tearing a long article into index cards. If you tear on exact character counts, a key sentence ends up half on one card, half on the next. Overlap is copying the last line of each card onto the top of the next — so no single thought is ever orphaned across the tear.
Try it yourself
Slide the chunk size down to 100 and watch the chunks multiply; slide it up to 800 and watch them grow. Notice the overlap region always carries the last 64 characters of the previous chunk into the next — that's the safety net at work.
Smarter splits when the document has structure
Fixed-size is the baseline. When a document has structure, you can do better:
- Paragraph-aware — never split inside a paragraph; cut on blank lines (and merge tiny paragraphs up toward the chunk size).
- Section-heading-aware — start a new chunk at every heading and keep that section together.
- Semantic — cut where the topic changes, measured by embeddings.
In this module's lab the chunking_strategy dropdown runs fixed-size,
paragraph, and section-heading for real. Switch strategy, hit Reprocess,
and watch the chunks re-split in the KB inspector — paragraph and section-heading
cut fewer chunks mid-sentence on structured documents like a handbook.
Common mistake
Blaming chunk size for a mid-sentence answer when the real culprit is overlap. A chunk ending mid-thought isn't about being too big or too small — it's that the sentence fell on a boundary with no overlap to carry it whole into a neighbor. Raise the overlap (or switch to a structure-aware strategy) before touching size.
Key takeaways
Key takeaways
- Retrieval works on chunks, not whole documents.
- Smaller chunks = precision; larger chunks = context. Pick per document.
- Overlap keeps boundary-straddling sentences intact in at least one chunk.
- Structure-aware strategies (paragraph, section-heading) beat fixed-size on structured docs.
Go deeper: chunk size is a retrieval decision, not a storage one
It's tempting to pick a chunk size to "fit nicely on screen." But chunk size is really tuning the retriever's behavior:
- On-topic hits vs. missed pieces. Smaller chunks make each hit tightly on-topic, but an answer that spans two chunks can get half-retrieved when only one of them comes back. Overlap and a slightly larger top-K (Module 4) recover the spillover.
- Embedding quality. Embedding models work best on a few hundred words at a time. Chunks much larger than that get blurred into an average, so they match less sharply — oversized chunks can actually reduce retrieval quality, not just bloat context.
- Re-chunking is cheap, re-deciding is not. You can re-run the chunker any time (the Reprocess button), but changing strategy mid-project means re-embedding everything. Get a feel for it on a sample before ingesting a large corpus.
The mental model: the chunker decides the smallest answerable bite; the retriever (next module) picks the few bites that look most like the question.