TL;DR
Ingestion is the one-time pipeline that turns a document into a searchable knowledge base: extract text → chunk → embed each chunk → store the vectors in an index. It runs once when you add a document. Searching that index at query time is Module 4 — this lesson is about building it.
The idea
The last two lessons gave you the two moving parts — chunking (split a document into answerable bites) and embeddings (turn each bite into a meaning-vector). Ingestion is the assembly line that runs them in order and stores the result:
- Extract the raw text from the source (a PDF, a text file, or a URL).
- Chunk the text into overlapping pieces.
- Embed each chunk into a vector with the fixed embedding model.
- Store every chunk and its vector in a vector index, ready to search.
Ingestion runs once per document: extract, chunk, embed, store. The stored vector index is what retrieval searches in Module 4.
Analogy
Ingestion is shelving a new book in a library. You don't photocopy the whole book every time someone asks a question — you index it once (catalog it, file it on the right shelf) so it can be found fast later. The vector index is the catalog; building it is a one-time job per book.
Example: a PDF becomes searchable vectors
Walk one document through the pipeline:
| Step | Input | Output |
|---|---|---|
| Extract | handbook.pdf (40 pages) | one long plain-text string |
| Chunk | the text + chunk_size / overlap | ~120 overlapping chunks |
| Embed | each chunk | ~120 vectors (384-D each) |
| Store | chunks + vectors | rows in the vector index, query-ready |
Notice ingestion is write-time work: it happens once, up front. Retrieval (Module 4) is read-time work: it embeds the question and searches these stored vectors. Doing the expensive embedding once at ingest is exactly what makes query-time retrieval fast.
Common mistake
Trying to "patch" a stored vector after editing the source. A vector isn't editable text — it's the embedding of a specific chunk. Fix the document, then re-ingest the affected piece so its chunk and vector are regenerated together. Editing one without the other leaves the index describing text that no longer exists.
Key takeaways
Key takeaways
- Ingestion = extract → chunk → embed → store, run once per document.
- It does the expensive embedding work up front so query-time search is fast.
- The output is a vector index: chunks plus their vectors, ready to retrieve.
- Edits mean re-ingesting the affected chunks, not patching stored vectors.
Go deeper: what 'store' really means, and what comes next
The "store" step is more than dropping vectors in a list:
- Chunk text travels with the vector. You store the original chunk text alongside its vector. Retrieval finds the nearest vectors, but it's the text that gets pasted into the model's prompt — so both must live together.
- Metadata earns its keep. Storing the source document, page, and position lets you build citations later ("from handbook.pdf, p.12") and lets you re-ingest or delete just one document's chunks without rebuilding everything.
- The index enables fast nearest-neighbor search. A naive search compares the query against every stored vector; production vector stores add an index (e.g. HNSW) so search stays fast as the corpus grows. Either way, the interface is the same: give it a query vector, get back the closest chunks.
With the index built, Module 4 picks up at read time: embed the question, retrieve the top-K chunks, and choose between vector, keyword, and hybrid search.