Ingestion is the pipeline that gets your documents into a searchable index.
Sources include PDFs, web pages, databases, wikis and ticket systems.
The stages are load, parse, chunk, embed and index.
It usually runs as a batch job rather than at query time.
You also need a strategy for updates and deletions, not just first load.
Ingestion is the pipeline that turns a pile of documents into something searchable.
It is unglamorous and it decides most of what a RAG system can ever do.
The stages are load, parse, chunk, embed and index.
Sources are rarely tidy, arriving as PDFs, wiki pages, tickets, spreadsheets and databases.
Each needs its own loader, and each fails in its own way.
It usually runs as a batch job, since a query cannot wait for a hundred PDFs to be parsed.
The part most often forgotten is what happens on the second run.
Documents change, get deleted, and are replaced by newer versions.
Without a plan for that, the index quietly fills with contradictory versions of the same page.
A stable identifier per document lets you update in place rather than duplicating.
Content hashing avoids re-embedding everything when only three files changed.
Permissions must be captured at ingestion, since retrieval cannot invent them later.
Failures need logging per document, or a silently skipped file becomes an unanswerable question.
Ingest a small representative sample first and read what comes out the other end.
Most RAG systems that answer badly are ingesting badly, not retrieving badly.
One question on this card. Every option explains itself once you pick, right or wrong.
19 concepts in this module, then a graded test with every answer explained.