What RAG actually does.
A step-by-step look at how documents become searchable context for a language model — and where the technique has real limits.
The problem RAG solves
A language model only knows what was in its training data, plus whatever text you put directly into its prompt. Retrieval-augmented generation exists because most useful questions people ask an AI tool depend on information that isn't in either place — an internal wiki page, a contract, a support ticket history. Instead of retraining a model on that data (slow, expensive, and instantly out of date the moment the source document changes), RAG finds the relevant passages at the moment of the question and inserts them into the prompt so the model can read them before answering.
How it actually works, step by step
First, documents are split into smaller chunks — usually a few hundred words each, since feeding an entire document into a search index makes it much harder to retrieve the one relevant paragraph out of twenty pages. Each chunk is then converted into an embedding: a vector of numbers produced by a separate embedding model, positioned so that chunks with similar meaning end up numerically close to each other, even if they don't share exact words. Those vectors are stored in a vector database.
When a user asks a question, the question itself gets embedded the same way, and the system searches the vector database for the chunks whose vectors are closest to the question's vector — this is the "retrieval" step. The top handful of matching chunks are then inserted into the prompt sent to the language model, along with the original question, and the model generates its answer using that inserted context rather than relying purely on what it memorized during training.
Where RAG quietly breaks
The most common failure mode isn't the language model — it's retrieval quality. If the chunking splits a table or a numbered list in the wrong place, the model may only see half the relevant information and answer confidently from an incomplete chunk. If a document uses different terminology than the question ("onboarding" vs. "new hire setup"), a purely vector-similarity search can miss it entirely unless the system also does keyword-based search alongside the vector search.
Document freshness is another practical issue: if the underlying source document changes, the index needs to be updated too, or the model will confidently answer using outdated chunks with no way to know they're stale. Most production RAG systems end up needing a re-indexing pipeline for exactly this reason, not just a one-time ingestion step.
What it's good for and what it isn't
RAG is well suited to answering questions grounded in a defined set of documents — internal knowledge bases, documentation, policy text. It's poorly suited to tasks that require reasoning across an entire large document holistically, or synthesizing patterns across hundreds of documents at once, since only a small number of chunks make it into any single prompt. Understanding that boundary is usually more useful than tuning the vector search further.