How RAG Actually Works

The model doesn't know your documents. It knows the internet as of a training cutoff, frozen, and nothing about your company's wiki, last night's support tickets, or the PDF you uploaded thirty seconds ago. RAG — retrieval-augmented generation — is the plumbing that hands the model the right pages at the right moment, so it answers from your world instead of its memory.

Here's the whole trick, in the order it happens.

1. Chop the corpus into chunks. Your documents get split into passages — a few hundred words each. Chunk too big and you drown the useful sentence in noise; too small and you sever the thought in half. Chunking is unglamorous and it decides everything downstream.
2. Turn each chunk into a vector — an embedding. An embedding is a list of numbers that acts as coordinates in a space of meaning: passages that mean similar things land near each other, even when they share no words. "Cancel my plan" and "how do I end my subscription" end up as neighbors.
3. Store the vectors in a vector database. The database's one job is fast nearest-neighbor search — given a point, find the closest ones — across millions of chunks.
4. At question time, embed the question the same way and ask the database for the nearest chunks: the passages whose meaning sits closest to the question's.
5. Paste those chunks into the model's context alongside the question, with an instruction: answer using these.
6. The model generates — grounded in the retrieved text, not its frozen memory.

That's it. No retraining, no fine-tuning; the model's weights never change. You're not teaching it — you're handing it a cheat sheet, freshly assembled for each question.

Why it works. It grounds the answer in real, current, checkable text (you can show the source). It's fresh — update the corpus and the next answer reflects it, no retraining. And it's cheap next to fine-tuning a model every time a document changes.

Where it breaks — the honest half, and the half most explainers skip:

• Retrieval is only as good as the chunks. Chop badly and you retrieve half a thought; the model fills the gap by guessing.
• "Nearest in meaning" is not "most relevant." The database returns what's similar, which isn't always what answers — a passage can be about the right topic and still be the wrong page.
• Missing answers don't come back empty. If the answer isn't in your corpus, the model will still answer — confidently, from frozen memory — unless you build the guardrail that makes it say I don't know. This is the failure that ships to production most often.
• More is not better. Stuff too many chunks in and the model loses the thread — the "lost in the middle" problem, where the passage that mattered was buried in the pile.

The one-sentence version: RAG doesn't make the model smarter. It makes it better-briefed — and a well-briefed model beats a brilliant one talking about things it half-remembers.

Anchored links (per the Layered Autopsy): → Layer III · FOUNDATION — Embeddings (what a vector actually is) · → Layer IV — Vector Databases (how nearest-neighbor search scales).