What RAG Actually Does (and When You Shouldn't Use It)

Retrieval-augmented generation, or RAG, is the technique behind most AI systems that can cite a source, answer a question about your own documents, or stay useful after their training cutoff. It is also one of the most over-applied patterns in applied AI: easy to stand up as a demo, and deceptively hard to run once real people depend on the answers.

Long context windows did not kill RAG: answering from a retrieved passage can cost on the order of 300 times less than loading an entire corpus into the prompt.²

This is a builder’s explainer. It covers what RAG is, how the pipeline actually works, what you can build with it, and the part most guides skip: when a simpler tool beats it. I will use two systems I have built to ground the tradeoffs. One is a customer-support assistant that worked well until it had to scale. The other is the personal knowledge base behind this site’s research, which is deliberately small enough to stay reliable.

What is RAG, in one sentence?

RAG lets a language model look information up before it answers, instead of relying only on what it absorbed during training. Picture a closed-book exam versus an open-book one. A bare model sits the closed-book exam: it answers from memory, and when memory fails it guesses, confidently. RAG hands the model the open book. It retrieves relevant passages from an external store, adds them to the prompt, and asks the model to answer using that material. The name spells out the three moves: retrieve the relevant content, augment the prompt with it, generate the answer.

What problem does RAG actually solve?

A model with no retrieval has four blind spots: it hallucinates confident wrong answers when it lacks grounding, it goes stale the moment facts change after its training cutoff, it cannot see private or internal data, and it cannot show what it answered from. RAG addresses all four by retrieving real, checkable source text before the model answers.

First, hallucination. When a model has no grounding for a question, it produces fluent, plausible, wrong answers. Ask a bare model for a product’s return window or a specific API parameter it has never seen, and it will often invent a confident wrong one rather than decline. Retrieved source text gives it something real to answer from, and something a reviewer can check the answer against.

Second, staleness. Training data has a cutoff. Anything that happened after it, or changed since, is invisible to the base model: a price change, a new policy, a product that shipped last week. A retrieval store can be updated the moment the underlying facts change, without retraining anything.

Third, private knowledge. Your internal documents, a client’s product catalog, last week’s support tickets: none of that is in a public model’s training, and most of it never should be. Retrieval is how a model reads content it was never trained on, while that content stays in your own store, under your own access controls.

Fourth, provenance. A RAG system can show which passage it answered from. For anything with a compliance, legal, or trust requirement, a citable source is not a nice-to-have; it is the requirement. A support answer a customer will act on, or a research summary an analyst will forward, has to trace back to something a human can verify.

How does RAG work under the hood?

Strip away the vocabulary and RAG is a data pipeline with a model bolted on the end.

The offline half prepares the knowledge. You collect the source documents, clean them, and split them into chunks small enough to retrieve precisely but large enough to carry meaning. Each chunk is turned into an embedding, a list of numbers that positions the text by meaning rather than by keywords, and stored in an index, often a vector database.

The online half runs at question time. The user’s question is embedded the same way, the system finds the chunks whose embeddings sit closest to it, and those chunks are pasted into the prompt alongside the question. The model then generates an answer grounded in that retrieved text.

“Closest” is doing real work in that sentence. Embeddings place text in a space where two chunks about the same idea land near each other even when they share no keywords, and an unrelated chunk lands far away; that is what lets a question match an answer worded completely differently. Retrieval is also rarely a single lookup. A working system pulls the top handful of candidate chunks, not just the one nearest, and a good pipeline adds a re-ranking step that reorders those candidates before any of them reach the prompt. Getting the right passage into that shortlist, and to the top of it, is the part that quietly decides answer quality.

If you have built ETL pipelines, this is familiar work wearing new words. Chunking is extract-and-transform. The embedding store is a feature store. Retrieval is a lookup. The unglamorous data-engineering steps, cleaning the input and choosing sane chunk boundaries, decide the quality of the whole system far more than the choice of model does.

One clarification that matters later: the store does not have to be a vector database. Any index the model can query at answer time counts, including a curated, hand-maintained one. Hold that thought for the knowledge-base example below.

What can you actually build with RAG?

RAG shows up wherever a model needs to answer from a specific body of content rather than from general knowledge: internal knowledge bases, customer support, document and contract analysis, code assistants, research pipelines, and the web pages that AI answer engines cite. The common shapes:

• Internal knowledge bases. Employee-facing assistants that answer from HR policies, engineering docs, or runbooks; the model answers “what is our parental-leave policy” from the current handbook instead of guessing.
• Customer support and documentation. Assistants that answer product questions from help content and past tickets, so “how do I reset this device” resolves from the actual manual rather than a plausible-sounding invention.
• Document analysis. Question-answering over contracts, filings, or research libraries, where the answer must trace back to a specific clause or paragraph.
• Code assistants. Tools that retrieve the relevant parts of a large repository so the model reasons over the actual code, not a guess about it.
• Analyst and research pipelines. Systems that pull the right rows, passages, or prior findings into a prompt so a model can summarize or compare them.
• Being the content that gets retrieved. When an AI answer engine cites a web page, that page was pulled into a RAG pipeline you do not own. More on that below.

The through-line: RAG is the right tool when the knowledge is specific, changes, or is too large to hold in the prompt, and when you would rather update a document than retrain a model.

Each of these has a version that does not need RAG. If the knowledge base is a single page, paste it into the prompt. If the support content genuinely never changes, a well-written system message may be enough. RAG earns its keep when the corpus is too big, too private, or too fast-moving for those shortcuts, which is exactly the test in the next section.

When should you use RAG, and when shouldn’t you?

RAG is one of four ways to give a model the knowledge or behavior you need, and choosing wrong is the most common applied-AI mistake I see. Use RAG for knowledge that is large, changing, or private; fine-tune to change behavior, voice, or format; keep small and stable knowledge in the prompt with long context; and reach for an agent when the task needs multiple steps. Most production systems in 2026 combine them.

RAG versus long context

The obvious 2026 objection is that context windows now hold a million tokens or more, so why retrieve at all? Because a large window is neither free nor perfectly reliable. Loading an entire corpus means paying to process every token on every request; retrieving only the relevant passages can cost on the order of 300 times less on a realistic query.² Accuracy also degrades as the window fills. On one 2026 test, answer accuracy fell from roughly 92 percent at 256K tokens to about 78 percent at a million, with information buried in the middle of a long prompt suffering the most.³ There is a latency cost too: a model asked to read a million tokens before it answers is slower than one handed three relevant passages. The consensus this year is not either/or; the strongest systems use retrieval to pull the relevant 50K to 200K tokens from a much larger corpus, then let a long-context model reason carefully over that.⁶

RAG versus fine-tuning

These solve different problems and get confused constantly. RAG changes what a model knows; fine-tuning changes how it behaves. Reach for fine-tuning when you need a consistent voice, format, or structured output that prompting cannot reliably enforce, or when you need to cut latency and cost by distilling behavior into the weights. Reach for RAG when the information is large, changing, or private, and when you need citations.⁵

A support assistant I built shows why this is rarely a single choice. I fine-tuned a model on a library of good answers so its tone and structure matched how the company actually wanted to respond, then used RAG to feed it the current, specific answer to each incoming question. Fine-tuning fixed the behavior; retrieval supplied the knowledge. Neither alone would have carried it.

RAG versus agents

An agent plans and executes a multi-step task, calling tools and recovering from errors, rather than returning a single answer. If the task is “answer this question from these documents,” that is RAG. If it is “research this, then take three actions based on what you find,” that is an agent, and it often uses RAG as one of its tools. That combination, an agent that retrieves as it works, is what 2026 write-ups call agentic RAG.

When to reach for something simpler

If the knowledge is small and stable, put it in the prompt or the system message and skip the pipeline entirely. If it never changes and behavior is the real problem, fine-tune. RAG earns its complexity only when the corpus is big or moving.

Where does RAG break in practice?

Every RAG failure I have hit lives in the pipeline, not the model: bad chunk boundaries, retrieval misses that surface plausible-but-wrong passages, a stale index, low-quality source content, and a missing evaluation harness. Each one maps to a stage in the pipeline above.

• Bad chunking. Split a document in the wrong place and you retrieve half an answer, or a chunk that looks relevant and is not.
• Retrieval misses. The store returns plausible-but-wrong passages, and the model faithfully answers from the wrong context. Research in 2026 is explicit that retrieval quality, not generation, is where these systems most often fail.⁴
• Stale index. The source changed; the index did not. Now the system answers confidently from last month’s truth.
• Garbage in. If the source content is messy, contradictory, or wrong, retrieval surfaces the mess faster, not slower.
• No evaluation harness. The demo answers five questions well, ships, and no one ever measures retrieval quality on the hundredth question or the thousandth.

The support assistant is my cautionary tale for the last two. It worked well in testing. But much of it ran locally, and the stress testing was not well defined before rollout. Maintenance, scaling, and hardening are where it bit us. A local RAG system that answers one-off questions is easy to keep running. The same architecture serving live customer traffic, where it becomes internal search and a front-line answer source, demands far more: monitoring, load testing, index-freshness guarantees, and a real evaluation loop. The demo and the production system look identical and are not the same thing.

The fix for most of these failures is boring and skipped anyway: a retrieval evaluation set. Pin a list of real questions to the passages that should answer them, then measure whether retrieval actually surfaces those passages as the content and the index change over time. Without that harness you are not measuring the system; you are sampling it once and hoping.

The knowledge base behind this site’s research is the opposite case, and it makes the same point from the other direction. It is a pure retrieval system: a curated wiki of concept, entity, and project pages that I query at research time instead of trusting a model’s memory. It has no vector database. Retrieval runs on a hand-maintained index and a graph of links between pages; the model reads the index, opens only the pages it needs, and answers from those. It is reliable precisely because its scope is honest. It serves one person, updates deliberately, and never pretends to be a real-time service. That is RAG sized to what it actually has to do, which is the whole lesson: the difficulty of a RAG system scales with how many people depend on the answer, not with how clever the retrieval is.

So what does this mean for SEO and getting cited?

If AI answer engines are RAG systems, then being cited by them is a retrieval problem, and that reframes what optimization means. When ChatGPT, Perplexity, or an AI Overview answers a question and links a source, that source was retrieved and chunked by a pipeline you do not control. Your page was a candidate chunk. It won or lost on whether it was retrievable and whether the relevant passage was clean, self-contained, and clearly about the question.

That maps directly onto the pipeline in this article. Chunking rewards content whose sections stand on their own; a paragraph that only makes sense after the three above it is a weak chunk. Retrieval rewards specificity and clear entities, because that is what makes an embedding land near a real question. Freshness matters because the index updates. Ahrefs’ analysis of ChatGPT citations found the first 30 percent of a page earning a disproportionate share of citations, and cited content carrying markedly higher entity density than average.¹ Read through the pipeline, those are not ranking tricks; they are descriptions of what retrieves well.

The practical version is unglamorous and familiar to anyone who has cleaned a data source: put the answer near the top, make each section self-contained, be specific and name entities, and keep it current. My inference, and I label it as one: as more discovery runs through retrieval, “writing for SEO” and “writing to be retrieved” converge into the same discipline, structuring content so a machine can extract a clean, self-contained answer. The chunk-level tactics deserve their own piece.

RAG is not magic and it is not obsolete. It is a data pipeline with a model at the end, and like any pipeline it is only as good as the plumbing, and only as reliable as the scope you hold it to.