“Powered by RAG” shows up in a lot of AI product pitches. RAG stands for retrieval-augmented generation: before the model generates an answer, the system retrieves relevant documents and adds them to the prompt.
This guide goes from the 2020 paper that popularized the term to what teams learned building real RAG systems. Every figure comes from a paper published under a CC BY 4.0 license.
Why an AI needs to look things up
A large language model stores what it learned during training in its weights. That has three limits:
- It doesn’t know anything that happened after training.
- It can’t answer from information it never saw, like your company’s internal docs.
- It can produce answers that sound right but aren’t, known as hallucinations.
The paper that popularized the name, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” was posted to arXiv in May 2020. It splits knowledge into two kinds: parametric memory stored in the model’s weights, and non-parametric memory the model can consult from outside. The authors used a dense vector index of Wikipedia as that outside memory. They report state-of-the-art results on three open-domain question-answering tasks, and more specific, diverse, and factual text than a parametric-only baseline.
Put simply, RAG turns a closed-book exam into an open-book exam. Instead of answering from memory alone, the model gets to look up the relevant pages first.
RAG runs in two stages
“Seven Failure Points When Engineering a Retrieval Augmented Generation System,” a 2024 experience report from teams that built RAG systems, describes two processes.
1. Indexing (done ahead of time)
- Split documents into chunks and convert each chunk into an embedding, a list of numbers that captures its meaning.
- Store each chunk and its embedding in a database. This is the job of a “vector database.”
- The paper flags chunk size as a real trade-off: too small and some questions become unanswerable, too large and answers pick up noise.
2. Querying (every time a question comes in)
- Rewrite the question so it searches better, then embed it.
- Retrieve the top K chunks whose meaning is closest.
- Re-rank them by relevance and trim them to fit what the model can take in one prompt.
- Finally, the LLM reads the question plus the chunks and writes the answer.
When retrieval still gets it wrong

Image: Asai et al., “Self-RAG,” left portion of Figure 1 (CC BY 4.0, cropped)
This figure shows a standard RAG pipeline. For “How did US states get their names?”, the retriever pulls three documents (step 1). The language model then gets the question plus all three documents and writes an answer (step 2).
The answer has problems. Document 3 says California was named after a fictional island in a Spanish book, but the model writes that California was named after Christopher Columbus, flagged as Contradictory. The claim that Texas and Utah were named after Native American tribes appears in none of the retrieved passages, flagged as No information in passages. You can also see an irrelevant document slip in: document 2 is about popular baby names in Texas.
The Self-RAG paper uses this example to make a point. Retrieving a fixed number of passages every time, without checking whether retrieval is needed or whether the passages are relevant, can actually produce unhelpful answers. The takeaway: adding retrieval doesn’t make an AI immune to mistakes.
Seven failure points from real deployments
The “Seven Failure Points” paper draws on three case studies in research, education, and biomedicine. Its conclusion describes an empirical investigation involving 15,000 documents and 1,000 questions. Oddly, the paper’s own case-study table lists 4,017 documents for the biomedical (BioASQ) dataset, and the paper alone doesn’t make clear how the two figures relate.
| # | Failure point | What happens |
|---|---|---|
| 1 | Missing content | The answer isn’t in the documents, but the system makes one up instead of saying it doesn’t know |
| 2 | Missed the top-ranked documents | The right document exists but doesn’t rank high enough to make the top K |
| 3 | Not in context | It was retrieved but got cut during consolidation before reaching the model |
| 4 | Not extracted | The answer is in the context, but noise or conflicting information stops the model from pulling it out |
| 5 | Wrong format | The model ignores instructions to answer as a table or list |
| 6 | Incorrect specificity | The answer is too vague, or too detailed, for what the user needed |
| 7 | Incomplete | Not wrong, but missing information that was available in the context |
Points 1–3 are retrieval problems; 4–7 are generation problems. The wrong answer in the figure above looks most like point 4: the system found a document with the right answer, and the model still didn’t use it correctly.
Two conclusions from the paper are worth remembering. First, you can only really validate a RAG system while it’s running, because you can’t predict every question users will ask. Second, robustness evolves over time rather than being designed in up front. For point 7, the authors suggest that instead of asking “What are the key points in documents A, B, and C?”, you ask about each document separately.
The parts that make or break performance

Image: Wang et al., “Searching for Best Practices in Retrieval-Augmented Generation,” Figure 1 (CC BY 4.0)
RAG is less a single technique than an assembly of parts. This 2024 paper maps the pipeline as query classification, retrieval, reranking, repacking, and summarization, each with several possible methods. Blue text marks the options that performed best in the authors’ experiments.
A few findings are useful even if you never build one:
- Skipping retrieval when it isn’t needed pays off. Adding a query-classification step raised the average score from 0.428 to 0.443 and cut latency from 16.41 to 11.58 seconds per query.
- The most accurate option isn’t always the right one. Combining HyDE (which drafts a hypothetical answer to search with) and hybrid keyword-plus-semantic search scored highest, with a RAG score of 0.58, but took 11.71 seconds per query. The authors recommend plain hybrid or original retrieval to keep latency down.
- Put the most relevant context closest to the question. A “Reverse” ordering that does this scored best among repacking methods, with a RAG score of 0.560.
These numbers come from a fine-tuned Llama2-7B-Chat model and the authors’ chosen datasets, so treat them as a map of what you can tune rather than universal answers.
RAG vs. fine-tuning
The other way to give a model new knowledge is fine-tuning: training it further. If RAG is an open-book exam, fine-tuning is cramming the night before.
| RAG | Fine-tuning | |
|---|---|---|
| Where knowledge lives | External documents, looked up on demand | Model weights, learned in training |
| Updating information | Swap the documents | Retrain |
| Showing sources | Can show the retrieved documents | Hard |
“Fine-Tuning or Retrieval?”, published in December 2023, compared the two directly. Across a range of knowledge-intensive tasks, RAG consistently outperformed unsupervised fine-tuning, for both knowledge the model had seen in training and entirely new knowledge. The authors also found that LLMs struggle to learn new facts through unsupervised fine-tuning alone, and that showing the model many paraphrases of the same fact can help.
What to check in a “RAG-powered” product
RAG is a widely used way to ground an AI in fresh or private information. But as the figure shows, adding retrieval doesn’t guarantee a correct answer. When you use a RAG-based product, check three things:
- Does it cite its sources? That’s one of RAG’s biggest advantages. If there’s a link, click it at least once and confirm the claim is actually there.
- Does it say “I don’t know” when the documents don’t cover a question? If not, it’s vulnerable to failure point 1.
- Does it drop details when summarizing several documents at once? For anything important, ask about each document separately.