RAG (retrieval augmented generation) is a technique where an application first retrieves information relevant to a question, such as passages from documents, a database or the web, and then passes that information to a language model together with the question so the model can ground its answer in it. Anthropic's 2024 Contextual Retrieval post puts it in one line: "RAG is a method that retrieves relevant information from a knowledge base and appends it to the user's prompt."
This guide explains how RAG works, when it beats simply pasting everything into the prompt, how coding agents retrieve code differently, and the security risks to plan for.
Why do LLM apps need retrieval at all?
A model's built-in knowledge comes from its training data. It does not know your company's docs, last week's release notes or the contents of a private repo. Ask anyway and it may guess.
NIST's Generative AI Profile (NIST AI 600-1, July 2024) calls this risk confabulation: "The production of confidently stated but erroneous or false content (known colloquially as “hallucinations” or “fabrications”) by which users may be misled or deceived." It explains that confabulations are "a natural result of the way generative models are designed."
RAG is a common fix for missing knowledge. Anthropic's glossary says retrieval "allows the model to access and use information beyond its training data, reducing the reliance on memorization and improving the factual accuracy of the generated text." It adds that RAG is "particularly useful for tasks that require up-to-date information, domain-specific knowledge, or explicit citation of sources."
How does RAG work, step by step?
A classic RAG pipeline has two phases. Anthropic's post Introducing Contextual Retrieval (September 2024) lays out the preparation phase:
- Chunk. "Break down the knowledge base (the “corpus” of documents) into smaller chunks of text, usually no more than a few hundred tokens."
- Embed. "Use an embedding model to convert these chunks into vector embeddings that encode meaning."
- Store. "Store these embeddings in a vector database that allows for searching by semantic similarity."
Then, at question time:
- Retrieve. The question is compared against the stored chunks and the closest matches are found.
- Augment. "The most relevant chunks are added to the prompt sent to the generative model."
- Generate. The model answers from the question plus the retrieved text, ideally citing which chunk each claim came from.
The "retrieval" can be done by your code before the model runs, or by the model itself. Anthropic's glossary notes that "the model itself does not necessarily retrieve the data (but can with tool use and a retrieval function)."
Here is the shape of the final prompt, whatever the pipeline:
<documents>
<document index="1" source="billing-faq.md">...retrieved chunk...</document>
<document index="2" source="refund-policy.md">...retrieved chunk...</document>
</documents>
Answer the question using only the documents above. Cite the document index
for each claim. If the documents do not contain the answer, say so.
Question: Can a customer get a refund after 30 days?
What are embeddings and semantic search?
An embedding is a list of numbers that represents the meaning of a piece of text, so that passages about similar things end up close together. OpenAI's retrieval guide defines semantic search as "a technique that leverages vector embeddings to surface semantically relevant results," which can match even when the query and the passage share few words.
Pure semantic search has a weak spot: exact identifiers. Anthropic's post notes that embedding models "can miss crucial exact matches," and that BM25, a keyword ranking method, "is particularly effective for queries that include unique identifiers or technical terms." Error codes, function names and ticket numbers are exactly that kind of query, so many systems combine both.
Anthropic's own contribution, Contextual Retrieval, adds a short explanation of each chunk's context before embedding it. The post reports that this "can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%" on its tests.
Do you always need RAG?
No. If the material is small enough, putting all of it in the prompt is simpler. In the same 2024 post, Anthropic wrote: "If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods." Context windows have grown since: many current Claude models have a 1M-token window, according to Anthropic's context windows guide.
Bigger is not automatically better, though. The same guide warns that "As token count grows, accuracy and recall degrade, a phenomenon known as context rot." Retrieval keeps the context small and relevant. See LLM tokens explained for how context windows fill up.
| Approach | How it adds knowledge | Good fit | Trade-off |
|---|---|---|---|
| Long context | Paste all the material into the prompt | Small, stable sets of documents | Pays for every token on every request; recall drops as context grows |
| RAG | Retrieve only relevant passages per request | Large or changing knowledge bases, citations | Answers are only as good as retrieval |
| Fine-tuning | Further training on extra data | Style, format, domain patterns | Harder to update; Anthropic's glossary says the Claude API "does not currently offer fine-tuning" |
How do coding agents retrieve code?
Coding agents need the same thing RAG provides: the right files in context at the right time. Many take a different route to it.
Anthropic's post on context engineering observes that "many AI-native applications employ some form of embedding-based pre-inference time retrieval," but that teams are increasingly adding "just in time" strategies, where agents "maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools."
Its example is Claude Code: "CLAUDE.md files are naively dropped into context up front, while primitives like glob and grep allow it to navigate its environment and retrieve files just-in-time, effectively bypassing the issues of stale indexing and complex syntax trees." Code changes constantly, so an index built this morning can be wrong by lunch. Searching the live files avoids that.
Agents also retrieve through tools:
- Web search and fetch for documentation and error messages.
- MCP servers that expose databases, issue trackers or internal docs. See Claude Code MCP and best MCP servers.
- Skills that load their full instructions only when relevant. See Agent Skills.
That broader job is context engineering: RAG is one way of choosing what the model sees.
Can you get RAG without building a pipeline?
Yes. The big providers offer managed retrieval:
- OpenAI has vector stores, which its docs call "the containers that power semantic search for the Retrieval API and the file search tool." Files you add are "automatically chunked, embedded, and indexed," with a default of 800-token chunks and 400 tokens of overlap.
- Google offers a File Search tool for the Gemini API, which its embeddings docs present as a managed option, noting that "Building Retrieval Augmented Generation (RAG) systems is a common use case for AI products."
- Anthropic lets you build retrieval as a tool Claude calls, and its Messages API can return citations that point to the passages a response relied on.
Managed tools save setup time. You still own the quality of the documents you feed them.
What are the security risks of RAG?
Retrieval puts outside text into the model's context, and that text is not always trustworthy.
The OWASP Top 10 for LLM applications includes vector and embedding weaknesses (LLM08:2025). Its listed risks include unauthorized access and data leakage through embeddings, cross-context leaks in shared vector databases, embedding inversion attacks and data poisoning. Its recommended mitigations include permission-aware vector stores, validating and authenticating data sources, and keeping "detailed immutable logs of retrieval activities."
The other big risk is indirect prompt injection: a retrieved document that contains instructions aimed at the model. A coding agent reading a malicious README faces the same problem. Our guide to prompt injection in coding agents covers defenses.
Where do prompts that use RAG live?
A RAG app usually ends up with a handful of carefully tuned prompts: the answer template, the citation rules, the "say you don't know" instruction. Kipt stores prompts like these as Markdown in a private GitHub repo you own, with {{name:default}} blanks for the question or document slots, so the versions you have tested stay easy to find.
What should you read next?
Retrieved text is measured and billed in tokens, so read LLM tokens explained next. The instructions that tell a model how to use retrieved documents usually live in the system prompt. And if your answers vary more than you would like, LLM temperature explains what you can and cannot control.