Fine-tuning vs RAG: which one does your LLM app need?

RAG and fine-tuning solve different problems. RAG (retrieval augmented generation) looks up relevant information at request time and adds it to the prompt, so the model can answer from facts it was never trained on. Fine-tuning further trains a model on your examples, so it changes how the model behaves: its format, style or skill at a narrow task.

A quick rule: if the model is missing knowledge, use RAG. If it has the knowledge but behaves inconsistently, consider fine-tuning. And before either, try better prompts. OpenAI's accuracy guide puts it plainly: "Prompt engineering is typically the best place to start."

This guide compares the two, shows when each fits, and covers which vendors still offer fine-tuning in October 2026, which has changed a lot this year.

What is the difference between fine-tuning and RAG?

Both make a model more useful for your case. They act at different points:

OpenAI frames it as two axes. You optimize context "when 1) the model lacks contextual knowledge," and you optimize the model "when 1) the model is producing inconsistent results with incorrect formatting." The first improves response accuracy; the second improves consistency of behavior.

What is RAG?

Anthropic's glossary defines RAG as "a technique that combines information retrieval with language model generation to improve the accuracy and relevance of the generated text, and to better ground the model's response in evidence."

The flow is simple. Your app takes a question, searches a knowledge base for relevant passages, and sends those passages to the model along with the question. The search is often semantic search over embeddings, sometimes combined with keyword search. The model itself is untouched.

The glossary also states the main limit: "the effectiveness of RAG depends on the quality and relevance of the external knowledge base and the knowledge that is retrieved at runtime." Bad retrieval means bad answers. Our guide to what RAG is covers the pipeline step by step.

What is fine-tuning?

The same glossary says: "Fine-tuning is the process of further training a pretrained language model using additional data. This causes the model to start representing and mimicking the patterns and characteristics of the fine-tuning dataset."

Vendors offer several methods. OpenAI's model optimization page lists four:

• Supervised fine-tuning: train on examples of correct responses.
• Vision fine-tuning: supervised training with image inputs.
• Direct preference optimization (DPO): train on pairs of better and worse responses.
• Reinforcement fine-tuning (RFT): a grader scores responses and the model learns from higher-scored ones; for reasoning models only.

Google's tuning docs list supervised fine-tuning, reinforcement learning fine-tuning and preference tuning for Gemini models.

When should you use RAG?

RAG is the default choice when the answer depends on information the model does not have. Anthropic's glossary says it "can be particularly useful for tasks that require up-to-date information, domain-specific knowledge, or explicit citation of sources." Typical cases:

• Answering questions over internal docs, tickets, wikis or code.
• Anything that changes often: prices, policies, inventory, release notes.
• Answers that must show where they came from.
• Per-user data that you could never bake into one shared model.

RAG also keeps you free to switch models, since the knowledge lives in your documents, not in weights.

One caveat before you build a pipeline: you may not need retrieval at all. Anthropic's Contextual Retrieval post notes that if your knowledge base is smaller than 200,000 tokens, about 500 pages, you can often include all of it in the prompt. See context engineering for how to manage a large context well.

When should you use fine-tuning?

Fine-tuning fits when you can show the model what good output looks like, but prompting alone does not get it there reliably. Google's supervised tuning docs say: "Supervised fine-tuning is a good option when you have a well-defined task with available labeled data." The same page suggests tuning when:

• You need specific structures or formats for output.
• You want specific behaviors, such as when to be terse or verbose.
• A behavior is hard to put into words in a prompt.
• "You want to reduce the context length by removing the few-shot examples."

OpenAI's page adds two efficiency reasons: shorter prompts once the examples are trained in, and training a smaller, cheaper model to do a task that would otherwise need a larger one.

Fine-tuning needs a real dataset, an evaluation set to prove the tuned model is better, and retraining whenever your requirements change. Google also notes that you pay inference costs for the tuned model after paying for training tokens.

Should you try prompting before either?

Yes, and OpenAI's and Google's docs both point that way. OpenAI's accuracy guide advises: "squeeze as much accuracy out of basic methods as you can before reaching for more complex RAG or fine-tuning." Its optimization page adds: "The prompt engineering process may be all you need in order to get great results for your use case."

Google's tuning page makes the same point from the other side: foundation models "work well when the expected output or task can be clearly and concisely defined in a prompt and the prompt consistently produces the expected output."

Both OpenAI and Anthropic recommend building evaluations first. Anthropic's prompt engineering overview assumes you already have clear success criteria and a way to test against them. Without evals, you cannot tell whether RAG or tuning actually helped. A clear system prompt with a few worked examples fixes many formatting problems that look like they need training.

Can you combine fine-tuning and RAG?

Yes. OpenAI's guide says the methods are "additive, not exclusive" and that "they stack, and some use cases will require you to use them together."

A combined setup usually looks like this: retrieval supplies current facts and source passages for each request, and a tuned model turns them into output in your exact format. Build them in order. Get retrieval working and measured first, because a tuned model cannot fix passages that were never retrieved.

Which vendors offer fine-tuning in October 2026?

This changed a lot in 2025 and 2026, so check before you plan around tuning. As of October 2026, from each vendor's docs:

OpenAI notes that existing fine-tuned models "will remain available for inference until their base models are deprecated." Its deprecations page gives the dates: since July 2, 2026, organizations that have not run inference on a fine-tuned model in the past 60 days cannot create jobs, and from January 6, 2027, active existing customers can no longer create new fine-tuning jobs. Open-weight models can still be fine-tuned on your own hardware or through cloud platforms, which is outside the scope of this guide.

The practical effect: for most teams building on hosted frontier models today, prompting plus retrieval is the main lever, and tuning is a specialist option on a few platforms.

Where do the prompts for either approach live?

Both approaches depend on prompts you refine over time: the instructions that wrap retrieved passages, and the examples you would use as a starting dataset for tuning. Kipt keeps prompts and Agent Skills as Markdown in a private GitHub repo you own, with tags and fill-in blanks, so the version that passed your evals is the one your team reuses.

What should you read next?

Start with what is RAG and what are embeddings if retrieval fits your problem. If your concern is wrong answers, read AI hallucinations for causes and the mitigations vendors document.