RAG supplies knowledge. Fine-tuning shapes behaviour.
Use RAG when the model needs access to facts it was not trained on — your documents, your product data, anything that changes. Use fine-tuning when the model already knows enough but behaves wrong — inconsistent format, wrong tone, ignoring your task structure.
If you are trying to make an LLM answer questions about your company's documents, the answer is RAG. Fine-tuning to inject facts is the single most common and most expensive mistake in LLM projects.
What each one actually does
RAG — retrieval-augmented generation
At query time, you search a corpus for passages relevant to the user's question, then paste those passages into the prompt alongside the question. The model's weights never change. It is, mechanically, automated context-stuffing with a search step in front.
# conceptually, every RAG query is this
chunks = vector_store.similarity_search(question, k=5)
context = "\n\n".join(c.text for c in chunks)
answer = llm.invoke(f"Context:\n{context}\n\nQuestion: {question}")Everything else in RAG — chunking strategy, embedding choice, hybrid search, reranking, query rewriting — exists to make line one return the right five chunks. That is where essentially all the engineering difficulty lives.
Fine-tuning
You take a base model and continue training it on your own input–output examples, adjusting its weights. The result is a model that has internalised a pattern of behaviour.
# fine-tuning data is examples of behaviour, not facts
{"messages": [
{"role": "user", "content": "Summarise this support ticket."},
{"role": "assistant", "content": "SEVERITY: P2\nCOMPONENT: billing\n..."}
]}Modern practice is parameter-efficient fine-tuning — LoRA and similar — which trains a small set of additional weights rather than the whole model. It is dramatically cheaper and is what most teams actually use.
The decision table
| Your problem | Use | Why |
|---|---|---|
| Model doesn't know about our internal docs | RAG | Knowledge problem. Retrieval puts the facts in front of it. |
| Answers must cite their source | RAG | Only RAG can point at the passage it used. |
| Information changes weekly | RAG | Re-index in seconds; retraining takes days. |
| Different users may see different data | RAG | Access control at retrieval time. Weights cannot be permissioned. |
| Output format is inconsistent | Fine-tune | Behaviour problem. Examples teach the shape. |
| Needs a specific tone or house style | Fine-tune | Style is learned far better from examples than from instructions. |
| Prompt has grown to 3,000 tokens of instructions | Fine-tune | Bake the instructions into weights; cut per-query cost. |
| Want a small model to match a large one on one task | Fine-tune | Narrow specialisation is exactly what fine-tuning is for. |
| Domain jargon is misunderstood | Both | Fine-tune for vocabulary, RAG for the facts expressed in it. |
The expensive mistake
Fine-tuning does not reliably store facts. Teams routinely fine-tune a model on a corpus of company documents expecting it to answer questions about them. What they get is a model that has learned the style of those documents and will generate confident, fluent, plausible statements that are wrong — and with no source to check them against.
The reason is structural. Fine-tuning adjusts weights toward producing text that looks like the training set. It has no mechanism for storing a specific fact in a retrievable, verifiable form. A price, a policy clause or a date may be absorbed, partially absorbed, or blended with something similar seen during pre-training — and you cannot tell which from the output.
RAG has the opposite property. The fact is in the prompt, verbatim, and you can show the user which document it came from. For anything where being wrong matters, that auditability is the whole game.
A useful test before choosing: if the answer changed tomorrow, how would the system find out? If the honest answer is "we would have to retrain," you want RAG.
Cost, latency and update speed
| RAG | Fine-tuning | |
|---|---|---|
| Setup cost | Low — embedding + vector store | Moderate — dataset prep dominates |
| Per-query cost | Higher — retrieved context lengthens every prompt | Lower — shorter prompts, possibly smaller model |
| Latency | Adds a retrieval hop (~50–300ms) | Often faster — less to process |
| Update speed | Seconds — re-index one document | Days — prepare data, retrain, re-evaluate |
| Attribution | Built in | Impossible |
| Access control | Filter at retrieval | Not possible |
| Main failure mode | Retrieved the wrong chunks | Confidently invented a fact |
The per-query cost difference is the one that surprises people at scale. A RAG system retrieving five chunks of 500 tokens adds 2,500 input tokens to every single request. At a million queries a month that is a substantial line item — and it is the point at which fine-tuning a smaller model starts to look attractive for the stable parts of the workload.
Using both together
In production the two are complements, not alternatives. The common pattern:
- Fine-tune a small model on a few hundred examples so it reliably produces your output structure and uses your domain vocabulary correctly.
- Wrap it in RAG so every query is grounded in current, retrievable documents.
You end up with consistent behaviour, current knowledge, source attribution, and often a lower bill than a frontier model with a 3,000-token system prompt. The trade is more moving parts to maintain and evaluate.
Try things in this order
- Better prompting. Genuinely — a large share of problems attributed to model capability are prompt problems. Few-shot examples and structured output schemas are free and take an afternoon.
- RAG. If the gap is knowledge, this is your answer, and it is the cheapest real fix.
- Improve the RAG. Most disappointing RAG systems are not RAG failures; they are chunking and retrieval failures. Fix retrieval before concluding the approach is wrong.
- Fine-tune. Only once you know the remaining gap is behavioural, and only with a real evaluation set to prove it improved things.
Skipping to step four is common and expensive. It also tends to produce a system nobody can debug, because a fine-tuned model gives you no visibility into why it produced a particular answer.
Both techniques, with the evaluation practices that make them measurable, are covered in our Generative AI course — and the retrieval engineering that determines whether RAG works at all is treated at length, because that is where the real difficulty sits.
Frequently asked questions
What is the difference between RAG and fine-tuning?
RAG retrieves relevant documents at query time and puts them into the model's prompt, so the model reasons over information it was never trained on. Fine-tuning adjusts the model's own weights on example data, changing how it behaves. The short version: RAG supplies knowledge, fine-tuning shapes behaviour.
Should I use RAG or fine-tuning to add my company's data to an LLM?
Use RAG. This is the most common mistake in LLM projects — teams fine-tune to inject facts, then find the model confidently invents plausible variations of them. Fine-tuning does not store facts reliably or citably. If your goal is for the model to answer questions about documents, RAG is almost always the correct choice.
Is fine-tuning more accurate than RAG?
Not for factual recall. Fine-tuning is more accurate for behaviour — consistent output format, domain tone, following a specific task structure. For questions whose answers live in a document, RAG is both more accurate and verifiable, because you can show which passage the answer came from.
Can you use RAG and fine-tuning together?
Yes, and in production systems it is common. Fine-tune a smaller model to reliably follow your output format and domain conventions, then use RAG to feed it current facts. You get consistent behaviour and up-to-date knowledge, often at lower cost than a large model with a long prompt.
Which is cheaper, RAG or fine-tuning?
RAG has near-zero setup cost but higher per-query cost, since retrieved context makes every prompt longer. Fine-tuning has meaningful upfront training cost but can cut per-query cost by shortening prompts and allowing a smaller model. At low volume RAG is cheaper; at very high volume with stable requirements, fine-tuning can win.
How long does fine-tuning take compared to setting up RAG?
A working RAG prototype takes hours; a production-quality one takes weeks, with most of the effort in chunking, retrieval quality and evaluation. Fine-tuning takes days to weeks, dominated by dataset preparation rather than training. Updating differs even more sharply: RAG updates in seconds by re-indexing a document, fine-tuning requires a retraining cycle.