Key takeaways
- The canonical 2026 sequence is prompt → RAG → fine-tuning → distillation. You move up a rung only when the previous one is genuinely exhausted, not when it gets boring.
- RAG is the right choice for roughly 80% of enterprise LLM applications: you can change the data without retraining, attribute answers to a document, and swap base models freely.
- The rule that settles 90% of debates: if the model doesn't know something, that's a RAG problem. If the model knows it but says it badly (tone, format, structure), that's a fine-tuning problem.
- Full fine-tuning is almost never the right call for a product team. LoRA and QLoRA train a small adapter: cheap, fast and reversible.
Context engineering, RAG and fine-tuning are three different ways of adapting an LLM to your company. Context engineering shapes what enters the prompt; RAG (retrieval-augmented generation) injects your knowledge at query time; fine-tuning modifies the model's weights. The decision comes down to one question: if the problem is knowledge, it's RAG; if it's behaviour, it's fine-tuning.
The three techniques, defined without ambiguity
Context engineering. The deliberate design of everything that enters the model's context window: system prompt, few-shot examples, reasoning scaffolding, constrained output schemas (JSON schema), information ordering and compression. It touches neither the model nor the data; it changes what the model sees. In 2026 it is far more powerful than most people assume, and it is the layer that makes RAG and fine-tuning actually work.
RAG (Retrieval-Augmented Generation). Given a question, the system first retrieves the relevant chunks from your documents or databases and hands them to the model as context. The model answers over your real data, with the option to cite the source. Knowledge lives in your infrastructure, not in the model.
Fine-tuning. You modify the model's weights by training it on your own examples of input and desired output. The model learns a behaviour: a tone, a format, a taxonomy, a reasoning style. It does not reliably learn updatable facts.
What fine-tuning is NOT: it is not "loading your documents into the model". That is the most expensive misunderstanding we see. Training a model on your procedures manual does not guarantee it will recite it correctly, and does guarantee it goes stale the moment the manual changes.
A useful analogy: context engineering is explaining the task properly to a professional. RAG is giving them access to the library and the case files. Fine-tuning is sending them on a specialist course so they work differently. Nobody sends someone to do a master's degree so they can memorise the company phone directory.
The comparison that settles the decision
| Criterion | Context engineering | RAG | Fine-tuning (LoRA) |
|---|---|---|---|
| Solves | Format, instructions, reasoning | Current, attributable knowledge | Tone, style, consistent structure |
| Updating information | Rewrite the prompt | Update documents (instant) | Requires retraining |
| Citing the source | No | Yes, natively | No |
| Switching base model | Trivial | Trivial | Adapter must be retrained |
| Time to first result | Hours | 2-6 weeks | 4-10 weeks (data included) |
| Needs labelled data | No | No | Yes, and it's the bottleneck |
| Main risk | Fragile, unmanageable prompt | Poor retrieval | Overfitting and staleness |
| Reversible | Yes | Yes | Yes with LoRA, no with full FT |
Read the table left to right: each column costs significantly more than the previous one and solves a different problem, not a "bigger" one. Jumping to fine-tuning because RAG underperformed usually means RAG was built badly, not that training was needed.
The decision tree we use
Four questions, in this order:
- Does the model have the information it needs? If not → RAG. No amount of fine-tuning makes a model reliably know this week's orders in your ERP.
- Does the information change over time? If yes → RAG, no debate. A model fine-tuned on March data still believes it's March.
- Do you need to cite the source of the answer? If yes → RAG. Attribution is an architectural property of retrieval, not something you can train.
- Does the model have the information but present it badly and systematically? If it's tone, format, length, structure or a proprietary taxonomy with hundreds of categories → then yes, fine-tuning with LoRA.
And one question before those four: have you exhausted context engineering? In our experience, between a third and half of the projects that arrive asking for "a model trained on our data" are solved with a well-built system prompt, representative few-shot examples and schema-constrained output. It is the cheapest intervention and the one almost nobody exhausts before escalating.
Key market data
- The canonical sequence consolidated in 2026 is prompt → RAG → fine-tune → distil, and RAG is the correct first choice for roughly 80% of enterprise LLM applications, because it lets you change data without retraining, attribute answers and swap base models freely (Winder.ai, 2026 decision framework).
- Full fine-tuning is almost never the right call for a product team in 2026. LoRA and QLoRA train a tiny adapter instead of the whole model: cheap, fast and reversible.
- 46% of organisations cite integration with existing systems as their primary challenge, ahead of model or technique choice. The bottleneck is rarely where the debate happens.
Our view, bluntly: most fine-tuning conversations in mid-market companies are badly framed RAG conversations. And a significant share of RAG projects that "don't work" have a retrieval quality problem, not a model problem.
The hybrid pattern that wins in production
The "RAG or fine-tuning" debate is a false one in mature systems. The pattern that works best combines all three layers:
- Light fine-tuning (LoRA) on a small model to lock in output format, corporate tone or a proprietary taxonomy. The model learns how to answer.
- RAG on top so it answers with the company's current data. The model gets what to answer.
- Context engineering to orchestrate both: what gets retrieved, in what order it enters, with what instructions and what output schema.
The result is a smaller, cheaper model than a large generalist, with the format consistency of a fine-tuned model and the data freshness of RAG. It is also the natural route towards models deployed on-premise when data sovereignty requirements apply.
Common use cases
Case 1: Classification with a proprietary 300-category taxonomy.
- Problem: classifying inbound tickets against an internal taxonomy no generalist model knows. The prompt with the full list was huge, slow and unreliable.
- Solution: LoRA fine-tuning on a small model using the history of already-classified tickets.
- Stack: open-weight model + LoRA + historical dataset.
- Result: consistent classification with short prompts and far lower latency. This is the textbook case where fine-tuning wins.
Case 2: Assistant over technical documentation that changes monthly.
- Problem: giving correct answers about manuals and procedures that are continuously updated.
- Solution: RAG with a vector store, version metadata and mandatory source citation.
- Stack: vector store + reranking + generalist model + output schema with citations.
- Result: the answer updates when the document updates, without touching the model. With fine-tuning you would retrain monthly.
Case 3: Report writing in a strict corporate format.
- Problem: the model has the information (it comes from RAG) but output formatting is inconsistent between runs.
- Solution: first, constrained output schema and few-shot examples. Only when that was exhausted, LoRA on 400 real reports.
- Stack: RAG + structured output + style LoRA.
- Result: 80% of the formatting problem was solved by context engineering; fine-tuning closed the rest.
How to decide, step by step
- Write the success criterion before choosing a technique. "Answers well" is not a criterion. "Gets the category right on 92% of last quarter's tickets" is.
- Build an evaluation dataset of 30-50 real cases. Without it you cannot compare techniques: you can only have opinions about which one you like.
- Exhaust context engineering. Explicit system prompt, 3-8 representative few-shot examples, JSON schema-constrained output. Measure against the dataset.
- If the problem is knowledge, build RAG and measure retrieval separately. Context recall and context precision before you look at final answer quality. If retrieval fails, no model can save you.
- Only if a behaviour problem persists, consider LoRA. You will need several hundred to a few thousand quality examples. Assembling that data is 70% of the project's real effort.
- Evaluate the hybrid. A small fine-tuned model + RAG usually beats a large generalist on cost and latency at equivalent quality.
- Document the decision and its reversal criterion. What would have to happen to go back. AI projects that skip this end up carrying architectures nobody defends any more.
Common mistakes (and how to avoid them)
Mistake: fine-tuning to "load our documents into the model". → Reality: fine-tuning teaches behaviour, not queryable facts. For knowledge, use RAG. This misunderstanding accounts for most of the failed AI projects we review.
Mistake: jumping to fine-tuning because RAG "doesn't work". → Reality: in almost every case, RAG fails at retrieval: badly sized chunking, no reranking, poor metadata. Measure retrieval before blaming the model.
Mistake: full fine-tuning instead of LoRA. → Reality: training all weights is slow, expensive, irreversible and locks you to that model version. LoRA trains a small adapter, swaps in minutes and can be discarded without consequences.
Mistake: starting from the technique instead of the metric. → Reality: with no measurable success criterion, any result looks good in the demo and bad in production.
Mistake: underestimating the fine-tuning dataset. → Reality: the model is not the problem; getting clean, consistent, representative examples is. A dataset with contradictory criteria teaches the model to be inconsistent.
Mistake: fine-tuning a model and never revisiting it. → Reality: an adapter trained a year ago on two-year-old data encodes an old version of your business. It needs periodic review, like any asset.
Realistic timelines and effort
- Serious context engineering (prompt, examples, output schema, measurement): days, not weeks. Best result-to-effort ratio of the three.
- RAG in production: 2-6 weeks depending on the quality and state of your documentation. If documents are messy or duplicated, that is where the time goes, not in the technical part.
- Fine-tuning with LoRA: 4-10 weeks, of which training itself takes hours. The rest is building and cleaning the dataset.
- Where the real effort goes: into the data, always. For RAG, document quality. For fine-tuning, labelled dataset quality. The model is the easy part.
Metrics to track from day 1: accuracy on the evaluation dataset, context recall and context precision where RAG applies, format consistency, p95 latency and cost per task. Plus an honest comparison against the baseline: what a well-built prompt achieves with nothing else. If nobody has that number, no further investment can be justified.
Frequently asked questions
When should you fine-tune instead of using RAG?
When the model already has the information but presents it badly and systematically: wrong tone, inconsistent format, or a proprietary taxonomy with many categories that does not fit comfortably in the prompt. If the problem is that the model doesn't know something, or that something changes over time, the answer is RAG.
Can I fine-tune a model so it knows my company's data?
Not reliably. Fine-tuning teaches behaviour, not queryable and updatable facts. A model tuned on data from one specific month goes stale and cannot cite sources. For enterprise knowledge, the correct technique is RAG.
What is LoRA and why is it used instead of full fine-tuning?
LoRA (Low-Rank Adaptation) trains a small adapter instead of modifying all the model's weights. It is far faster and cheaper, it is reversible, and it lets you keep several specialised adapters over the same base model. In 2026, full fine-tuning rarely pays off for a product team.
What exactly is context engineering?
It is the deliberate design of everything that enters the context window: system prompt, few-shot examples, reasoning scaffolding, output schemas and information ordering. It is the cheapest of the three layers and the one that fully solves the most projects before RAG or training are needed.
Can RAG and fine-tuning be combined?
Yes, and it is the pattern that wins in production: a small model fine-tuned with LoRA to lock format and tone, wrapped in RAG so it answers with current data. It comes out cheaper and faster than a large generalist model at equivalent quality.
How many examples do I need for useful fine-tuning?
Between several hundred and a few thousand clean, consistent examples, depending on task complexity. Quantity matters less than coherence: a dataset with contradictory criteria between examples teaches the model to be inconsistent.
Does fine-tuning reduce hallucinations?
Not reliably, and sometimes it makes them worse: a fine-tuned model answers more confidently within its trained domain, even when it lacks the fact. Factual hallucinations are reduced with RAG and source citation, not with training.
Which is better if I have GDPR or data sovereignty requirements?
RAG with a model deployed in a private cloud or self-hosted is usually the cleanest route: personal data is retrieved at query time and never enters a training process. If you fine-tune on personal data, that data becomes embedded in the weights, with all the data-subject rights implications that follow.
Not sure whether your case needs RAG, fine-tuning or just a better prompt?
At Naxia we start from the metric, not the technique: we define the success criterion, build the evaluation dataset and measure what each layer achieves before investing in the next. In many projects, the right answer turns out to be the cheapest one.
If you want an honest recommendation for your specific case, including the possibility that you need to train nothing at all, talk to us. No commitment, no 40-slide decks.
Or explore our implementation process first.