Every enterprise AI programme reaches the same meeting. Someone has read that retrieval-augmented generation is cheaper. Someone else has read that fine-tuning is more accurate. Both are sometimes right, and the argument usually runs on vendor blog posts rather than on the properties of the actual problem.
Here is the framing we use, and it starts somewhere unfashionable: with what happens when the system is wrong.
Start with the failure mode you can tolerate
Retrieval fails visibly. When the right document is not in the index, a well-built retrieval system says it cannot find the answer, or it produces something with a citation you can check and discard. The error is inspectable. You can look at what was retrieved and see immediately why the answer was poor.
Fine-tuning fails invisibly. A fine-tuned model produces a fluent, confident answer in the correct house style that happens to be based on a policy you retired last quarter. There is nothing to inspect. The error is distributed across weights, and the only way to find it is to have tested for it.
For most enterprise use cases — policy questions, product support, document analysis, anything where a wrong answer has a cost — visible failure is worth a lot. That alone pushes the default toward retrieval.
Then ask who owns the data, and how often it moves
The second question is operational. If the knowledge changes weekly, retrieval is not just better, it is the only sane option: you re-index, and the change is live in minutes. Fine-tuning on weekly-changing content means a retraining cadence, a model version to track, an evaluation run per version, and a rollback story. That is a machine-learning operation, and someone has to staff it.
There is also a governance angle that gets missed. With retrieval, access control can be enforced at query time: filter the index by the user's permissions and the model never sees what the user cannot see. With a fine-tuned model, whatever was in the training data is in the weights, for everyone. If your corpus has any per-role or per-tenant restriction, that fact usually settles the argument on its own.
Where fine-tuning genuinely wins
This is not an argument against fine-tuning. It wins clearly in three situations, and we have used it in all three.
Form, not facts
When you need consistent output structure, tone, or a domain vocabulary the base model handles poorly, fine-tuning is the right tool. Extracting a fixed schema from clinical notes, producing text in a specialised register, classifying into a taxonomy with 200 labels: these are shape problems, not knowledge problems. A few thousand examples move the needle more than any prompt.
Latency and unit cost at volume
A fine-tuned small model can match a large general model on a narrow task at a fraction of the cost and latency. When you are running millions of requests on one well-defined task, that difference stops being an optimisation and becomes the business case. We have replaced a frontier-model call with a fine-tuned small model and cut the per-request cost by more than an order of magnitude with no measurable quality loss — but only because there was an evaluation set to prove the second half of that sentence.
Behaviour that cannot be prompted reliably
Long, brittle prompts full of edge-case instructions are a smell. When the instruction list grows past what fits comfortably in a reviewer's head, and the model still drops rules under load, that behaviour is a candidate for training rather than prompting.
The answer is usually both, in a specific order
The mature architecture is retrieval for knowledge and, if warranted, a fine-tune for form. Retrieval supplies the facts and the citation; the tuned model reliably renders them into the structure you need. Build the retrieval layer first, measure it, and only then decide whether the residual errors are shape errors that a fine-tune would fix.
Build in this order:
- A graded evaluation set. Nothing below this line is decidable without it.
- A strong retrieval baseline: structure-aware chunking, hybrid search, metadata filters, re-ranking.
- Prompt and context engineering measured against the graded set, not against impressions.
- Only if the remaining errors are systematically about form: fine-tune.
What people actually get wrong
In practice, most projects that conclude "retrieval does not work for us" have not built a serious retrieval system. They have built a naive one: fixed-size chunks that cut tables in half, dense embeddings only, no filters, no re-ranker, and one nearest-neighbour query per question.
| Symptom | Usual cause | Fix before considering a fine-tune |
|---|---|---|
| Answers miss content that is definitely in the corpus | Fixed-size chunking splitting the relevant passage | Chunk on document structure; overlap; include heading context in the chunk |
| Exact identifiers, codes or part numbers are never found | Dense-only retrieval | Hybrid: add BM25 or another lexical channel and fuse the rankings |
| Right document, wrong section surfaced | No re-ranking | Retrieve broadly, then re-rank the top 50 with a cross-encoder |
| Stale or out-of-scope answers | No metadata filtering | Filter by effective date, region, product line, permission at query time |
| Confident answers with no support | No grounding requirement | Require span citations; validate that cited text exists; refuse when it does not |
Work through that table before spending anything on training runs. In our experience, four out of five "we need to fine-tune" conclusions dissolve somewhere in the middle of it.
The one-paragraph version for your steering committee
Use retrieval when the answer depends on knowledge that changes, is access-controlled, or must be citable. Use fine-tuning when the output's form, cost or latency is the problem and the underlying knowledge is stable. Decide with an evaluation set rather than a preference, and expect to end up with both — retrieval for what is true, tuning for how it is said.
Written by the Bytescope Tech engineering team from live engagement work. If you want to argue with any of it, or you have a case where the opposite held, tell us — those are our favourite emails.