RAG versus fine-tuning: pick the boring one

The RAG versus fine-tuning debate generates more heat than light. After two dozen LLM projects, our view is unromantic: for 80% of use cases, RAG plus strong prompting is enough, fine-tuning is overengineering, and 'neither' is more often the right answer than the question allows. Here is the actual decision tree.

The cost curve, in real numbers

Fine-tuning a 7B model on a domain-specific dataset costs between 200 and 2,000 dollars in compute, plus weeks of evaluation. RAG infrastructure costs the same in development but scales linearly with usage. The crossover point where fine-tuning is cheaper than RAG is much further out than most teams think, because RAG's marginal cost is small and fine-tuning's eval cost is large.

When RAG is a band-aid

RAG fails when the model needs to know something that is not in any document. If your task requires the model to follow a long, complex reasoning chain that depends on facts not in the corpus, retrieval will not help. We have seen teams spend six months building elaborate retrieval pipelines to mask what was a fine-tuning problem all along.

When fine-tuning is overengineering

Fine-tuning fails when the training data is not representative of production traffic. If you fine-tune on 5,000 hand-curated examples and your production traffic has 50,000 distinct query types, you have built a system that is excellent at your training set and mediocre everywhere else. The eval suite catches this; most teams do not have a representative eval suite.

The eval methodology that decides

We always start with RAG and a strong eval suite. We measure end-to-end accuracy. If we are below the business threshold, we look at the failure cases. If the failures are 'model could not find the right context,' we improve retrieval. If they are 'model found the right context and still gave a wrong answer,' we fine-tune. The eval suite tells us; opinion does not.

The decision tree, distilled

Knowledge that changes frequently: RAG. Style and output format consistency: fine-tune. Reasoning patterns specific to your domain: fine-tune, but only after RAG has plateaued. Generic tasks with general knowledge: prompting. Anything where you cannot write a representative eval suite: do not ship an LLM. Anything where a rules engine or a simpler model works: do not use an LLM at all.