"Should we fine-tune a model or build a RAG system?"
This is the first question every business asks when they want AI to work with their internal data. And most of them get the answer wrong — usually because someone on their team read a blog post about fine-tuning GPT-3 in 2023 and assumed that's still how things work.
I build both for clients. Here's the honest breakdown.
The One-Line Difference
RAG (Retrieval-Augmented Generation) gives the AI your data at query time. The model stays the same — you just feed it relevant context before it answers.
Fine-tuning changes the model itself. You train it on your data so it learns new patterns, terminology, or behaviors.
Think of it this way:
- RAG = handing an expert a reference manual before asking them a question
- Fine-tuning = sending that expert back to school for specialized training
The expert is equally smart either way. The question is whether you need them to know something new or have access to something new.
How RAG Works (30-Second Version)
- Your documents are chunked and converted into vector embeddings
- Embeddings are stored in a vector database (we use pgvector)
- When a user asks a question, the system finds the most relevant chunks
- Those chunks + the question are sent to the LLM as context
- The LLM generates an answer grounded in your actual documents
The model never changes. Your data is never baked into the weights. The LLM sees your documents only at the moment it needs them. See the full interactive pipeline →
How Fine-Tuning Works (30-Second Version)
- You prepare training data — usually thousands of input/output pairs
- You run a training job that adjusts the model's internal weights
- The result is a new version of the model that "knows" your patterns
- You deploy and serve this custom model (or use a hosted fine-tune)
The model itself changes. It costs more upfront. It takes longer. And here's the part people miss: it can still hallucinate. Fine-tuning doesn't give the model a source of truth — it shifts the probability distribution of what it generates. It's more likely to use your terminology and style, but it's not grounded the way RAG is.
The Real Comparison
Cost
- RAG: $15K–$50K to build. ~$0.002 per query to run. Embedding model can run locally (free) or via API (~$0.0001/query). Vector storage is PostgreSQL — you probably already pay for it.
- Fine-tuning: $5K–$100K+ to build (depending on model size, data prep, iteration cycles). Training runs cost $50–$500+ each. You'll run many. Serving a fine-tuned model costs 2-4x more per query than the base model. If self-hosted: GPU infrastructure at $2–$8/hr.
Data Freshness
- RAG: Add a new document, it's searchable in seconds. Delete a document, it's gone. Your knowledge base is always current.
- Fine-tuning: New data requires retraining. A training run takes hours to days. You'll need to version your models, manage rollbacks, and schedule retraining cycles. Most teams retrain monthly at best.
Hallucination
- RAG: Answers cite specific source documents. If the answer isn't in the retrieved chunks, a well-built system says "I don't have that information." You can verify every claim.
- Fine-tuning: The model generates from learned patterns. It doesn't "know" where information came from. It can confidently produce answers that blend your training data with its general knowledge — and you can't tell which is which.
Accuracy
- RAG: As good as your retrieval. If the right chunks are found, answers are excellent. If retrieval misses (bad chunking, poor embeddings), answers suffer. Retrieval quality is tunable and measurable.
- Fine-tuning: As good as your training data. Garbage in, garbage out — but the garbage comes out sounding confident. Harder to debug because failures are inside the model's weights, not in a retrieval step you can inspect.
Complexity
- RAG: Needs a vector database, an embedding model, and a retrieval pipeline. Total infrastructure: a PostgreSQL extension + a Python service. Fits in a single Docker Compose file.
- Fine-tuning: Needs training data curation (the hardest part), GPU access, training pipeline, model versioning, evaluation framework, and either hosted fine-tune management or self-hosted GPU serving infrastructure.
When RAG Wins (Most of the Time)
Use RAG when:
- You need answers from specific documents. SOPs, contracts, manuals, policies, knowledge bases, regulatory filings, research papers. If you can point at a document and say "the answer is in there," that's RAG.
- Your data changes. New policies, updated procedures, fresh reports. If your knowledge base isn't static, RAG handles it without retraining anything.
- You need citations. "According to SOP-002, section 3.1..." — regulated industries, legal, compliance, medical, anyone who can't just trust the AI's word.
- You're working with sensitive data. RAG keeps your data in your database. The model never learns it permanently. Delete the document, the knowledge is gone. Try doing that with a fine-tuned model.
- You have less than 10M documents. That's most businesses. pgvector handles millions of vectors without breaking a sweat.
When Fine-Tuning Wins (Less Often Than You Think)
Use fine-tuning when:
- You need to change the model's behavior, not its knowledge. Specific output format, tone, writing style, or reasoning patterns. Example: training a model to always respond in your brand voice, or to structure legal analyses in a specific framework.
- You need specialized domain language. If the base model genuinely doesn't understand your terminology (rare with modern LLMs, but happens in niche scientific/medical domains).
- You need to compress knowledge for latency. If you need sub-100ms responses and can't afford the retrieval step. This is an edge case — RAG retrieval adds ~50-200ms, which is fine for 99% of applications.
- You're building a product feature, not a knowledge tool. Code completion, structured data extraction, classification — tasks where the model needs to do something differently, not know something new.
The Hybrid Approach
Here's what experienced teams actually do: RAG first, fine-tune later if needed.
Start with RAG. Get your data indexed. Build the retrieval pipeline. See where it falls short. In most cases, the answer is "nowhere important" and you ship it.
If you find the model consistently misformats outputs, struggles with domain terminology, or needs a specific reasoning style — then consider fine-tuning. But fine-tune for behavior, and keep RAG for knowledge. They're complementary, not competing.
In practice, we've built over a dozen AI systems for businesses. Exactly one needed fine-tuning (a medical device classification system with very specific regulatory output requirements). Every other one was RAG, or RAG + prompt engineering.
The Decision Framework
Ask yourself three questions:
- "Do I need the AI to know my data, or act differently?"
Know your data → RAG. Act differently → Fine-tuning. - "Does my data change more than once a quarter?"
Yes → RAG. Fine-tuning can't keep up. - "Do I need to cite sources?"
Yes → RAG. Fine-tuned models can't tell you where they learned something.
If you answered "RAG" to any of those, start with RAG. If you answered "fine-tuning" to all three, you might actually need fine-tuning — and you should budget accordingly.
Real Numbers
For a typical business knowledge base (500-5,000 documents):
- RAG build cost: $15K–$40K (4-6 weeks)
- RAG running cost: ~$50-200/month (API + infrastructure)
- Fine-tuning build cost: $30K–$80K (6-12 weeks, including data prep and iteration)
- Fine-tuning running cost: $200–$2,000/month (GPU serving or premium API pricing)
- Fine-tuning retraining: $500–$5,000 per cycle, recommended monthly
RAG is cheaper to build, cheaper to run, easier to update, and provides citations. For the vast majority of business use cases, it's the right call.
See RAG in Action
We built a live RAG demo with 9 real documents — data center operations SOPs, incident reports, real estate feasibility analysis. Ask it anything in plain English and watch it retrieve relevant chunks, cite sources, and generate grounded answers.
No fine-tuning involved. Just PostgreSQL, pgvector, an embedding model, and Claude.
Want to understand the full pipeline? Read our interactive guide to how RAG works, or get in touch to discuss what it would look like for your data.