DEV Community: TankDev-Tech

RAG vs. Fine-Tuning: How to Choose the Right Technique for Enterprise LLM Systems

Murat Onur Kaderoglu — Wed, 30 Sep 2026 11:00:00 +0000

RAG vs. Fine-Tuning: How to Choose the Right Technique for Enterprise LLM Systems

Choose between RAG and fine-tuning through knowledge freshness, behaviour, data security, cost, and evaluation.

RAG and fine-tuning are often presented as competing answers to one question: should we give company documents to the model, or train the model on company data? That framing is incomplete. RAG changes which knowledge is available at answer time. Fine-tuning changes how a model behaves on a task. A sound production system chooses where rules, RAG, fine-tuning, and human review are actually needed.

This is TankDev's decision framework: retrieve the source first when the work requires a current contract, price, procedure, or customer record; evaluate fine-tuning experimentally when the problem is a repeatable response format, classification boundary, or behaviour; and make deterministic services—not an LLM—the decision maker for exact calculations, authorization, and irreversible actions.

1. Separate the problem first: knowledge, behaviour, or rule

When a user asks, “What is the return window for this customer?” the problem is access to current knowledge. The policy may change tomorrow, and the evidence for a correct answer is the relevant policy revision. That is a RAG problem. “Classify an incoming email into one of six operations categories and return valid JSON only” is a consistency and output-contract problem; it may be a fine-tuning candidate. “A second approval is required above a 5,000 TRY refund” is a deterministic business rule and belongs in SQL, a workflow, or a rules service.

Need	First choice	Why	Evidence
Current policy, product, or contract	RAG	The source revision changes	Citation + document ID
Repeated classification or format	Prompt → fine-tuning experiment	Behaviour repeats	Held-out test set
Exact calculation, permission, limit	Deterministic code	An error is costly	Tests + audit log
Ambiguous high-risk case	Human-in-the-loop	Outcome is irreversible	Review record

2. What RAG is—and is not

Retrieval-augmented generation selects evidence from authorized sources for a user's question and supplies that evidence to the model context. A production path acquires sources, prepares text and metadata, retrieves candidates, reranks evidence, produces a cited answer, and records observability events. A vector database can be one component of this path; it is not a synonym for RAG.

Good RAG does not end at splitting a PDF into smaller pieces. It preserves revision, effective date, product scope, language, access control, heading hierarchy, and source link. Retrieval may combine lexical and semantic search, then use a reranker to narrow evidence. The model should answer from evidence or explicitly report that evidence is insufficient.

Here, actor_id is not merely a log field: retrieval must filter out customer or contract fragments that the actor cannot access. Adding “do not reveal private information” to a prompt afterward is not access control. If retrieval returns no sufficient evidence, the system should enter an insufficient_evidence state rather than generate a plausible answer.

3. What does fine-tuning change?

Fine-tuning moves a base model toward selected input-output examples for a specific task. Classification labels, structured output format, domain language, concise response style, or preferred tool selection can be suitable targets. A provider treats it as a separate operation with a training dataset and job; for example, the OpenAI API specifies JSONL training data for a fine-tuning job. OpenAI fine-tuning reference

Fine-tuning does not turn training facts into a reliable document store. When a policy or price changes, it is hard to identify every affected answer. A memorized-looking example is neither citable, current, nor access-controlled evidence. Knowledge-base questions can therefore require RAG alongside a fine-tuned model.

A training example represents the ideal work outcome and permitted output contract.
A validation set supports prompt, hyperparameter, or decision-threshold choices.
A test set stays untouched until final comparison; the same customer, template, or document family must not leak across splits.
Model, dataset, prompt, and evaluation versions belong in the same release record.

4. Concrete example: a dealer support assistant

A B2B support assistant can perform three different jobs. When a user asks for the current return condition, the system retrieves the authorized contract revision with RAG and displays the clause link. When a user submits free-form text, a model classifies it as returns, pricing, delivery, account, technical, or other. Before an action creates a return, a deterministic service validates the customer, order, time window, and authorization.

For classification, compare a baseline prompt against 600 labelled examples on a held-out test set. If failures are mostly caused by changing policy knowledge, fine-tuning is the wrong investment; repair retrieval, filters, or source quality. If evidence is present but category format or domain language stays inconsistent, a fine-tuning experiment has a meaningful hypothesis. This distinction prevents the costly reflex to train first.

5. How do you measure RAG quality?

A convincing answer is not proof that the system is good. RAG evaluation needs at least two layers: did retrieval bring back the correct evidence, and did generation provide a correct and sufficient answer using only that evidence? A wrong document can be summarized fluently. A correct document can be used to support the wrong clause. These failures need different owners and different fixes.

Measure	Question	Example signal
Recall@k	Is the correct source in the first k results?	Critical clause in top 5
Citation precision	Does the displayed source support the claim?	Human review
Grounded correctness	Does the answer follow evidence and the business rule?	Expected-answer rubric
Abstention	Does it stop when evidence is absent?	Review rather than answer
Latency / cost	Are time and spend acceptable?	p95 + cost per query

An evaluation set should include easy FAQs alongside name collisions, stale revisions, multi-document answers, permission boundaries, OCR errors, empty retrieval, and conflicting sources. Google Cloud's RAG guidance also emphasizes that retrieval quality is central and irrelevant retrieval can lead to off-topic or incorrect generation. Google Cloud RAG guide

6. Make fine-tuning a measured experiment

Fine-tuning is not a deployment decision; it is an experiment. State a measurable objective first: “the JSON schema is valid” or “priority classification reaches at least X accuracy under expert adjudication.” Compare a base model with a strong system prompt, a base model with RAG, and a fine-tuned version using blind evaluation on the same test set. More complexity does not make an approach more correct.

Classify failure examples: wrong knowledge, wrong retrieval, schema failure, wrong tool selection, missing context, safety violation, or evaluation ambiguity. Adding every failure to training can harm dataset quality. Examples should show ideal behaviour; personal data, access keys, stale conflicting rules, and unverified model output do not belong in training data.

7. Cost and latency are different accounting models

RAG cost includes source preparation and indexing plus query-time embedding, search, reranking, context tokens, and generation. When a document changes, only affected fragments can be processed again. Fine-tuning has dataset preparation, training-job, evaluation, and custom-model inference costs. The material cost, however, includes correction effort, operational delay, and the impact of a wrong answer—not the model invoice alone.

8. Security, access, and data lifecycle

Before a RAG system adds a document to context, it applies tenant, role, record-level, contract-scope, and effective-date filters. Source fragments, answer, and actor identity may be recorded for observability with data minimization. When a document is deleted or permission is revoked, remove it from the index and caches as well.

Fine-tuning data adds a second governance question: examine the provider's data retention, usage, and deletion terms. Governance is not solved by calling a file “fine-tuning data.” Define which data class may leave the boundary, how long it is retained, and how a model revision is retired.

9. A hybrid architecture is often the right result

Mature systems can use both methods: a fine-tuned or carefully prompted model structures the query and stabilizes response form; RAG brings current, authorized evidence; a rules service validates permissions, calculations, and side effects; human review closes uncertain or high-risk cases. This assigns each component the responsibility it can actually carry instead of trying to teach everything to a model.

10. A practical decision sequence

Write the success contract: what are the correct source, required fields, side effect, and deadline?
Establish a simple baseline: strong prompt, schema validation, rule checks, and a measured example set.
When knowledge changes, start with versioned sources and RAG; evaluate retrieval separately.
When a behaviour problem repeats, test the fine-tuning hypothesis on a fixed test set.
Do not execute a high-risk decision without deterministic validation or human approval.
Monitor versions, cost, false acceptance, source evidence, and rollback in production.

Frequently asked questions

Is RAG or fine-tuning more accurate?

Use RAG first for current, citable knowledge; fine-tuning can be evaluated for repeated behaviour, format, or classification. Deterministic code remains the right layer for exact business rules. The same held-out work scenarios should decide.

Is fine-tuning suitable for teaching company documents to a model?

It is usually not the first choice for frequently changing policies, prices, contracts, or customer records. Those require versioning, citation, deletion, and access control, which RAG supports more directly.

Does RAG eliminate hallucinations?

No. Retrieval can be wrong or incomplete, and a model can still go beyond evidence. Use source filters, reranking, citation checks, abstention when evidence is absent, and evaluation together.

Is a vector database enough for RAG?

No. Source quality, chunking, metadata, access filters, hybrid search, reranking, version management, and evaluation matter at least as much as the database choice.

Can RAG and fine-tuning be combined?

Yes. Fine-tuning or a strong prompt can improve behaviour and response form while RAG supplies current evidence. Permissions, calculations, and persistent side effects still need deterministic validation.

What should the first production release measure?

Measure business-outcome correctness, correct-source retrieval, citation support, schema validity, abstention without evidence, p95 latency, cost per query, and the rate of human review.

Author

Murat Onur Kaderoğlu

Algorithm and Architecture Developer

Published by TankDev · Original article: RAG vs. Fine-Tuning

Ending LLM bloat: How to integrate Jev and System One decision models into production architecture

Murat Onur Kaderoglu — Sun, 27 Sep 2026 19:59:37 +0000

Ending LLM bloat: How to integrate Jev and System One decision models into production architecture

Not every workflow needs generated text. A practical guide to turning Jev Choice, Score, and Noul decisions into a production layer with thresholds, auditability, and human review.

A meaningful share of AI automation cost comes from asking a large language model to make decisions that do not require generated text. Routing a ticket to billing, sales, or support; selecting a tool; or screening a request for risk does not inherently require a paragraph. Software needs a value it can execute, not prose for a person to read.

Jev is TypeSafe's System One decision model: an application sends state and typed questions, and receives a choice, score, or probability instead of generated text. The official documentation defines confidence for Choice and Score and a 0–1 probability for Noul. This does not make LLMs obsolete. It gives decisions and expression separate places in an architecture.

The problem: structured output is not automatically a structured decision

JSON Schema, function calling, and Pydantic can make an LLM response parseable. The model still interprets instructions, generates tokens autoregressively, and makes the application wait for that generation. A valid route: billing field does not establish that the route is correct, that uncertainty is handled, or that the call is economical for the workload.

Note: When a model result causes code to run, the design question is not merely ‘does it return JSON?’ It is ‘which incorrect decisions can this operation tolerate, and what happens when the system is uncertain?’

System 1 and System 2: a useful architecture metaphor

Kahneman's System 1 / System 2 distinction is useful here as a technical metaphor. A System 1 layer makes a fast, narrow judgment: which queue, whether risk exists, which tool to select. A System 2 layer reasons across context, explains alternatives, and writes text. A conventional LLM often serves the latter role; decision models such as Jev fit narrow, measurable questions in the former. This is a division of work, not a hierarchy of intelligence.

User request
     │
     ▼
[Validation + authorization] ── invalid ──► reject / explain
     │
     ▼
[Jev: route, risk, human_needed]  ← one state, parallel questions
     │
     ├─ confidence ≥ threshold ─► deterministic workflow / tool
     ├─ confidence < threshold ─► human-review queue
     └─ prose required ─────────► LLM (draft, summary, explanation)
                                      │
                                      ▼
                              audit log + metrics

Three decision primitives: Choice, Score, and Noul

Jev constrains the shape of a decision before the request is sent. Instead of leaving the answer open-ended, the application states which kind of value it will consume. Independent judgments can be collected as one decision package because multiple questions on the same state are evaluated in parallel.

Primitive	Question	Production example	Value consumed by code
Choice	Which option?	Which team owns this ticket?	`billing
Score	What level on a defined rubric?	How urgent is the incident?	probability-weighted score + confidence
Noul	Is this statement true?	Does this require human review?	{% raw %}`P(true) ∈ [0, 1]`

The three decision types documented by TypeSafe.

{
  "state": {"subject": "My card was charged twice", "channel": "web", "account_tier": "enterprise"},
  "questions": {
    "route": {"type": "choice", "instructions": "Route to the accountable team.", "criteria": {"billing": "payment, invoice, or collection", "support": "product issue", "sales": "purchase or quote"}},
    "urgency": {"type": "score", "instructions": "Assess operational urgency.", "criteria": ["low", "normal", "high", "critical"]},
    "human_needed": {"type": "noul", "instructions": "Is human review needed because of financial harm or account security?"}
  }
}

Here, route is not a label for a dashboard: it selects a queue, SLA, and access boundary. urgency can drive prioritization. human_needed can stop an automated reply. Keep questions atomic. ‘Route this safely, urgently, and correctly’ combines unrelated judgments and is difficult to evaluate.

Cost and latency: compare the right unit

A universal benchmark would be misleading because context length, caching, output length, region, and concurrency change the result. TypeSafe advertises Jev at $42 per billion input tokens, or about $0.042 per million input tokens. An LLM decision cost includes more than input pricing: system instructions, generated structured output, possible repair calls, and waiting time. Measure p50/p95 latency, tokens per call, fallback rate, and the cost of incorrect decisions with your own traffic.

Measure	Jev / System One decision layer	Structured-output LLM
Output	Typed choice, score, or probability	Generated JSON matching a schema
Execution	Narrow parallel questions over one state	Autoregressive generation with output tokens
Cost model	Input tokens plus platform cost	Input + output + possible retries
Latency	Measure on the target short decision	Varies with model, load, and output length
Best fit	Routing, gates, classification, risk signals	Explanation, summary, advice, and generation

Not a product benchmark; a framework for a production measurement.

The TankDev decision benchmark: a measurement contract

A provider demo cannot prove that an architecture is fast or inexpensive for a production domain. Decision quality depends on the data domain, the definition of options, and the cost of a wrong result. TankDev's recommended benchmark runs one closed evaluation set through three paths: deterministic rules, Jev/System One, and a structured-output LLM. The aim is not the largest headline percentage; it is to show which decisions can safely be automated in which layer.

```benchmark flow
Labeled evaluation set (for example, 500 anonymized requests)
│
├──► Deterministic rule ─────────► result + duration + cost
├──► Jev / System One ───────────► result + distribution + confidence
└──► Structured-output LLM ──────► result + output tokens

For every result: correct label • p50/p95 • call cost • fallback • human override
Segment separately: channel • language • account type • novel/rare category




| Measure | How to calculate it | Why it matters |
| --- | --- | --- |
| Selective accuracy | Correct rate among decisions actually automated | Shows automation quality above the confidence threshold |
| Coverage | Share of all records that received an automatic action | Prevents looking good only on easy examples |
| Risk-weighted error | Weight each wrong result by business impact | Separates a misroute from a money or access error |
| p50 / p95 latency | Time from client request to decision | Exposes queues and tail values that an average hides |
| Fallback and override | Rate of human review or later correction | Shows threshold and question-design maturity |

*A publishable benchmark reports coverage and error cost, not accuracy alone.*

This article makes no numerical benchmark claim; it provides a **results template**. Before publishing actual results, document the data source, sample size, labeling method, model/question version, date range, and exclusion criteria. Without that, a comparison is a marketing chart rather than engineering evidence.

| Method | Selective accuracy | Coverage | p95 | Call cost | Note |
| --- | --- | --- | --- | --- | --- |
| Rule engine | to measure | to measure | to measure | to measure | Reference for explicitly governed examples |
| Jev / System One | to measure | to measure | to measure | to measure | Report with confidence threshold and calibration |
| Structured-output LLM | to measure | to measure | to measure | to measure | Separate schema validity from decision correctness |

*TankDev benchmark result card — fields that must remain empty until measurement.*

## Hybrid production architecture: decision model, policy, and LLM

Do not attach a decision model directly to an external side effect. Put a policy layer between the decision and the action. The policy engine applies thresholds, user roles, transaction limits, operating hours, retry count, and risk classes. The model may say that a vendor payment is high risk; explicit application policy decides whether money can move.



```typescript
const decision = await jev.decide(ticket);
const autoRoute = decision.route.confidence >= 0.85;
const needsReview = decision.human_needed.noul >= 0.20;

if (!autoRoute || needsReview) {
  await reviewQueue.enqueue({ ticketId, decision, policyVersion: "2026-09-27" });
  return { status: "pending_review" };
}

await workflow.dispatch({ queue: decision.route.choice, ticketId });
// Call an LLM only when a human-readable draft or explanation is required.

Confidence is not authority: calibration and thresholds

A confidence of 0.85 only approaches ‘85% correct’ when the model is calibrated on your domain. Begin in shadow mode: record recommendations, compare them with an existing rule or human outcome, and inspect error by segment. Language, customer tier, channel, product family, and newly introduced categories need separate measurement.

Log a safe state reference, model version, question/policy version, result, distribution, threshold, and final action for every decision.
Choose thresholds from false-positive and false-negative cost, not headline accuracy.
Do not turn low confidence into an automatic rejection; design review, an alternate path, or a request for richer context.
Compare old and new versions under controlled traffic whenever the model or question text changes.

What Jev does not do

Jev does not write text. It is therefore not the right tool for explanations, long summaries, code, or multi-step research. A fixed output space prevents an invented schema value, but it does not prevent a high-confidence selection of the wrong defined option. ‘Zero hallucinations’ must not be interpreted as correct business decisions. If a deterministic rule already exists—such as amount > 100000—use ordinary code rather than a model call.

Note: Good automation does not send every classification to AI. Ask whether the rule is explicit. Use code for an explicit rule, a decision model for a narrow uncertain judgment, and an LLM when language or explanation must be produced.

Production release checklist

Write the business goal, allowed actions, and reversibility of each decision.
Describe every Choice option and Score level with observable criteria.
Prepare an evaluation set from labeled or human-decided examples; do not reuse training examples.
Specify a fallback for low confidence, timeout, and service failure.
Monitor queue delay, model latency, confidence distribution, override rate, error rate, and call cost.
Apply dual approval, limits, idempotency, and audit logs to high-impact actions.

Frequently asked questions

Is Jev an LLM?

No. TypeSafe defines Jev as a System One decision model that does not generate text. It accepts state and typed questions, then returns Choice, Score, or Noul results. It does not replace an LLM for conversation, long explanation, or content generation.

Why use Jev when Structured Outputs exist?

Structured Outputs constrain shape; they do not by themselves solve an LLM's generation cost, latency, or decision uncertainty. Jev provides typed probability and confidence signals for narrow decisions that code consumes. The right approach depends on measured performance and the cost of an error.

Does Jev avoid hallucinations?

It cannot create a class outside the defined schema. It can still select the wrong existing option. Confidence, evaluation data, thresholds, fallback behavior, and human approval for high-impact actions remain necessary.

Why does Jev not provide a written rationale?

It is designed to produce a fast machine-consumable decision, not an explanation. When a user needs a rationale, build a separate explanation flow using the decision record and relevant business data with an LLM or human review.

Where does Jev fit in an agent architecture?

Place it at early decision points: tool selection, risk gating, routing, spam or suitability filters, and triggers for human review. A Jev result must not be authority on its own; combine it with policy, permissions, limits, and an audit log.

Should the confidence threshold always be 0.85?

No. There is no universal threshold. Set it using calibration on your labeled examples and the cost of an incorrect outcome. Low-impact routing can tolerate a different threshold from a payment or permission change that requires human approval.

Published by TankDev · Original article: Jev and System One decision models