Python LLM Fine-Tuning Evaluation Gate

🚀 Technical Briefing: This tutorial is part of our deep-dive series on Agentic Workflows at Gate of AI. For the full technical breakdown, interactive code sandbox, and the native Arabic translation, visit the original article here.

Python LLM Fine-Tuning Evaluation Gate Workflow

Build a local, auditable release gate for fine-tuning datasets and model outputs. This tutorial validates JSONL files, fingerprints datasets, detects exact holdout overlap, scores structured predictions, and produces a promotion report before a model is approved for use.

Editorial note: this guide intentionally does not prescribe a provider-specific upload endpoint, model identifier, price, or training-job API. Those details must be verified against the current official documentation and your organization’s approved data-processing terms before a dataset is submitted to a model provider.

Why an Evaluation Gate Belongs Before Fine-Tuning

Fine-tuning changes model behavior using task-specific examples. Research on large-language-model adaptation distinguishes fine-tuning from prompt engineering: prompting guides a model at inference time, while fine-tuning adapts behavior from a training corpus. A broad review of fine-tuning practice describes a lifecycle that spans data preparation, model initialization, optimization, evaluation, and deployment. That lifecycle is important because a completed training run is not itself evidence that a model should be released.

A release gate turns that lifecycle into an engineering control. Before any training submission, validate that examples follow the expected schema, that target responses are present, and that the holdout set is separate. After a provider returns a candidate model, run the same holdout prompts against the baseline and candidate, calculate task-specific metrics, preserve the evidence, and approve promotion only when the defined requirements pass.

This pattern is especially useful for stable and measurable tasks such as classification, extraction, controlled formatting, routing, and code-review conventions. It is less suitable as a way to inject rapidly changing facts. When a workflow needs current policy, inventory, account, or incident information, obtain that information from approved retrieval or internal systems at runtime rather than assuming a fine-tuned dataset remains current.

Prerequisites

• Python 3.10 or newer.
• A labeled training dataset and a separately curated holdout dataset.
• A baseline model and a candidate model that can both be invoked through your organization’s approved inference path.
• An approved process for reviewing data rights, sensitive information, and provider data-processing requirements before remote submission.
• Basic familiarity with JSON, JSON Lines, command-line tools, and Python virtual environments.

The code below uses only the Python standard library. It is therefore useful before choosing a provider and does not make unverified assumptions about a particular SDK. Its input is JSONL data and saved model predictions. Your approved provider integration can generate those predictions later.

Step 1: Create Separate Training and Holdout Files

Use JSONL, where each non-empty line is one JSON object. For this tutorial, the task is support routing. The training file includes an assistant target. The holdout file includes an expected label used only by the evaluator. Do not submit the expected metadata as part of a provider training file unless its documented format explicitly permits it.

Create data/train.jsonl:

{"messages":[{"role":"system","content":"Return JSON with queue and priority."},{"role":"user","content":"My invoice shows two annual subscription charges."},{"role":"assistant","content":"{\"queue\":\"billing\",\"priority\":\"high\"}"}]}
{"messages":[{"role":"system","content":"Return JSON with queue and priority."},{"role":"user","content":"The dashboard fails when I export usage data."},{"role":"assistant","content":"{\"queue\":\"technical\",\"priority\":\"high\"}"}]}
{"messages":[{"role":"system","content":"Return JSON with queue and priority."},{"role":"user","content":"Someone changed our payout bank account without permission."},{"role":"assistant","content":"{\"queue\":\"security\",\"priority\":\"urgent\"}"}]}

Create data/eval.jsonl with prompts that are not duplicates of the training prompts:

{"messages":[{"role":"system","content":"Return JSON with queue and priority."},{"role":"user","content":"Our card was billed twice after adding seats."}],"expected":{"queue":"billing","priority":"high"}}
{"messages":[{"role":"system","content":"Return JSON with queue and priority."},{"role":"user","content":"A former employee can still enter our organization."}],"expected":{"queue":"security","priority":"urgent"}}
{"messages":[{"role":"system","content":"Return JSON with queue and priority."},{"role":"user","content":"The mobile application closes immediately after launch."}],"expected":{"queue":"technical","priority":"high"}}

The tiny files above are syntax examples, not sufficient evidence for a production decision. A real dataset needs broad, reviewed coverage of common cases, edge cases, language variation, and known failure modes. Where appropriate, split by customer, incident, document family, or time period. A random row split can place nearly identical material in training and evaluation, inflating results through leakage.

Step 2: Build the Local Validation and Scoring Tool

Create fine_tune_gate.py. The complete program validates both datasets, computes SHA-256 fingerprints, rejects exact normalized prompt overlap, and evaluates saved predictions. A prediction file contains one JSON object per holdout case, in the same order as the evaluation file. Each object must contain a content string containing the model response.

from __future__ import annotations

import argparse
import hashlib
import json
import sys
from datetime import UTC, datetime
from pathlib import Path
from typing import Any

VALID_ROLES = {"system", "user", "assistant"}
VALID_QUEUES = {"billing", "technical", "security", "general"}
VALID_PRIORITIES = {"low", "normal", "high", "urgent"}


def now() -> str:
    return datetime.now(UTC).isoformat()


def load_jsonl(path: Path) -> list[dict[str, Any]]:
    if not path.is_file():
        raise ValueError(f"Missing file: {path}")
    records = []
    for number, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
        if not raw.strip():
            continue
        try:
            value = json.loads(raw)
        except json.JSONDecodeError as error:
            raise ValueError(f"{path}:{number} is invalid JSON: {error.msg}") from error
        if not isinstance(value, dict):
            raise ValueError(f"{path}:{number} must be a JSON object")
        records.append(value)
    if not records:
        raise ValueError(f"{path} has no records")
    return records


def fingerprint(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def validate_messages(record: dict[str, Any], location: str, require_target: bool) -> None:
    messages = record.get("messages")
    if not isinstance(messages, list) or len(messages) < 2:
        raise ValueError(f"{location}: messages must contain at least two items")
    for message in messages:
        if not isinstance(message, dict):
            raise ValueError(f"{location}: each message must be an object")
        if message.get("role") not in VALID_ROLES:
            raise ValueError(f"{location}: unsupported role")
        if not isinstance(message.get("content"), str) or not message["content"].strip():
            raise ValueError(f"{location}: content must be a non-empty string")
    if require_target and messages[-1]["role"] != "assistant":
        raise ValueError(f"{location}: training example must end with assistant target")


def user_prompts(records: list[dict[str, Any]]) -> set[str]:
    prompts = set()
    for record in records:
        for message in record["messages"]:
            if message["role"] == "user":
                prompts.add(" ".join(message["content"].lower().split()))
    return prompts


def validate(train_path: Path, eval_path: Path, minimum_train: int) -> dict[str, Any]:
    train = load_jsonl(train_path)
    evaluation = load_jsonl(eval_path)
    if len(train) < minimum_train:
        raise ValueError(f"Training set has {len(train)} records; minimum is {minimum_train}")
    for index, record in enumerate(train, 1):
        validate_messages(record, f"train:{index}", True)
    for index, record in enumerate(evaluation, 1):
        validate_messages(record, f"eval:{index}", False)
        expected = record.get("expected")
        if not isinstance(expected, dict) or expected.get("queue") not in VALID_QUEUES or expected.get("priority") not in VALID_PRIORITIES:
            raise ValueError(f"eval:{index}: expected queue and priority are required")
    overlap = user_prompts(train) & user_prompts(evaluation)
    if overlap:
        raise ValueError(f"Exact train/eval prompt overlap: {sorted(overlap)[0]}")
    return {"validated_at": now(), "status": "passed", "training_records": len(train), "evaluation_records": len(evaluation), "training_sha256": fingerprint(train_path), "evaluation_sha256": fingerprint(eval_path)}


def parse_output(content: str) -> dict[str, str]:
    value = json.loads(content)
    if not isinstance(value, dict):
        raise ValueError("response is not a JSON object")
    if value.get("queue") not in VALID_QUEUES or value.get("priority") not in VALID_PRIORITIES:
        raise ValueError("response violates routing contract")
    return value


def score(eval_path: Path, predictions_path: Path, minimum_accuracy: float, max_error_rate: float) -> dict[str, Any]:
    cases = load_jsonl(eval_path)
    predictions = load_jsonl(predictions_path)
    if len(cases) != len(predictions):
        raise ValueError("Evaluation and prediction counts differ")
    correct = errors = 0
    results = []
    for index, (case, prediction) in enumerate(zip(cases, predictions), 1):
        expected = case["expected"]
        try:
            actual = parse_output(prediction.get("content", ""))
            passed = actual["queue"] == expected["queue"] and actual["priority"] == expected["priority"]
            correct += int(passed)
            results.append({"case": index, "expected": expected, "actual": actual, "passed": passed, "error": None})
        except (ValueError, json.JSONDecodeError) as error:
            errors += 1
            results.append({"case": index, "expected": expected, "actual": None, "passed": False, "error": str(error)})
    total = len(cases)
    accuracy = correct / total
    error_rate = errors / total
    return {"evaluated_at": now(), "cases": total, "exact_routing_accuracy": accuracy, "parse_error_rate": error_rate, "minimum_accuracy": minimum_accuracy, "maximum_parse_error_rate": max_error_rate, "promotion_status": "approved" if accuracy >= minimum_accuracy and error_rate <= max_error_rate else "rejected", "results": results}


def main() -> None:
    parser = argparse.ArgumentParser()
    sub = parser.add_subparsers(dest="command", required=True)
    check = sub.add_parser("validate")
    check.add_argument("--train", type=Path, required=True)
    check.add_argument("--eval", type=Path, required=True)
    check.add_argument("--minimum-train", type=int, default=20)
    assess = sub.add_parser("evaluate")
    assess.add_argument("--eval", type=Path, required=True)
    assess.add_argument("--predictions", type=Path, required=True)
    assess.add_argument("--minimum-accuracy", type=float, default=0.85)
    assess.add_argument("--max-error-rate", type=float, default=0.10)
    args = parser.parse_args()
    report = validate(args.train, args.eval, args.minimum_train) if args.command == "validate" else score(args.eval, args.predictions, args.minimum_accuracy, args.max_error_rate)
    print(json.dumps(report, indent=2, sort_keys=True))
    if report.get("promotion_status") == "rejected":
        sys.exit(2)


if __name__ == "__main__":
    main()

Step 3: Validate Before Remote Submission

Run validation locally. The tutorial has three training examples, so set the instructional threshold to three. In a real release process, use a larger threshold that reflects the task’s diversity and risk.

python fine_tune_gate.py validate --train data/train.jsonl --eval data/eval.jsonl --minimum-train 3

The report includes record counts and SHA-256 hashes. Store these alongside the repository commit, annotation approval, provider job identifier, base-model identifier, and any approved training configuration. A filename such as train-final.jsonl does not identify the actual content used for a run; a content fingerprint does.

The overlap check is intentionally narrow. It catches identical normalized user prompts, but it cannot recognize paraphrases or near duplicates. For sensitive or high-impact tasks, add review procedures that group examples by source account, incident, document, or time window. Consider similarity analysis only when it is technically and legally appropriate for your data environment.

Step 4: Collect Candidate Predictions Through an Approved Integration

After your organization has verified a provider’s current fine-tuning documentation and completed the training job, send each holdout prompt to the baseline and candidate through the approved inference integration. Save each model’s raw response separately. Do not alter model output before preserving it, because raw evidence is needed to investigate parsing failures and unexpected behavior.

For example, data/candidate_predictions.jsonl might contain:

{"content":"{\"queue\":\"billing\",\"priority\":\"high\"}"}
{"content":"{\"queue\":\"security\",\"priority\":\"urgent\"}"}
{"content":"{\"queue\":\"technical\",\"priority\":\"high\"}"}

Keep generation conditions consistent when comparing models. The same holdout prompts, system instructions, output contract, and decoding policy should apply to both baseline and candidate unless the change is a deliberate part of the experiment. Otherwise, the comparison measures multiple interventions rather than the effect of the candidate model.

Step 5: Score the Candidate and Enforce Promotion Rules

python fine_tune_gate.py evaluate --eval data/eval.jsonl --predictions data/candidate_predictions.jsonl --minimum-accuracy 0.85 --max-error-rate 0.10

The evaluator calculates exact routing accuracy and JSON-contract error rate. Exact routing accuracy requires both fields to match. This is appropriate when downstream automation depends on both queue and priority. The parser metric is separate because an answer can be semantically plausible yet operationally unusable when it violates the machine-readable contract.

The command returns exit status 2 when the release gate rejects the candidate. In CI, use that nonzero status to block promotion, but always archive the printed JSON report. Case-level evidence tells reviewers whether failures arise from ambiguous labels, inconsistent targets, insufficient coverage, output-format drift, or a task that fine-tuning does not improve.

Do not lower a threshold merely because a candidate fails. First inspect the errors. Improve label definitions, add approved examples representing failure clusters, or revise the system instruction. Compare the candidate with baseline results on the same holdout set. For consequential workflows, add category-specific minimums, a separately reviewed adversarial set, latency and cost measurements, and human approval before any staged rollout.

Prompt Engineering, Fine-Tuning, and Data Governance

Fine-tuning is not automatically the right answer. Prompt engineering may be preferable when the behavior can be expressed clearly in instructions or examples at inference time. Fine-tuning may be worth testing when a stable task needs consistent behavior across many requests and the organization can curate high-quality examples and evaluate the result. The cited fine-tuning literature also highlights resource constraints, so teams should treat training and repeated evaluation as deliberate investments rather than a default response to every quality issue.

Before any external upload, remove credentials, authentication material, unnecessary personal data, payment data, health information, and other restricted content unless your organization has a documented, lawful basis and approved safeguards for processing it. Maintain a clear record of dataset ownership, approval, purpose, retention, and access. Fine-tuning examples are not ordinary logs: they are selected material intended to influence a model’s future behavior.

For rapidly changing facts, route the model to an approved retrieval or internal service after classification. For example, a routing model can select a support queue while deterministic systems retrieve current invoices, incidents, or account status. That separation makes factual updates independent of retraining and gives teams clearer points for authorization, logging, and review.

Key Takeaways

• Validate training and holdout data before a fine-tuning submission.
• Keep holdout examples separate from training material and reject obvious overlap.
• Fingerprint both datasets so a release can be traced to exact content.
• Evaluate candidate behavior against a task-specific contract, not training completion alone.
• Use a nonzero CI exit code to block failed releases, while preserving case-level evidence for review.
• Verify all provider-specific models, APIs, costs, supported formats, and data terms directly from current official documentation before integration.

Sources

• Pornprasita, C. and Tantithamthavorna, C. “Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automation,” arXiv:2402.00905.
• “The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs,” arXiv:2408.13296v1.

Prepared by the Gate of AI Editorial & Engineering Teams, GateOfAI, LLC.