Making a Small Decision Model: Part 4, Customization

In Part 1 and Part 2, we built and trained Gev, a 270M-parameter decision model built on Google's T5Gemma 2 backbone. In Part 3, we pushed Gev to play 1993 Doom at 35 frames per second. Now let's look at how you actually wire Gev into your own day-to-day services. But first, why even use something like this in the first place?

The real-world appeal of a 270M decision model comes down to straight-up operational simplicity:

• Low latency right out of the box: It returns decisions in 45 ms on a single GPU.
• Lightweight compute footprint: A full fine-tuning pass requires about 14 GiB of VRAM (Video RAM on a graphics card), fitting comfortably on a single developer workstation rather than a massive compute cluster.
• Zero prompt parsing headaches: Because the model does not generate free-form text, you never have to maintain complex prompt templates, JSON schema retry loops, or fragile regex parsers.

Zero-Shot Steering with GevPredictor

If your task resembles standard ticket triage, policy routing, or risk scoring, fine-tuned Gev works right out of the box without extra training.

Everything in Gev centers on Question and GevPredictor, exposing our three canonical primitives: noul, choice, and score:

from gev_t5gemma.predict import GevPredictor
from gev_t5gemma.schema import Question

predictor = GevPredictor(
    base_model="google/t5gemma-2-270m-270m",
    adapter_dir="gs://.../gev-t5gemma-270m-mix4b",
)

# Define a routing decision using the choice primitive
q_route = Question(
    qid="routing",
    qtype="choice",
    text="Which engineering team should be paged for this database alert?",
    options=["infra_storage", "platform_db", "app_backend"],
    descriptions=[
        "hardware, disks, network connectivity, and raw VM storage",
        "database replication, query performance, and schema migrations",
        "application code logic, API endpoints, and web services",
    ],
)

decision = predictor.predict_one(state=alert_log_text, question=q_route)
print(f"Choice: {decision.choice} (Confidence: {decision.confidence:.2f})")

Why option descriptions act as an instant steering knob

Think of a steering knob as a simple, practical way to adjust model decisions right at inference time without retraining anything or rewriting massive system prompts. Because Gev reads the full description text of each option right before making its choice, updating that description instantly changes how the model evaluates the choice.

Instead of hoping a model follows vague top-level instructions, you are updating the exact rubric right next to each checkbox. If you need a team or category to cover new types of issues, you can just update the description string in your application code, and the model adjusts its routing behavior immediately.

Synthetic Data Generation without Labeled Examples

What if you run a service with a proprietary domain (such as triaging internal infrastructure bugs or reviewing pull requests), but you have zero labeled training examples in hand?

The process for generating good domain data can be a tricky one, but you can use Gemma 4 as a synthetic teacher to generate realistic examples and extract calibrated soft labels:

Figure 1: The end-to-end synthetic teacher pipeline: Python factor sampling, scenario generation with Gemma 4, bidirectional debiasing, and automated quality filtering.

Here is the three-part recipe for building a synthetic teacher pipeline:

1. Sample ground facts in Python first: If you just ask a language model to "invent 1,000 support tickets," it repeats the same handful of clichés over and over. Instead, start with simple Python lists of concrete details (which service broke, the error code, the region, and the customer tier) and randomly combine them first: for example, service=payment-gateway, error=ETIMEDOUT, and customer=enterprise-tier. Then ask Gemma 4 to write a realistic ticket matching those exact facts.
2. Extract position-debiased soft labels: Present the generated scenario to the teacher model and evaluate candidate options twice: once in forward order (A, B, C, D) and once in reversed order (D, C, B, A). Averaging probabilities across both orders cancels out the language model's prompt-order bias, giving true model uncertainty.
3. Filter for factual consistency and smooth labels: Automatically discard any synthetic scenario where the teacher's predicted choice contradicts the underlying Python parameters. Then apply slight label smoothing: softening 100% hard targets to 90% and distributing the remaining 10% across other choices so the student model does not learn overconfident probabilities on borderline tickets.

Fine-Tuning Without Breaking General Skills

When you have labeled rows (human-curated or synthetic), fine-tune from our pretrained checkpoint.

Two habits keep the model honest during fine-tuning:

• Keep general data in the mix: Retain a small slice of general decision data in your training batches so the model preserves broad language comprehension.
• Keep option shuffling and placeholder labels enabled: Shuffling options and replacing key names with generic placeholders 40% of the time forces the model to read descriptions rather than memorizing option slots.

Why Domain Temperature Calibration Matters

Across different dataset distributions, a key factor to watch out for is domain shift:

A temperature setting fit on one domain does not transfer to another.

While Gev's top choice transfers well across domains, raw confidence values can become overly optimistic when the domain shifts. When confidence scaling is tuned on general decision data, running those same settings on domain-specific enterprise workflow tickets can lead to a notable calibration gap, making the model appear overly confident on difficult decisions.

If you want automated rules like "auto-route when confidence is above 95%" to mean that the model is genuinely right 95% of the time, you must re-fit temperature on your own held-out calibration split:

python -m gev_t5gemma.calibrate \
  --base-model google/t5gemma-2-270m-270m \
  --adapter-dir ./my-trained-model \
  --dataset-id my-org/my-domain-dataset \
  --eval-split calibration

The model produces raw match scores called logits: unnormalized output values generated before converting to probabilities. In temperature scaling, the calibration script divides those raw scores by a single learned scalar factor T (temperature) before computing softmax percentages:

calibrated_prob_i = softmax(logits_i / T)

Dividing logits by T softens overconfident probability spikes without changing which option won. The calibration script optimizes T by minimizing three core metrics:

1. Expected Calibration Error (ECE): Measures whether the model's confidence matches reality. If Gev says it is 85% sure across a group of tickets, it should get roughly 85% of them right. Lower is better.
2. Negative Log-Likelihood (NLL): Heavily punishes overconfident mistakes. Being wrong when you claim 55% confidence is fine, but claiming 99% confidence on the wrong answer causes this score to spike.
3. Brier Score: Measures the overall distance between the model's predicted percentages and the true outcomes across all options, rewarding models that give sharp, honest probabilities.

Figure 2: Reliability curve showing how domain temperature scaling aligns predicted confidence with empirical accuracy, eliminating overconfident error spikes.

Good habits and practical takeaways

Here is a quick checklist for rolling out Gev to your own workflows:

1. Start with GevPredictor: Test questions and option descriptions zero-shot using noul, choice, and score.
2. Steer with descriptions: Refine option boundaries using plain-English descriptions before considering any fine-tuning.
3. Generate synthetic data: If you need fine-tuning but lack data, sample facts in Python and distill soft labels with Gemma 4.
4. Calibrate post-hoc: Always run the calibration script on 500 to 1,000 held-out examples of your actual data.

Looking back across all four parts of this series, building Gev was a fun reminder that not every AI problem needs a massive 70-billion-parameter model writing paragraphs of text. When your service just needs to make a fast, structured choice (whether that is triaging a support ticket, catching a phishing email, or dodging a fireball in 1993 Doom), a 270M model that simply points at the right option in 45 ms can go a surprisingly long way. Feel free to jump into the codebase, try a pass on your own data, and see where your own curiosity takes you!