Making a Small Decision Model: Part 2, Training

In Part 1, we introduced Gev, our 270M-parameter decision model built on Google's T5Gemma 2 backbone that scores options in a single 45 ms pass. But taking a compact architecture from raw pretrained weights to a decision engine we might be able to use wasn't straightforward. We ran directly into tough engineering hurdles: causal attention blind spots, severe position biases, and small models' tendency to take lazy shortcuts.

In this post, we'll dive deep into how we solved these challenges: why a pretrained model still requires a one-time training pass, how a subtle left-to-right reading trap caused early runs to blindly select Option 0 over 90% of the time, the essential anti-shortcut habits that force compact models to truly evaluate choices, and the step-by-step training progression we used.

Why do we have to train a pretrained model?

When engineers hear about Gemma as the base model, a natural question comes up: "Wait, if it is a pretrained model, why do we have to train it?"

The answer lies in the model's three-stage lifecycle:

Figure 1: The three stages of Gev: pretrained base model, one-time decision training, and zero-shot deployment.

1. Stage 1: Base T5Gemma 2 270M (Pretraining). Google pretrains this model on trillions of tokens of web text, books, code, and images. Pretraining gives the model broad English grammar, code syntax, and general world knowledge. At this stage, the model knows what an invoice or database error looks like, but it has no concept of <opt> or <decide> tokens, and its pointer comparison head does not exist yet.
2. Stage 2: One-Time Decision Training (Pointer Training with LoRA). We train lightweight adapter layers and the pointer head on a diverse mix of classification, routing, and scoring tasks. Instead of retraining all 270M weights (which takes massive GPU memory and risks overwriting general knowledge), we use LoRA (Low-Rank Adaptation) adapters: small side-layers attached to the frozen base model. Think of LoRA like clipping a lightweight lens onto a camera instead of rebuilding the camera from scratch: it trains a tiny set of extra weights alongside the frozen base model. This teaches the model how to use prompt markers, evaluate candidate options against questions, and produce calibrated probabilities.
3. Stage 3: Zero-Shot Production Deployment. Once Stage 2 is complete, you use Gev zero-shot: passing brand-new questions and custom option descriptions in production without any fine-tuning or weight updates.

One thing to note though is that you only train the model again if you adapt it to a radically different domain, like processing game video frames in Doom.

Why does <opt> go after each option while <q> stays in front?

When designing prompt syntax for an encoder-decoder model, there is a subtle trap: T5Gemma's decoder reads text with causal self-attention (strictly left to right). Every token can only attend to earlier tokens; it cannot look forward into future text.

Suppose you place the option marker before the option text:

<q> Is this invoice approved? <opt> approve: within budget <opt> reject: over limit <decide>

When the decoder reaches that first <opt> token, it has only read the question. It has not seen "approve: within budget" yet. Asking the pointer head to score that <opt> token is like asking a teacher to grade an exam before reading what the student wrote. The marker has no access to the option text.

Figure 2: Placing markers before descriptions blinds causal attention; placing markers after descriptions allows the model to read full option text.

Moving the marker after the option description fixes this completely:

<q> Is this invoice approved? approve: within budget <opt> reject: over limit <opt> <decide>

We actually discovered this trap the hard way: in our early runs, <opt> sat before each option's text, and the model survived only by memorizing fixed option positions. The moment we shuffled the option order during training to break that habit, accuracy crashed from 65% down to 35% because each <opt> tag was blind to its own option description. Moving <opt> to the end of each option let the model read the full text before scoring it, sending multi-choice accuracy past 77%.

One other thing to note is why <q> stays at the front of the question. <q> is just a start marker telling the model where the question begins. The pointer head never scores <q>. It only compares each <opt> token against the <decide> token at the very end, after reading both the question and all candidate options.

How do you stop a 270M model from taking shortcuts during training?

Small models love shortcuts. If options always appear in the same order, a 270M model stops reading and defaults to a positional prior: a lazy habit where the model favors Option 0 simply because it appears first.

Before you drop a decision model into production, here are three good habits to be getting into:

1. Shuffling option order during training

A positional prior is an accidental bias where a network learns that a specific slot (like option 0) is usually correct. Human dataset creators often put the right answer first. During training, Gev randomly shuffles candidate options on every step to break this bias. This shuffling happens only during training. In production inference, you pass options in whatever natural order you want; every option gets a fair evaluation regardless of position.

2. Hiding option names 40% of the time

In Gev, options have both a key name and a description (billing: payment and invoice problems). If the model always sees meaningful key names, it takes a shortcut: it skims the key name and ignores the detailed description. To fix this, we replace semantic key names with generic placeholders (option_0, option_1, or A, B) roughly 40% of the time during training. Doing this forces the model to read detailed descriptions when provided, while still handling short option titles cleanly.

3. Distilling soft labels from a synthetic teacher

Knowledge distillation means training a small student model (Gev) by learning from a larger teacher model (Gemma 4). Instead of training on hard labels (where one option is 100% and all others are 0%), Gev trains on the teacher's soft labels: the nuanced probability distribution output by the teacher (for example, [72%, 28%] rather than [100%, 0%]). Soft labels teach the student model not just what is right, but how close the runner-up option was.

Because large teacher LLMs also have their own prompt-order biases, we run bidirectional debiasing on every synthetic example, querying Gemma 4 in both forward and reversed option order and averaging the distributions. We walk through the full step-by-step Gemma 4 teacher recipe and how to run it on your own data in Part 4.

What's next?

Now that we have a solid decision engine that evaluates options cleanly, we can test what else a 270M-parameter model can handle when pushed past standard text routing.

In Part 3, we take a sudden turn into multimodal territory: enabling T5Gemma's built-in vision encoder to process raw 1993 Doom visual frames in real time at 35 frames per second. We'll walk through how we extracted spatial embeddings, wired the image patches into our decoder without running out of VRAM, and managed the latency budget so the model could actually make split-second navigation choices.

After that, Part 4 brings everything back to software architecture, breaking down the exact recipe to fine-tune, calibrate, and ship Gev into your own stack. We'll share our full Gemma 4 teacher distillation scripts, soft-label loss functions, quantization tricks, and the production server setup we use to keep inference strictly under 45 ms. We will keep it at a high level so you can better understand how it might be applicable to your specific stack.