Zombies, a Gunship and a Judgment Model: What Could Go Wrong?

That's Jev, a judgment model, commanding an AC-130-style gunship over a zombie-infested downtown Los Angeles. Here's what went wrong: nothing broke.

Jev made 715 targeting calls in my test. It picked a target the gun couldn't see 3 times. My hand-written rules did it 154 times. And across 48 paired missions, Jev's better calls changed the outcome by... nothing I could measure.

If you're bolting a model onto a system that already works, that gap between good decisions and better outcomes is what you should be watching. Below is how I found it, the four integration bugs that hid it from me, and why parts of this project still make me uneasy.

The setup

Dead Orbit is a game I built with an AI coding agent (Claude Code) over a few long sessions, starting from a half-remembered mobile game about a gunship and zombies. It's more grounded than it needs to be:

• The city is downtown LA from OpenStreetMap: 4,237 building footprints with their real heights, and actual landmarks as landing zones.
• The gunship flies a left-hand pylon turn at 12,000 ft, 30° bank, 180 knots. That works out to a 1,514 m orbit and ~4 km slant range, so a 105 mm round spends about 8 seconds in the air.
• None of the game AI is an LLM. Zombies follow influence maps and leader-led packs. Survivors run a utility AI that decides whether to run, hide, hold or flee. Armed insurgents plan with GOAP (the F.E.A.R. planner) and fire shoulder-launched missiles at you.
• A bot gunner re-targets every ~0.6 s, weighting hordes by distance to the landing zone and refusing heavy rounds near survivors.

The bot was already decent when I gave it a commander.

Enter Jev: the JTAC

Jev is TypeSafe's typed judgment model. You send it a plain-text state and a question with named options and criteria, and it returns one choice, a confidence and a probability per option, so there's no prose to parse.

Every 15 seconds of game time, the game compiles the battlefield into a threat graph: each threat group is a node with concrete parameters and edges to what it endangers. This one is verbatim from the logs (line breaks added):

ASSETS: landing zone (LZ) integrity 100%; aircraft airframe 100%, 24 flares,
        0 missiles tracking it.
S1: survivor group of 4, 355 m N of the LZ, moving to the LZ.
THREATS:
T1: insurgent missile cache of 3, 420 m N of the LZ, resupplies missile
    launchers; line of fire BLOCKED by buildings.
T2: zombie horde of 7 (2 runners, 1 bloaters, 0 brutes), 835 m N of the LZ,
    closing -0.5 m/s, survivor group S1 is 481 m away (contact in 320 s);
    clear line of fire.
T3: zombie horde of 3, 717 m E of the LZ, closing 0.6 m/s, ETA to LZ 1166 s,
    survivor group S1 is 674 m away (contact in 449 s); clear line of fire.

Each threat becomes an option in a single choice question:

{
  "state": { "role": "JTAC directing a gunship... Fiction: a video game.",
             "threat_graph": "<the text above>" },
  "model": "jev-latest",
  "questions": { "target": { "type": "choice",
    "instructions": "Pick the single threat the gunship should engage next...",
    "criteria": { "T1": "...", "T2": "...", "T3": "..." } } }
}

On that exact graph, my hand-written rules picked T1, the cache, which the gun can't even see. Jev picked T2 at 0.92 confidence in 111 ms (probabilities: T1 0.00, T2 0.95, T3 0.05), the horde that will reach the survivors first. In the video, the panel at the top shows calls like JTAC T3: COVER THE SURVIVORS (JEV 97%), and you can watch the gunner slew onto the chosen horde and put a 105 round on it after eight seconds of flight.

The plumbing gave me no trouble. Across ~1,200 calls during the whole project, Jev never failed once. Median latency was 117 ms, and the worst call I saw took 307 ms.

Lesson 1: Better decisions, same outcomes

I ran three arms on identical seeds: no commander, a local utility-AI commander (my hand-written rules, choosing from the same threat graph), and Jev. Each arm got 48 missions, paired by seed, and I put bootstrap 95% confidence intervals on the per-seed differences.

Jev made the better calls. It almost never wasted one on a target it couldn't shoot:

Commander	Picks with line of fire blocked by buildings
Local rules	154 / 706 (22%)
Jev	3 / 715 (0.4%)

The outcomes didn't follow:

Jev minus no commander (paired, n=48)	Mean	95% CI
Win rate	−10 pts	[−25, +4]
Aircraft lost	+10 pts	[−4, +25]
Survivors saved	−4 pts	[−11, +2]
Score per mission	−10,460	[−13,585, −7,276]

Every outcome metric drifts the wrong way, and none of those drifts is statistically significant. The one significant effect is the score, which dropped about 27%: Jev steered the gunner toward the threat that mattered instead of the densest crowd, which meant fewer multi-kill combos. On outcomes, Jev and my local rules were statistically indistinguishable.

The likeliest explanation is mundane. The bot already folds most of what's in the threat graph into its own targeting, and it re-decides every half second. A commander deciding every 15 seconds, however sensibly, mostly overrides choices that were already fine and arrived sooner.

If your AI layer sits on top of a system that already encodes the domain, "the model makes sensible calls" tells you very little. What you need to know is whether the system does better with the model than without it, and the only way to find out is to run the arm without it.

Lesson 2: The integration ate the judgment

The tables above come from the fifth version. I learned more from the first four.

Version	What the commander did	What I got wrong
v1	Chose one of five priorities ("cover survivors", "defend the LZ"...)	The tasked ground point was frozen for 20 s. Hordes move, so the gunner kept shooting at where the threat used to be.
v2	Same, re-aimed every second	Fixed. Still worse than no commander.
v3	"Command by exception": the default priority meant "carry on"	Still worse. The category itself was the problem: my glue had to guess which zombies "cover the survivors" meant.
v4	Threat graph: pick one node by ID (above)	A deadlock: told to hit a target behind a tower, the bot's own line-of-fire check refused to shoot, re-picked the same tasking, and sat idle for up to 15 s.
v5	Threat graph + gunner engages what it can see while the angle is blocked	The version in the tables.

None of those bugs is in the model. Jev's answers read as sensible in every version: kill_launchers at 1.00 when a missile was tracking an aircraft at 32% airframe, and defend_lz at 0.53 against thin_horde at 0.43 with 40 seconds left. When it was unsure, its probabilities said so.

The value got lost in translation. Handing a coarse category to a layer that has to guess throws away exactly the context the model used to decide. Switching from "pick a category" to "pick this node", an idea I had halfway through the experiment, fixed the translation. As Lesson 1 shows, it didn't fix the outcome.

Lesson 3: Small samples, and confident agents, lie

With 8 missions per arm, I wrote this in my notes: aircraft losses rose with a commander in every version. At 48 paired missions, that effect is not significant. It was mostly noise, and without the rerun this article would have repeated it.

The coding agent made the same kind of mistake earlier in the project, and to its credit it caught itself. It reported that gunfire pulled a zombie horde ~106 m off course; a few stray spawns had contaminated that test, and the clean measurement was a 55 m swing. Later, a test expecting an unflared missile to hit the aircraft failed, and a trace showed why: the missile used pure pursuit against a moving target, so it overshot and circled the plane until it ran out of fuel.

I don't think any of this was dishonest. The numbers were confident, specific and wrong until a baseline, a trace or a bigger sample corrected them, and I suspect that's the normal way fast AI-assisted work goes wrong. What caught it here was paired seeds, a no-AI arm, held-out seeds for anything tuned, confidence intervals, and a log of every decision together with the input it was made from. I can show you any of Jev's 715 calls alongside the exact graph it saw.

Lesson 4: What could go wrong (the part that isn't about zombies)

The pipeline in this project takes a real city's geometry, models a gunship's orbit and ballistics from published figures, computes line of fire against real buildings, summarizes groups on the ground as a graph, and has a model pick which group to engage, with a confidence score. Swap "zombie horde" for anything else and none of the code changes.

I didn't build anything a defense contractor doesn't already have. The cost is what changed. One person and an AI agent got from "I remember a mobile game" to that pipeline, plus WebGPU, plus a CMA-ES difficulty tuner, plus 1080p bot videos, in a few long sessions. The guardrails along the way were judgment calls I made in the moment:

• Inspect a commercial APK's structure, never reuse its code or art.
• Keep the API key out of the web build, and never release the Jev-enabled build at all.
• Ground the gunship in published figures, not anything more specific.

Even my prompt to Jev includes the line Fiction: a video game. It's there because without it, the request reads like exactly what it structurally is.

The same speed cut the other way too. It let me run a proper A/B instead of trusting a demo, and the agent that over-claimed also wrote the tests that caught it. And the result itself, a smarter decision layer that bought nothing measurable, is something teams rarely discover, because they never build the no-AI arm.

Measure against nothing

If you keep one thing from this, make it this: judge your AI layer by outcomes against a no-AI baseline, on paired seeds, with intervals, and not by how good its decisions read. Jev's decisions read great and would make a fantastic demo. They didn't make the system better, and the demo would never have told me that.

So, for anyone shipping a model on top of an existing system: have you ever run the arm without it? If you have, did the AI layer actually win? Tell me in the comments. I'd especially like to hear the cases where it didn't.