Yesterday my drone early-warning demonstrator recorded four AI-generated explanations. All four were discarded by the validation gate that sits in front of them.
I looked at that number and felt good about it. A gate that rejects four out of four is a gate doing hard work. It is exactly the statistic I would put in a technical brief.
It was wrong. Not the gate. My reading of it.
What the system does
The demonstrator fuses signals from several sources and decides a threat level using deterministic rules. No model touches that decision. A local language model is then allowed to do one thing: rewrite the machine's reasoning into a sentence an operator can read at speed.
Because a small local model is not reliably obedient, the sentence is checked before anyone sees it. It has to name the threat level the rules computed, and no other level. Anything else is thrown away and the deterministic wording stands. I tested that gate when I built it. Seven of twenty-five sentences were unfit for an operator display. After adding the gate, thirty out of thirty conformed.
So when the production counter said zero of four survived, the story wrote itself. Sporadic escalations, a small model, a strict gate. Of course some get discarded.
The counter was measuring two different things
The function that calls the model returns nothing when the gate rejects the text. It also returns nothing when the model does not answer at all. Timeout, connection refused, anything. Both cases came back as the same empty result, and I recorded both as "rejected by the gate."
That is the whole bug, and it is not really a coding mistake. It is a naming mistake. I gave one counter to two different facts.
One of those facts says the safety mechanism is working. The other says the feature is dead. They are opposites, and I had made them indistinguishable.
What was actually happening
The model runs locally on the same machine, which is the point: an operational deployment sits on a network with no route to a commercial API. The runtime unloads an idle model from memory to free it up. Reloading two gigabytes on four CPU cores takes longer than the thirty second timeout I had set.
Escalations are sporadic by design. The system is quiet most of the time. So nearly every escalation arrived to find the model cold, waited thirty seconds, gave up, and fell back to the deterministic wording.
Warm, the same prompt answers in 7.8 seconds and passes the gate on the first try. I measured it directly against the running model, with the exact production prompt, after the diagnosis.
The feature had not failed. It had never run.
The fix took ten minutes. Finding it took the audit trail
Three changes. Ask the runtime to keep the model resident so a quiet hour does not guarantee the next escalation fails. Raise the timeout well past a cold load, since a slow answer costs nothing when the fallback is already correct. And split the counter into four: accepted, rejected by the gate, empty, never answered.
That last one is the change that matters. The first two are configuration. The third is the difference between a number you can act on and a number that comforts you.
I would not have found any of it without something I had built the same afternoon: durable history, so every assessment is written to disk with what the rules decided, what the model said, and whether its sentence survived. That storage layer existed for a completely different reason. It was built so a host site could eventually be handed a measured false alarm rate. Four hours after it went in, it told me the headline AI feature had been inert for a day.
The second thing it caught
Once the model started answering again, one of its sentences read: "at HIGH confidence level justifies the HIGH threat level."
The gate accepted it, correctly by its own rule, because it names the computed level and no other. But look at what the sentence asserts. The system computes a confidence per detector. It never computes an aggregate confidence about anything. That sentence states a measurement nothing in the system produced.
The prompt already forbade it. The instruction is right there in the system message, and the model ignored it, which is the entire reason a gate exists instead of trust.
So the gate now rejects that phrasing rather than tolerating it. One existing test asserted the old, permissive behavior. I flipped it deliberately and wrote down why, because a test that quietly changes meaning is worse than no test.
What I would take from this
If you run a model in production behind any kind of check, the useful question is not how often the check fires. It is whether you can tell the difference between the check firing and the model never answering.
Those look identical from the outside. Both produce no output. Both leave your fallback in place. Both can be reported as a single tidy number that makes your safety mechanism look busy.
Instrument the difference. A counter that merges a working safeguard with a dead dependency is worse than no counter, because it does not stay silent. It reassures you.
The uncomfortable part is that my failure mode was the same one I write about: a confident output with nothing real behind it. It just happened to be a metric rather than a model.
A note on what this system is, because it matters in this category. DroneWatch AI is a working concept demonstrator. The threat signals are simulated. The air traffic feed is live and real. It is not an operational system, it does not detect real drones, and no field validated performance is claimed. It is strictly defensive, detection and early warning only, with no interdiction or countermeasure function.
Top comments (0)