Context. A reader asked a plain question under one of my articles: what does a kill-gate even look like for a social robot, when "did the conversation work" does not give you a clean number the way a benchmark does. The question came from a reader building Bite, a continual-learning project for social robots. Answering it honestly surfaced a different question underneath, one that applies well beyond robots. Anything built between pure research and a product that will eventually ship raises the same problem: who is allowed to decide the threshold the research has to pass?
Decision. The threshold has to come from the research question itself, never from whatever a later product, client, or roadmap decides to call success. A kill-gate exists to tell me honestly whether an idea works, not to confirm a number someone already needed. The moment an external target gets to define "good enough," the measurement stops being a test and becomes a formality the work is expected to pass.
In practice this means I write the threshold before I know whether a product, an article, or a client deliverable depends on the result. I write it from the mechanism I am trying to understand: how much does fine-tuning actually cost in forgetting, where does a masking layer silently miss, how much does quantisation degrade a model's output. None of those questions care what a roadmap wants the answer to be, and the threshold should not either.
Alternatives considered. The obvious shortcut is to let the application define success first and work backward: decide what counts as "fixed," "retained enough," or "accurate enough" based on what the product needs, then run the experiment to confirm it. I rejected this, and not on principle alone. My EWC-versus-L2 comparison is the concrete case. The registered retention gate technically fired, both methods looked like they reached the same bar. If I had stopped at that single pass or fail number, the way a product requirement would, the article would have reported a tie. The full picture, the cost each method paid on the new task to reach that same retention, told a different and more useful story. A threshold borrowed from an application would have hidden exactly the part that mattered.
The same risk shows up anywhere a pass or fail number is the only thing reported. A silent miss, like a recognizer never active for one language, produces zero errors and a clean-looking pass rate right up until someone checks per entity type instead of on average. The kill-gate has to be shaped by the question, not by the report a stakeholder wants to see.
Consequences. What this buys me is a measurement I can trust when it disagrees with what I hoped to find. The canary experiment that caught a masking recognizer silently missing an entire language only caught it because the gate was built around detection fidelity, not around "does the pipeline run without errors." A product-shaped threshold would have called that pipeline healthy.
What it costs is control over the timeline. Sometimes the honest answer is that a method does not clear the bar I would have liked, and the product, the article, or the client conversation has to adapt its claims to the result, not the other way around. That is slower than writing the requirement first and designing a test that is guaranteed to satisfy it. It is also the only version of the measurement worth publishing.
Top comments (0)