If you write code that decides whether a command touches production, you will get two kinds of wrong answer. The unknown case decides which kind you get.
Treat a command you can't classify as staging, and sooner or later a production deploy runs without anyone being asked. Treat it as production, and you get an extra prompt for something harmless.
The second mistake is annoying. The first one is an incident. So in the tool I'm building, Styx, a command that doesn't say staging or production counts as production. It asks, and production writes, deploys and deletes need Touch ID, Windows Hello or your system password on top.
Three things made this workable.
Fix noisy prompts by naming the environment, not by loosening the default. If a harmless command keeps asking, make it say what it is.
Write the test for the unknown case first. One row per input you can't classify, each expecting the dangerous bucket. It's the row people forget.
Be honest about the limit. A classifier only sees commands that go through it. An agent that calls a CLI by its full path, or an SDK with a token it found on disk, never reaches it. Fail-closed is a default, not a sandbox.
When you write a classifier like this, which way does your unknown case fall?
Top comments (2)
I would fail closed too, because an extra prompt is recoverable and a missed production write is not. One addition I would make is to separate reads from writes before the classifier runs, so read-only commands do not train people to click through prompts. That keeps the production prompt meaningful when it does appear, and it gives the unknown case a safer middle path without weakening the default.
Agree on the default. One test row I would add: a command that says staging but resolves to production through its config, like a kube context or a cloud profile left pointing at prod from an earlier session. The word in the command passes, the target does not. Does Styx classify by the text of the command, or by the context the CLI will actually resolve when it runs?