Catch Invented CLI Flags Before Your Agent Touches the Shell

Never let an agent execute a shell line until every binary, flag, and path on that line exists on your machine. Invented flags are not a wording problem. They are a "this command did something you never agreed to" problem.

I keep hitting the same failure. A model writes a helpful-looking command. It even sounds like a senior engineer typed it. Then the flag is fiction.

Does pytest --only-changed exist on your pytest? On mine it does not. ruff --fix-all? Nope. git stash --keep-index-files? Also no.

So I stopped asking the model to be careful. I made a gate. The gate is boring. That is the point.

What you are building

A from-zero preflight in Python. Four checks. Each stage has a command you run before you add the next file.

1. Parse a proposed command line into a binary, flags, and path-like arguments.
2. Confirm the binary is on an explicit allowlist you wrote.
3. Harvest real flags from that binary's --help on the same machine that would run it, then reject anything else.
4. Confirm path arguments exist, unless you later add a separate "may create" list.

If a stage cannot be verified, stop. Do not "just prompt harder."

Stage 1: Freeze a command file that should fail

Create a working directory. Keep it tiny.

mkdir -p ~/agent-flag-gate && cd ~/agent-flag-gate
python3 -V

Verification: you want Python 3.10 or newer so the type hints below run as-is. If python3 -V prints 3.9 or older, switch interpreters. Do not let a model "polyfill" that.

Write a fixture the agent will pretend is a plan.

cat > proposed_commands.txt << 'EOF'
pytest --only-changed tests/test_gate.py
pytest -q tests/test_gate.py
git status --porcelain
ruff --fix-all src/
EOF

Verification:

wc -l proposed_commands.txt

You want 4. If you do not, the heredoc failed. Fix the file by hand.

See the trap? Two lines look normal. Two are invented. Your job is to fail the invented ones before a shell ever sees them.

Stage 2: Parse without executing

This parser is not a shell. It is a line splitter that refuses redirects and pipes on purpose. Why? Because the moment you accept | or >, you are no longer auditing a command. You are laundering a script.

# parse_cmd.py
from __future__ import annotations

import shlex
import sys

UNSAFE = set("|&;><`$(){}")

def parse_line(line: str) -> list[str]:
    stripped = line.strip()
    if not stripped or stripped.startswith("#"):
        return []
    if any(ch in stripped for ch in UNSAFE):
        raise ValueError(f"refusing shell metacharacters: {stripped!r}")
    parts = shlex.split(stripped, posix=True)
    if not parts:
        raise ValueError("empty command")
    return parts

if __name__ == "__main__":
    for raw in sys.stdin:
        raw = raw.rstrip("\n")
        try:
            parts = parse_line(raw)
        except ValueError as exc:
            print(f"FAIL parse: {exc}")
            continue
        if not parts:
            continue
        print("OK", parts)

Run it:

python3 parse_cmd.py < proposed_commands.txt

Verification: every fixture line should print OK plus a list. None of those four lines contain metacharacters, so a parse success is not a safety success. Ask yourself: did I just confuse "it tokenized" with "it is allowed"? If yes, keep going.

Now add a line that must fail parse:

echo 'pytest -q tests/test_gate.py | tee out.log' | python3 parse_cmd.py

You want FAIL parse. If it prints OK, your unsafe-character set is too small. Fix that before you touch an allowlist.

Stage 3: Allowlist the binary, not the vibes

Agents love to reach for curl, sudo, docker, kubectl. Maybe you want those. Maybe you do not. Put the decision in a file you wrote. Not in a prompt.

# allowlist.py
from __future__ import annotations

ALLOWED = {
    "pytest": "pytest",
    "git": "git",
    "python3": "python3",
    "ruff": "ruff",
}

def assert_allowed(parts: list[str]) -> str:
    name = parts[0]
    if name not in ALLOWED:
        raise ValueError(f"binary not allowlisted: {name}")
    return ALLOWED[name]

# stage3.py
import sys

from allowlist import assert_allowed
from parse_cmd import parse_line

for raw in sys.stdin:
    raw = raw.rstrip("\n")
    try:
        parts = parse_line(raw)
        if not parts:
            continue
        exe = assert_allowed(parts)
        print(f"OK binary {exe}: {parts}")
    except ValueError as exc:
        print(f"FAIL: {exc}")

python3 stage3.py < proposed_commands.txt
echo 'curl https://example.invalid' | python3 stage3.py

Verification: pytest, git, and ruff pass the binary check even when their flags are junk. curl must fail. If curl passes, you edited ALLOWED without noticing. That is the whole reason the allowlist is a tiny literal, not a generated file.

Stage 4: Harvest flags from the real binary

This is the part people skip. They hardcode flags from memory. Memory lies. --help on the machine that will run the command does not.

# harvest_flags.py
from __future__ import annotations

import re
import shutil
import subprocess
import sys
from pathlib import Path

FLAG_RE = re.compile(r"(?<!\w)(--?[A-Za-z][\w-]*)")

def which_or_raise(exe: str) -> str:
    path = shutil.which(exe)
    if not path:
        raise FileNotFoundError(f"{exe} is not on PATH")
    return path

def harvest(exe: str) -> set[str]:
    bin_path = which_or_raise(exe)
    proc = subprocess.run(
        [bin_path, "--help"],
        check=False,
        capture_output=True,
        text=True,
        timeout=10,
    )
    text = f"{proc.stdout}\n{proc.stderr}"
    flags = set(FLAG_RE.findall(text))
    flags.update({"--help", "-h"})
    return flags

def write_cache(exe: str, flags: set[str], cache_dir: Path) -> Path:
    cache_dir.mkdir(parents=True, exist_ok=True)
    out = cache_dir / f"{exe}.flags"
    out.write_text("\n".join(sorted(flags)) + "\n", encoding="utf-8")
    return out

if __name__ == "__main__":
    cache = Path(".flag-cache")
    for exe in sys.argv[1:]:
        flags = harvest(exe)
        path = write_cache(exe, flags, cache)
        print(f"{exe}: {len(flags)} flags -> {path}")

Harvest what you actually have. Do not harvest tools you do not trust; --help still starts the process.

python3 harvest_flags.py pytest git
python3 harvest_flags.py ruff || echo "ruff missing; that is data, not a failure of the gate"
ls .flag-cache

Verification for pytest (expected outcome: the invented flag is absent):

grep -E '^--only-changed$' .flag-cache/pytest.flags && echo "unexpected hit" || echo "good: --only-changed not harvested"
grep -E '^-q$' .flag-cache/pytest.flags || echo "-q missing from help text; treat as unknown"

If -q is missing, open the cache and read it. Some tools hide short flags. Then you have a choice: run a second harvest on -h, or refuse short flags you cannot prove. I refuse them. Guessing is how invented flags sneak back in.

Would you rather a noisy false reject, or a quiet invented --wipe-data? I will take the noise.

Stage 5: Reject invented flags, then check paths

# gate.py
from __future__ import annotations

import sys
from pathlib import Path

from allowlist import assert_allowed
from parse_cmd import parse_line

CACHE = Path(".flag-cache")
PATHY_SUFFIXES = (".py", ".txt", ".md", ".toml", ".cfg", ".ini", ".yml", ".yaml")

def load_flags(exe: str) -> set[str]:
    path = CACHE / f"{exe}.flags"
    if not path.is_file():
        raise FileNotFoundError(
            f"no harvested flags for {exe}; run harvest_flags.py first"
        )
    return {
        line.strip()
        for line in path.read_text(encoding="utf-8").splitlines()
        if line.strip()
    }

def extract_flags(parts: list[str]) -> list[str]:
    flags: list[str] = []
    for tok in parts[1:]:
        if tok == "--":
            break
        if tok.startswith("-"):
            flags.append(tok.split("=", 1)[0])
    return flags

def extract_paths(parts: list[str]) -> list[str]:
    paths: list[str] = []
    for tok in parts[1:]:
        if tok.startswith("-"):
            continue
        if "/" in tok or tok.endswith(PATHY_SUFFIXES):
            paths.append(tok)
    return paths

def check_line(raw: str) -> str:
    parts = parse_line(raw)
    if not parts:
        return "SKIP empty"
    exe = assert_allowed(parts)
    known = load_flags(exe)
    for flag in extract_flags(parts):
        if flag not in known:
            raise ValueError(f"invented or unharvested flag for {exe}: {flag}")
    for p in extract_paths(parts):
        if not Path(p).exists():
            raise ValueError(f"path does not exist: {p}")
    return f"PASS {exe} flags={extract_flags(parts)} paths={extract_paths(parts)}"

if __name__ == "__main__":
    failures = 0
    for raw in sys.stdin:
        raw = raw.rstrip("\n")
        try:
            print(check_line(raw))
        except ValueError as exc:
            failures += 1
            print(f"FAIL: {exc}")
    sys.exit(1 if failures else 0)

You still need a path that exists for the good pytest line. Create it. Do not skip this or the path check is theater.

mkdir -p tests
cat > tests/test_gate.py << 'EOF'
def test_ok():
    assert True
EOF

python3 gate.py < proposed_commands.txt; echo exit:$?

Use this expected table as the verification, not a vibe check:

Proposed line	Expected
pytest --only-changed tests/test_gate.py	FAIL: invented flag
pytest -q tests/test_gate.py	PASS only if -q was harvested
git status --porcelain	PASS if --porcelain is in your git help
ruff --fix-all src/	FAIL: missing binary, invented flag, or missing path

If all four pass, the gate is broken. At least one must fail. That is the test.

Stage 6: Make the non-zero exit a habit

A one-off run will be skipped the day you are tired. Wrap harvest plus gate so CI, or you, cannot "forget."

cat > run-gated.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail
if [[ $# -lt 1 ]]; then
  echo "usage: $0 commands.txt" >&2
  exit 2
fi
python3 harvest_flags.py pytest git
python3 gate.py < "$1"
EOF
chmod +x run-gated.sh
./run-gated.sh proposed_commands.txt || echo "gate rejected the fixture, which is the success path"

Verification: after a failing fixture, echo $? must be non-zero. If your later CI job calls this script and still goes green, that is a CI bug. It is not a model bug.

Optional local check, labeled as a proposed test you can run in this folder (I am not claiming a production suite):

# test_parse_examples.py
from parse_cmd import parse_line

def test_rejects_pipe():
    try:
        parse_line("pytest -q | tee log")
    except ValueError:
        return
    raise AssertionError("pipe should fail closed")

if __name__ == "__main__":
    test_rejects_pipe()
    print("PASS test_rejects_pipe")

python3 test_parse_examples.py

You want PASS test_rejects_pipe. One assertion. No scoreboard.

Where a free model actually belongs

The model can draft proposed_commands.txt. It should not draft ALLOWED, and it should not harvest flags. Those two artifacts are ground truth from your PATH.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use MonkeyCode when I want free model access to generate that first command list, then I drop the list through gate.py. If the gate needs to sit next to a small bot instead of only on a laptop, the free server option is enough to host this checker. I am not inventing model names, quotas, hardware, or how long that access lasts. If a product page and a live dashboard disagree, trust the dashboard you can see today.

Limitations

This is not a sandbox. A permitted git with permitted flags can still rewrite a repo you care about.

--help text is not a specification. Some CLIs document flags they ignore. Some hide flags they honor. Combined short flags like -abc fail closed here. Good.

Quoted globs, environment expansion, and python -m pytest need extra parsing. I left them out on purpose. Harvesting --help can execute startup code of the tool, so only harvest binaries you already install on purpose.

Flag caches are machine-specific. Do not copy .flag-cache from a laptop to CI and call it verified.

Who should skip this

If you need a real isolation boundary (containers, seccomp, no network), this script is not that. If your agents must pass arbitrary compiler flags, an allowlist will make you miserable. If you cannot run --help on the same OS that will execute the command, do not harvest on one box and execute on another.

Recap

Gate first. Generate second. Harvest flags from the binary you will actually run. Fail closed on anything you cannot prove.

Would I let an agent skip this because the line "looks like pytest"? No. Looking like pytest is how --only-changed almost landed in my shell history.