I Debugged a Silent Accept for 48 Hours. python -O Deleted Every Assert.

The ticket said our upload API accepted a payload that my laptop had rejected an hour earlier. Have you ever watched a guardrail disappear between two machines that supposedly ran the same commit? I spent the next forty-eight hours writing field notes, because guessing at framework magic had already failed me. The handler looked responsible on the first read, which is exactly how this kind of miss happens.

An assistant had generated a compact validator that used assert for required fields, types, and size. I skimmed it, nodded at the familiar pattern, and merged, because the tests on my laptop were green. That green bar was the first lie, and I did not notice until a second interpreter ran the module.

Hour 0–8: I blamed the proxy, then the client

The incoming JSON on the failing host looked boring, which made me suspicious of everything except my own code. I compared Content-Length headers, dumped access logs, and reran curl with --http1.1 in case the body changed. Nothing in the proxy path moved, so why would two CPython processes disagree on a one-line assert? The stack had no network after the handler started, and that should have been my first clue.

I hexdumped the multipart boundary because transfer encoding has burned me on older upload services before. The bytes, filename, and content type all matched, and that left me without a convenient villain. Field notes from that block still read like a person arguing with a load balancer that was not in the room.

Hour 8–20: The payloads matched, so I stopped trusting pytest

Locally, pytest -q failed fast on an empty name, which is exactly the failure I wanted to keep. On the other machine the test file was missing, so I called the function from a REPL instead. Have you had that moment where the REPL stays calmer than pytest, and you immediately distrust both? I copied the module byte for byte with sha256sum and still watched two behaviors come back.

Commands I reran, in order, before I was willing to blame the interpreter flags:

1. sha256sum validate.py on both hosts, which matched and killed the "wrong file landed" theory.
2. python -c "import inspect, validate; print(inspect.getsource(validate.validate_upload))" which also matched.
3. python -c "import sys; print(sys.flags.optimize, sys.version)" which finally disagreed.

That third command is the whole story, even though I spent twelve hours circling logs to reach it. The failing interpreter printed optimize=1, while mine printed optimize=0 like a smug local default. Everything after that line was documentation, because the runtime had already explained the disagreement.

python -c "import sys; print(sys.flags.optimize, sys.version)"
python -O -c "import sys; print(sys.flags.optimize, sys.version)"
printf 'PYTHONOPTIMIZE=%s\n' "$PYTHONOPTIMIZE"

Hour 20–32: The generated validator was never a validator

Here is the shape the assistant had written, and the comments still sounded sure of themselves.

def validate_upload(name: str, body: bytes) -> None:
    assert name, "name is required"
    assert body, "body is required"
    assert len(body) <= 1_048_576, "body too large"
    assert name.endswith(".csv"), "only csv is allowed"

Does that look like request validation to you, or like a tutorial that escaped into a pull request? Asserts are compiled out when CPython runs with -O or when PYTHONOPTIMIZE is set in the environment. The exception never fires, the message never appears, and the caller keeps going with an empty name.

I needed a second interpreter that was not my laptop, plus a second reader for the generated code. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to review the validator as a hostile patch, then reproduced the same module on the free server option with optimize enabled. The review was useful for listing replacements, and the extra machine helped because it did not inherit my shell aliases.

Hour 32–40: Who turned optimize on?

The Dockerfile comment promised faster CPython for production, which is a rumor that refuses to die quietly. Someone, or some assistant, had stacked two optimize switches on top of each other in the image. Would you catch a one-character flag in a wall of YAML when you are tired and the diff looks operational? I did not catch it, and that is the part of the field notes I am least proud of.

ENV PYTHONOPTIMIZE=1
CMD ["python", "-O", "app.py"]

Two switches, the same compiled-out asserts, and still no ValueError on the request path. I also found PYTHONOPTIMIZE=1 in a process unit I had not reviewed, because generated units look copied from a confident blog post. After that, "it works on my laptop" stopped being a joke and started being a flag mismatch.

A reproducible check I will keep

This test does not need the original service, because it only spawns a subprocess with the same interpreter family. I am labeling it as a local recipe you can run, not as a benchmark and not as a claimed outage replay.

# test_optimize_guards.py
import os
import pathlib
import subprocess
import sys
import tempfile
import textwrap

MODULE = textwrap.dedent(
    """
    def validate(name):
        assert name, "name required"
        return "ok"
    """
)


def run(optimize: bool, name: str) -> subprocess.CompletedProcess:
    with tempfile.TemporaryDirectory() as td:
        path = pathlib.Path(td) / "validate.py"
        path.write_text(MODULE, encoding="utf-8")
        probe = "from validate import validate; print(validate(%r))" % name
        cmd = [sys.executable, "-c", probe]
        if optimize:
            cmd.insert(1, "-O")
        env = os.environ.copy()
        env["PYTHONPATH"] = td
        return subprocess.run(cmd, capture_output=True, text=True, env=env)


def test_assert_fires_without_optimize():
    result = run(optimize=False, name="")
    assert result.returncode != 0
    assert "name required" in result.stderr


def test_assert_is_gone_with_optimize():
    result = run(optimize=True, name="")
    assert result.returncode == 0
    assert result.stdout.strip() == "ok"

Run it with pytest test_optimize_guards.py -q and watch the second test prove the hole exists. The second test feels rude, because it shows your validation becoming a no-op under a single flag. If you want the one-liner version while the kettle boils, this pair is enough to startle you.

python -c "assert False, 'nope'"
python -O -c "assert False, 'nope'"

The first command should explode with AssertionError, and the second command should print nothing and exit zero. I still flinch when the second one succeeds, because that success is the bug wearing a calm exit code.

Decision table I wish I had pasted into the PR

I keep this table next to generated handlers now, because assistants love assert when a short line looks clean in a diff. Clean is not the same thing as still being executable after someone enables optimize in a container.

Guard style	Default python	python -O or PYTHONOPTIMIZE=1	Use on a request path?
assert cond, "msg"	raises AssertionError	compiled out, silent	no
if not cond: raise ValueError("msg")	raises	raises	yes
Pydantic or msgspec parse	raises	raises	yes
if __debug__: check()	runs	skipped	internals only

What broke, in one list

• The assistant used assert as input validation, which is a tutorial habit that looks tidy in review.
• A container flag enabled optimize, so those asserts never existed by the time a request arrived.
• My laptop tests ran without -O, so they could not see the hole they were supposed to protect.
• The failing host did not ship the test file, so pytest could not save me when the REPL stayed quiet.
• Comments about faster production Python survived review because they sounded operational instead of dangerous.

What I would repeat

1. Print sys.flags.optimize in every boot log, sitting next to the git SHA and the Python version string.
2. Ban assert on request paths with Ruff (B011) and a handler-specific grep before anyone merges.
3. Run one CI job with PYTHONOPTIMIZE=1 so generated asserts fail in public instead of on a quiet host.
4. Replace assistant-written asserts with if/raise or a real parser before the pull request is called done.
5. Keep a second machine in the loop when the first machine is also the author of the green bar.

Limitations, and who should skip this

This write-up is not a reason to disable assertions in your test suite, and it is not a performance study. I did not measure request latency, and I will not pretend that -O is free speed you should turn on. Asserts are still fine for internal invariants that should never happen when the surrounding code is correct.

Skip this approach if you need a compliance boundary, because a shared scratch machine is not your regulated environment. Skip it if your runtime is not CPython, because optimize flags do not mean the same thing on every interpreter. Skip it if the real bug is authorization, because swapping assert for ValueError will not fix missing permission checks.

If your laptop is the only interpreter you trust, add an -O job before the next generated validator lands. The test above is the souvenir I kept from the forty-eight hours, and I will paste it into the next review.