I Chased a Missing 403 for 48 Hours. python -O Had Deleted Every assert.

Have you ever watched an endpoint accept a write that your laptop rejected, then blamed the load balancer for two days? I did that in a small lab notebook, and the trail started in a generated helper that looked perfectly serious. The function raised nothing on the remote process, even though the same file printed a clean AssertionError on my machine. Why would one interpreter honor a guard that another interpreter quietly treated as a comment?

This is a 48-hour field notebook, not a product tour. I wanted a boring authorization helper, a remote process I could SSH into, and a record of every false lead. The useful part survives if you never touch the tool I used to draft the first file.

Hour 0: the question I should have asked first

I asked a coding model for a tiny Flask-style gate that rejected writes from a non-admin actor. The sketch came back fast, with type hints, a docstring, and a confident assert on the caller. Did I read the process flags before I trusted that line? I did not, and that omission ate the next two days.

I used MonkeyCode's free model access to draft the helper, then ran the same tree on the free server option so local and remote flags could disagree in public. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The product only mattered as a generator plus a second interpreter. The bug is CPython's, and you can reproduce it with the stdlib alone.

Here is the generated shape, trimmed to the failure and labeled as a lab example rather than production code.

# lab_gate.py — reconstructed from the generated helper
from dataclasses import dataclass


@dataclass(frozen=True)
class Actor:
    name: str
    role: str


def require_admin(actor: Actor) -> None:
    """Reject writes from anyone who is not an admin."""
    assert actor.role == "admin", "forbidden"


def apply_write(actor: Actor, payload: dict) -> dict:
    require_admin(actor)
    return {"ok": True, "payload": payload, "by": actor.name}

On my laptop the guest path failed loudly, which felt like security. On the remote start command, the same guest path returned ok: true. Same file. Same commit. Different __debug__.

Hours 1–8: I blamed everything except the compiler flag

I always start by accusing the network, because that is the cheap story. Was a reverse proxy stripping a header? Was the session cookie expired? Was the remote process reading a stale .pyc from a previous copy?

What I tried, in order:

1. Printed the actor at the top of apply_write and confirmed role="guest" on both machines.
2. Curled the same JSON body with httpie and with curl -v, hunting for a rewritten header.
3. Diffed PYTHONPATH, sys.path, and the file checksum with sha256sum lab_gate.py.
4. Restarted the remote process twice, because restarting is cheaper than thinking.

Nothing moved. The payload was identical, the module path was identical, and the hash matched. So why did only one interpreter raise?

I added a log line above the assert, and that line appeared on both sides. I added a log line inside a branch I expected the assert to prevent, and only the remote side printed it. That is the moment the notebook stopped being a networking story.

Hours 8–18: the start command the model left behind

The generated README had a “faster runtime” snippet. It looked harmless if you skim it after midnight, which I did. Does your model ever gift you python -O as a performance tip without mentioning that -O erases assert?

# the start line I copied without reading the flag
PYTHONOPTIMIZE=1 python lab_gate.py
# equivalent
python -O lab_gate.py

CPython documents this without drama. -O sets sys.flags.optimize to 1, sets __debug__ to False, and removes assert statements at compile time. -OO also drops docstrings. The language reference is the primary source, not a blog rumor: Python command-line -O and the assert statement.

I printed the flags on both interpreters. Locally, optimize was 0. Remotely, it was 1. The “403” had never been a response code. It had been an assertion that the remote compiler deleted before the request arrived.

import sys
print({"optimize": sys.flags.optimize, "debug": __debug__})

Run that as a boot banner. If you only print it after an incident, you will spend hours 1 through 8 the way I did.

Hours 18–30: a reproduction you can run without the original server

I wanted an artifact that does not depend on Flask, Docker, or anyone's free tier. Save this as repro_optimize.py and run both commands on the same machine.

# repro_optimize.py
import sys
from lab_gate import Actor, apply_write


def main() -> None:
    guest = Actor(name="sam", role="guest")
    print("flags", sys.flags.optimize, "debug", __debug__)
    try:
        print(apply_write(guest, {"amount": 50}))
    except AssertionError as exc:
        print("blocked", exc)
        raise SystemExit(2)


if __name__ == "__main__":
    main()

Commands:

python repro_optimize.py
# expected: blocked forbidden, exit 2

python -O repro_optimize.py
# expected: {'ok': True, ...} and exit 0

PYTHONOPTIMIZE=1 python repro_optimize.py
# same as -O

python -c "import dis, lab_gate; dis.dis(lab_gate.require_admin)"
python -O -c "import dis, lab_gate; dis.dis(lab_gate.require_admin)"

The dis dump is the part I would repeat on every future incident. Under a normal interpreter you still see LOAD_ASSERTION_ERROR. Under -O the function body collapses toward return None. You are not looking at a flaky network. You are looking at missing bytecode.

A tiny pytest plan belongs next to that script, because “it worked on my laptop” is not a test.

# test_lab_gate.py
import subprocess, sys
from lab_gate import Actor, apply_write
import pytest


def test_guest_is_rejected_when_asserts_exist():
    if not __debug__:
        pytest.skip("this process already ran with -O")
    with pytest.raises(AssertionError):
        apply_write(Actor("sam", "guest"), {"amount": 50})


def test_guest_must_fail_even_under_dash_o():
    code = (
        "from lab_gate import Actor, apply_write\n"
        "apply_write(Actor('sam', 'guest'), {'amount': 50})\n"
    )
    proc = subprocess.run(
        [sys.executable, "-O", "-c", code],
        capture_output=True,
        text=True,
    )
    # If this ever becomes 0, authorization still lives in assert.
    assert proc.returncode != 0, proc.stderr

The second test is the one I did not have at hour 8. It fails the original helper on purpose. That failure is the point.

Hours 30–40: a decision table I now keep in the repo

I still like assert for internal invariants while I am iterating. I do not like it on any path that a stranger can tickle. The table below is the whole policy.

Situation	Use	Do not use	Why
Impossible state inside your own module	assert or if __debug__:	User-facing denial	-O may delete it
Authn, authz, quota, money, delete	if + raise a real exception	assert	The compiler is allowed to drop it
Parsing untrusted JSON	Schema check, then raise ValueError	assert key in data	Missing keys become silent
Tests	assert in pytest	assert in the code under test as the only gate	Pytest is not your production interpreter
Performance experiment	Measure first	python -O copied from a README	You may delete the only check

Replacement for the helper:

class Forbidden(Exception):
    pass


def require_admin(actor: Actor) -> None:
    if actor.role != "admin":
        raise Forbidden("forbidden")

That if stays in the bytecode under -O and -OO. The exception type also gives the web layer a real 403 instead of a surprising AssertionError that some frameworks turn into a 500.

Hours 40–48: what broke, and what I would repeat

What broke was not Flask, not SSH, and not “the model being bad at HTTP.” The model wrote a plausible invariant. I promoted that invariant into a security boundary. The remote start line then asked CPython to optimize, and CPython obeyed the language spec.

What I would repeat:

• Print sys.flags and __debug__ on process boot, in the same banner as the git sha.
• Grep the request path for assert before calling the change done: rg -n "assert " -g '*.py'.
• Keep one subprocess test that executes the denial under python -O.
• Treat generated READMEs as untrusted, especially lines that mention -O, -OO, or PYTHONOPTIMIZE.
• Disassemble any function that is supposed to refuse work, because missing bytecode is clearer than missing logs.

What I would not repeat: chasing cookies for eight hours when a log line after the guard already proved the guard never ran.

Limitations, and who should skip this workflow

This notebook is about one CPython switch. It does not prove that free models are careful, that free servers match production, or that your orchestrator uses the same flags tomorrow. I am not naming models, quotas, or hardware here because I did not measure those things.

Do not use assert as authorization, even after you “know” your server is not optimized. A later image, a distro packager, or a well-meaning -O in a systemd unit can flip the flag without editing your module. Do not use this write-up as a reason to disable assertions in tests. Pytest lives on assert, and that is a different interpreter contract.

Skip the whole remote-generator loop if you cannot read the start command, cannot print sys.flags, or cannot run untrusted code in an isolated tree. A free interpreter is still an interpreter. If your threat model includes secrets, this is the wrong sandbox.

Also skip it if your actual bug is import shadowing, a bound name from from x import get, or a TTY-less stdout. Those are different 48-hour stories, and mixing them will hide the dis output you need.

The durable habit is smaller than the incident. Ask whether the refusal is an invariant for developers or a contract for strangers. If it is a contract, write an if and a real exception. Then run the denial once with python -O before you trust the banner that says the service is up.