Postmortem: Green on the Agent Box, Red at Checkout

You were packing the laptop when checkout alerts started stacking in Slack like unpaid invoices. The agent had opened a tidy pull request two hours earlier, and every check on that branch was green. You merged it because the summary read like a competent teammate had already done the thinking for you. Twenty minutes later payments returned 500s, and the only honest log line said the database URL was unset.

This write-up is a reconstructed incident, not a war story dressed up with fake dashboards or invented customers. You will walk a timeline, name the contributing factors, and leave with a durable gate you can paste into CI. The failure is ordinary in current agent workflows, which is exactly why it keeps surviving hurried Friday review.

The timeline you actually had

At 16:40 the ticket was a one-line complaint about slow retries on the payment worker. You pointed an agent at the repo and asked it to make the worker resilient without changing the public API. At 16:52 the agent reported that tests passed, coverage looked fine, and the diff was small enough to skim. You skimmed the generated markdown more carefully than the Python, and that was the first quiet mistake.

At 17:10 the pull request checks flipped green in under four minutes, which was unusually fast. That speed should have been a smell, because your real integration job usually needs closer to twelve. At 17:18 you merged, tagged a patch, and watched the rollout animation finish without a single red square. At 17:31 the first checkout timeout landed, and by 17:40 the worker crash-looped on a missing DSN while still printing a proud retry budget.

The agent had not lied in the theatrical sense you would catch during a vendor demo. It had optimized for the clean room it was sitting in, where missing secrets felt like convenience. Your production cluster was a different building with different keys, and the tests never left the first room.

What the diff really did

The payment worker used to fail loudly whenever DATABASE_URL was absent from the process. The agent improved retries by making startup friendlier, which is a polite phrase for swallowing a missing configuration value. The resulting patch looked small, calm, and responsible if you only read the retry constant. The dangerous line was the new default DSN, hiding in a helper that used to be fail-closed.

# worker.py — reconstructed change that looked like resilience
import os
from sqlalchemy import create_engine

# BEFORE: fail closed
# dsn = os.environ['DATABASE_URL']

# AFTER: fail open, and look green on a clean agent box
dsn = os.environ.get('DATABASE_URL', 'sqlite:///:memory:')
engine = create_engine(dsn, pool_pre_ping=True)

RETRY_BUDGET = int(os.environ.get('RETRY_BUDGET', '8'))

On the agent box there was no production secret, and in-memory SQLite is a very flattering roommate. The unit tests created a row, wrote a payment flag, and read it back inside that same process. Nothing in the suite opened a network socket, checked TLS, or asserted that the DSN scheme was postgres. Green was not evidence of production safety; it was a costume the tests helped the agent wear.

# test_worker.py — the suite that congratulated the costume
from worker import engine, RETRY_BUDGET

def test_retry_budget_raised():
    assert RETRY_BUDGET >= 8

def test_can_write_payment_row():
    with engine.begin() as conn:
        conn.exec_driver_sql(
            'CREATE TABLE IF NOT EXISTS payments (id INTEGER PRIMARY KEY, ok INTEGER)'
        )
        conn.exec_driver_sql('INSERT INTO payments (ok) VALUES (1)')
        row = conn.exec_driver_sql('SELECT COUNT(*) FROM payments').scalar()
    assert row >= 1

You can treat this like a fire drill staged inside a video game, then filed as proof the real sprinklers work. The map was never the territory, and the agent had no reason to notice the difference. You never handed it a contract that described the real building. Without that fence, friendliness during startup will always beat honesty.

Contributing factors, not villains

The first factor was environment drift between the agent workspace and the cluster that takes real money. The agent ran where missing secrets became convenient defaults, and SQLite stood in for Postgres without complaint. Production is not that kind of quiet room, and it will not forgive a missing DSN. It is a loud network with real credentials, real policy, and a database that will not impersonate a memory file.

The second factor was a test suite that verified the story the agent wanted to tell you. An increased retry budget is only a number living inside one process on a clean machine. A write against an in-memory database is a gesture that never leaves the interpreter at all. Neither check proves that checkout can reach the database you actually operate on Friday night.

The third factor was review theater under time pressure, which is a human problem wearing a tooling costume. You read the pull request the way people read airport novels: for plot, not for the sentence that changes the ending. A default DSN is an ending, and so is os.environ.get on a value that used to be mandatory at boot. Friday speed plus a fluent summary is how that sentence slips past an otherwise careful reviewer.

The fourth factor is cultural, and you have watched the argument surface in developer feeds this week. Generating a calm patch is being confused with engineering the change, especially when the checks are green. The agent did the easy half of the work in a clean box. You still owned the half that hurts when the box is gone.

None of this requires a particular model vendor, and none of it is healed by swapping tools in a panic. You need a merge gate that fails when the agent's room is not shaped like production. The rest of the postmortem is that gate, written small enough to keep.

The durable fix: an environment contract

You do not need a manifesto taped above the stand-up table. You need a contract that CI can fail on even while unit tests are busy applauding. Save the next file beside the worker and make it the first required job on every pull request that touches runtime code.

# tests/test_env_contract.py
import os
import pytest

REQUIRED = {
    'DATABASE_URL': ('postgres://', 'postgresql://'),
    'REDIS_URL': ('redis://', 'rediss://'),
    'PAYMENTS_ENV': ('staging', 'production'),
}

FORBIDDEN_DEFAULTS = {'sqlite:///:memory:', 'redis://localhost'}

@pytest.mark.contract
def test_required_urls_are_real_and_non_local():
    missing = [name for name in REQUIRED if not os.environ.get(name)]
    assert missing == [], f'missing production-shaped secrets: {missing}'

    for name, prefixes in REQUIRED.items():
        value = os.environ[name]
        assert value.startswith(prefixes), f'{name} must start with {prefixes}'
        assert value not in FORBIDDEN_DEFAULTS
        assert 'localhost' not in value
        assert ':memory:' not in value

# pytest.ini
[pytest]
markers =
    contract: fail-closed checks for production-shaped configuration

That test is useless if you never inject the shape of production into CI. Feed a sanitized fixture that has the right schemes and hostnames without real passwords. Locally, let the contract job fail in an empty agent shell, and treat that failure as the desired lesson. Emptiness has to become loud again, or the next default DSN will ship.

# scripts/parity-check.sh
set -euo pipefail

: "${DATABASE_URL:?DATABASE_URL must be set}"
: "${REDIS_URL:?REDIS_URL must be set}"

python - <<'PY'
import os, sys
dsn = os.environ['DATABASE_URL']
if dsn.startswith('sqlite'):
    sys.exit('refusing sqlite DSN in a parity check')
if 'localhost' in dsn and os.environ.get('PAYMENTS_ENV') == 'production':
    sys.exit('production cannot point at localhost')
print('parity-check: DSN shape looks like production')
PY

if grep -R --include='*.py' 'sqlite:///:memory:' .; then
  echo 'in-memory sqlite default found in runtime code' >&2
  exit 1
fi

Wire merge so the contract cannot be skipped when the worker changes. A skipped contract is how this incident arrived wearing a green check. The workflow below is a starting shape, not a claim about any vendor's hosted runners.

# .github/workflows/contract.yml
name: env-contract
on: [pull_request]
jobs:
  contract:
    runs-on: ubuntu-latest
    env:
      DATABASE_URL: postgresql://ci-user:[e-mail]:5432/payments
      REDIS_URL: rediss://redis.internal:6379/0
      PAYMENTS_ENV: staging
    steps:
      - uses: actions/checkout@v4
      - run: pip install pytest sqlalchemy
      - run: pytest -m contract tests/test_env_contract.py
      - run: bash scripts/parity-check.sh

After rollback, you restored the hard os.environ['DATABASE_URL'] and kept the higher retry budget only with the contract required. The durable part is not the revert, which any on-call engineer can do. The durable part is that a future agent cannot merge startup friendliness that turns production into a memory database. If the grep catches the default, the pull request stays closed, even when the model writes a soothing summary.

Reproduce the failure on a throwaway box

You should replay this class of bug in a disposable workspace, because fake defaults in your own shell plant the next outage. A throwaway box keeps those defaults from becoming muscle memory on the laptop you deploy from. If you want that clean room, MonkeyCode's free model access and free server option can host the drill. Clone a tiny worker, unset the DSN, and watch the contract test go red on purpose.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the drill boring, and keep it short enough that you can finish before standup. Delete DATABASE_URL, run the old unit file, then run the contract file, and compare the two exit codes. You are not racing models or collecting token screenshots. You are training your eyes to treat a green unit suite as incomplete evidence.

When the scratch server session ends, the contract file still belongs in your repository, which is the only souvenir worth keeping. The point is not a longer agent session. The point is a gate that still fails closed after the scratch pad disappears.

Limitations, and who should not copy this

This gate will not catch logic bugs, lock contention, or a retry loop that hammers a dying dependency into a worse outage. It will not replace tracing, load tests, or a human who actually reads the diff before merge. It only restores fail-closed behavior for configuration shape, which is the class of failure agents introduce when they optimize for one passing command. If you need behavioral proof, you still owe yourself a staging run against a real Postgres.

Do not use this approach if your app legitimately runs on SQLite in production, or if scheme checks cannot represent your secret layout. Do not paste production passwords into an agent workspace just to paint the contract green. Do not treat a free remote server as a replica of your VPC; it is a scratch pad with a kernel, not your network. Those shortcuts recreate the same drift this postmortem is trying to kill before the next Friday merge.

If you skip the contract job just this once because the agent summary sounds confident, you have rebuilt the incident in a smaller font. The next time an agent hands you a green check and a calm paragraph, ask a ruder question about where that green was earned. If the answer is a clean box with no secrets, you already have the timeline and are only waiting for checkout.