My Staging Agent and My Production Agent Were Different Agents. A Customer Proved It by Forwarding Two Emails.

A customer forwarded me two emails on October 14th. One was from that morning. One was from August 12th. Both were replies from my AI agent to the same question about my refund window.

They read like two different companies.

The August reply was careful, cited the 30-day policy, offered a partial credit, and signed off with my name. The October reply was three sentences, quoted a price I stopped charging in 2024, and told her the window was 14 days.

She wasn't angry. She was confused. "Did you change how your agent works? The demo on your site sounds like the first one."

I had not changed the agent. git log on the agent repo showed nothing but dependency bumps in eleven weeks. I told her so, and then I spent the next six hours proving myself wrong.

What I assumed was broken, and what was actually broken

My mental model was: prompt in repo → prompt in production. One file, one path, one truth.

The reality was: prompt in repo → rendered with environment variables → written into a systemd unit file eight months ago → read at boot by the service → passed to the model.

Four hops. I had only ever inspected hop one.

The rollback instructions in my runbook said "redeploy from main." That's a code fix. My problem was not code. It was that the same code, run with a different environment, produces a different employee.

Here's the diff that mattered, once I finally dumped the resolved config from the running process instead of reading the committed file:

TONE_PROFILE         prod: "concise"     staging: "supportive"
PRICE_SHEET_VERSION  prod: "v3-2024-06"  staging: "v5-2026-08"
REFUND_WINDOW_DAYS   prod: "14"          staging: "30"

None of those values were in the repo. They were in /etc/agent/prod.env, which had been written by a deploy script I ran in March and never re-read. When I added the tone and refund variables to the code, I gave them defaults so nothing would crash. The defaults were reasonable in March. They were wrong by June. They were actively lying to customers by August.

The agent never threw an exception. That's the part that still bothers me.

The failure mode that has no stack trace

I want to be precise about why this survived 63 days, because "I was careless" is not a diagnosis you can fix.

Code drift is loud. You deploy, something 500s, you see it in the first ten minutes.

Config drift is silent, because the process is doing exactly what it was told — by a version of you who is no longer in the room. The env var was present. The config parsed. The model call returned 200. My logs said job completed, and my greps for error and exception and traceback came back clean, and clean logs are the most reassuring form of evidence that your system is fine right up until a customer forwards you two emails side by side.

The second discovery was worse. While I was in there, I diffed every variable. Three more had drifted:

• MAX_REFUND_AMOUNT was 200 in prod and 50 in staging. Nobody had ever decided that. It was whatever each file happened to contain the day it was created.
• MODEL was pinned to a version that had been superseded twice, so I was paying old-model latency for new-model quality.
• ESCALATION_EMAIL pointed at an address I'd stopped reading in July.

Four variables, four different origin stories, zero of them intentional.

Then my first fix made it worse

I did the obvious thing: wrote a sync script to copy staging's env onto prod, since staging was the one I'd been keeping current.

It ran for fourteen minutes before checkout failures paged me.

The script copied DATABASE_URL and the Stripe key too — which meant production was briefly pointed at the staging database with a test-mode payment key. Checkout attempts failed in a way that looked like a card problem, so two customers retried with different cards before one of them emailed to ask if my site was broken.

What saved me from something much worse is that the staging database credential is read-only in production's network. Nothing was written. Nothing was charged. Fourteen minutes of failed checkout and a very apologetic email.

So: I attempted to fix a config drift problem by copying config wholesale between environments, and I pushed test credentials into production. The lesson is not "be more careful." The lesson is that any operation that touches production config wholesale is a production change and needs to be treated like one — reviewed, scoped to specific keys, and gated.

What I actually run now

Five things, in the order they earned their place:

1. Fingerprint the resolved config, not the file. At boot and on every job start, log a hash of the sorted key=value pairs after all defaults, env files, and overrides have been applied. If that hash changes between runs, something changed under me and I want to know before a customer does.

import hashlib, os

KEYS = ["TONE_PROFILE", "PRICE_SHEET_VERSION", "REFUND_WINDOW_DAYS",
        "MAX_REFUND_AMOUNT", "MODEL", "ESCALATION_EMAIL"]
SECRET_PREFIXES = ("DB_", "STRIPE_", "OPENAI_", "ANTHROPIC_")

def config_fingerprint(env=os.environ):
    items = []
    for k in sorted(KEYS):
        v = env.get(k, "<MISSING>")
        if k.startswith(SECRET_PREFIXES):
            v = f"len={len(v)}"
        items.append(f"{k}={v}")
    joined = "\n".join(items)
    return hashlib.sha256(joined.encode()).hexdigest()[:12], joined

Twelve hex characters in every log line. Searching for config=9f3c1a2b7e04 across six weeks of logs gives me an exact boundary for when behavior changed. That one grep would have found this in March.

2. Business values get no defaults. A default for a timeout is fine. A default for a price list or a refund window is a decision made by whoever wrote the fallback, at a moment you don't remember, in a context that no longer exists. Those keys are required. If they're missing, the process refuses to start. A service that won't boot is a five-minute outage; a service that boots with a 2024 price list is a two-month one.

3. One source of truth, rendered at deploy. The systemd unit now contains a path and nothing else. Config is rendered from a single versioned file at deploy time, and the render step fails if a required key is absent or if an unknown key appears — unknown keys are almost always typos that silently do nothing.

4. A config diff command in the runbook. Before any deploy, and any time a customer says "this sounds different," run the same command against both environments and read it. It takes four seconds and replaces an hour of assuming.

5. A canary that tests output, not config. After each deploy, three fixed inputs go through the real pipeline and get diffed against golden outputs. It doesn't verify that my config is what I think it is. It verifies that the agent still answers the way I expect, which is the thing I actually care about. Config equality is a proxy; output equality is the claim.

The uncomfortable summary

I built monitoring for exceptions, latency, spend, and error rates. Every one of those signals was green for 63 days while my agent quoted a retired price to customers and told them a policy I don't have. The system was healthy and the product was wrong, and there is no dashboard for that until you decide to build one.

The cheapest version of that dashboard turned out to be a twelve-character hash in a log line and a habit of printing the resolved config instead of trusting the committed one.

I keep the rest of this — the boot banner, the schema with required keys, the golden-output canary and the deploy-day config diff — in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.

If you've been bitten by this, I'd like to know which variable it was. Mine was a refund window I didn't know I'd changed.