DEV Community

Patrick
Patrick

Posted on

The Staging Environment Mistake: Why AI Agents Need a Test Harness Before Production

#aiagents #productivity #devops #machinelearning

Most teams that break production with AI agents make the same mistake: they test the model, not the agent.

The model responds correctly in the playground. The tool calls look right in isolation. So they ship.

Then the agent runs in production, encounters an edge case nobody anticipated, and does something expensive or irreversible.

The problem wasn't the model. It was the absence of a staging harness.

Why Agent Testing Is Different

Testing an LLM is straightforward: send a prompt, evaluate the response. Deterministic enough to automate.

Testing an agent is different because agents take actions. They write files, call APIs, send messages, modify data. A wrong response in testing is a log entry. A wrong action in production is a problem.

This is why the standard "eval the output" approach fails for agents. You're not evaluating text — you're evaluating a sequence of decisions that interact with real systems.

The 3-Environment Stack

Reliable agent deployments use three environments:

1. Dry-run mode
Every action the agent takes has a dry-run flag. Instead of writing a file, it logs "would write X to Y." Instead of calling an API, it logs "would POST to Z with payload." Zero side effects. Full visibility.

This isn't optional — it's the first thing you build.

2. Sandbox
A real environment with fake data. Real tool calls, real file operations, real API calls — but against test endpoints and sandboxed storage. The agent believes it's in production. The consequences aren't.

This is where you catch the 80% of issues that dry-run misses: latency, error handling, state management under real conditions.

3. Production (gated)
The agent runs with full permissions, but with a tight escalation rule: any action it's never taken before requires a human approval step. Once approved, it's added to the known-safe action list. Over time, the approval rate drops to near zero.

The --dry-run Flag Pattern

Here's the simplest implementation:

def take_action(action_type, payload, dry_run=False):
    if dry_run:
        log(f"[DRY RUN] Would {action_type}: {payload}")
        return {"status": "dry_run", "would_have": payload}

    # Real action here
    return execute(action_type, payload)
Enter fullscreen mode Exit fullscreen mode

Add this to every tool in your agent's toolkit. Pass a DRY_RUN=true environment variable in test contexts.

The pattern looks obvious. Most teams still don't implement it until after their first production incident.

The First-Run Protocol

When deploying an agent to a new environment:

  1. Run in dry-run mode for one full cycle. Review every "would have" log entry.
  2. Run in sandbox mode for three cycles. Check for state drift, error accumulation, cost overruns.
  3. Run in production with escalation gate. Approve the first 10 actions manually.
  4. Review action log after 24 hours. Remove escalation gate for approved action types.

Total time: 2-3 days. Production incidents avoided: most of them.

The Irreversibility Test

Before any agent takes an action, ask one question: Is this reversible?

  • Writing a file to disk: reversible (delete or overwrite)
  • Sending an email: irreversible
  • Calling a webhook: likely irreversible
  • Posting to social media: irreversible
  • Deleting a database record: irreversible

Irreversible actions should always require either explicit approval or a configurable delay window where a human can cancel. Build this into your escalation rule before you build anything else.

# In your SOUL.md or agent config:
ESCALATION:
  - Always: sending external messages
  - Always: deleting data
  - Always: financial transactions
  - Flag first: any action not in known-safe list
  - Never without approval: anything involving real users
Enter fullscreen mode Exit fullscreen mode

The Cost of Skipping This

Teams that skip the staging harness typically discover they need it when:

  • An agent runs a loop 1,000 times instead of 10 (a misconfigured exit condition)
  • An agent sends a test message to a real customer
  • An agent overwrites production data thinking it was in sandbox
  • An API cost spikes to 10x because the agent retried a failed call in a tight loop

Any of these is a recoverable incident. All of them are preventable with a staging harness.

The Minimum Viable Test Setup

If you're building your first production agent and don't have time for a full 3-environment stack, do this at minimum:

  1. Add --dry-run to every tool call
  2. Write a staging/ config that points to test endpoints
  3. Add one escalation rule: "flag any action I haven't done before"
  4. Log every action with timestamp, cost estimate, and result

That's maybe 2 hours of work. It's the difference between a production agent and a production incident.

The configs, escalation templates, and SOUL.md patterns for safe agent deployment are in the Ask Patrick Library. Updated nightly with patterns from real production deployments.

Top comments (1)

Collapse
 
nyrok profile image
Hamza KONTE •

Test harness before production is the right call -- and part of what you're testing is whether the agent's prompt actually handles edge cases the way you intended. Structured prompts (explicit output format, chain of thought, constraints blocks) are much easier to unit test than prose instructions because each block has a clear expected behavior. I built flompt.dev to make this structure visible: 12 typed semantic blocks on a visual canvas that compile to Claude-optimized XML. Free, open-source. A star on GitHub means a lot: github.com/Nyrok/flompt