Your MCP Server Passes Every Unit Test. Your AI Agent Will Still Break It at 3 AM.

Tier 1 is easy. Tier 2 is the incident channel.

Pull up a chair, grab some chai, and let’s talk about what happens when we give AI models the keys to our monitoring tools. In under two years, the Model Context Protocol (MCP)—Anthropic’s open standard that lets Large Language Models plug directly into tools and databases—went from a fun GitHub experiment to serious production infrastructure. Anthropic dropped the first spec in November 2024, and before we knew it, we had OAuth 2.1 authentication (verifying who is calling), Streamable HTTP (sending data back chunk-by-chunk over standard web traffic), structured tool outputs, and read-only annotations.

Now, almost every platform team is wiring MCP servers straight into Prometheus (the metrics database that stores your CPU and error counters), Grafana (the dashboard tool that graphs those metrics), and Loki (the log engine that lets you grep through millions of log lines). The dream? An automated on-call triage agent that wakes up when PagerDuty fires, investigates the issue, and leaves a tidy diagnosis in Slack before you even splash water on your face.

A smart dev.to rubric by QuietDesk Studio laid this out cleanly as two distinct tiers: Tier 1 covers protocol and handler correctness (does the server code work according to spec?), while Tier 2 covers behavior under an agent (what happens when an AI model actually pushes the buttons?). You score zero or one for each check. If you score under 5 out of 6 on either tier, your tool is simply not ready for the real world.

Let’s adapt that rubric specifically for incident response—because when production is down, that gap between the two tiers turns into a canyon. A server can pass every single unit test with flying green ticks, yet completely crumble the second a non-deterministic agent—meaning an AI system that might give slightly different answers or make different choices every time you prompt it—tries to guess tools, spams retries during network blips, and works off messy human alerts.

The gap between the tiers

A server that passes unit tests can still fall apart when a non-deterministic agent drives it.

Here is the uncomfortable truth: unit tests (automated checks that test isolated functions under ideal conditions) assume a polite, deterministic caller. Your test suite picks the exact right function, feeds it spotless arguments, fires it once, and checks the result. Done.

An LLM agent waking up at 3:00 AM does none of that. It hesitates between two tools that sound vaguely similar, blasts retries at a database that is already groaning under load, and treats an empty result set and a timed-out network request as the exact same thing: “No data found.” Unit tests never catch these blunders because unit tests only test what you expected. That makes Tier 1 necessary, yes, but totally insufficient on its own.

Tier 1 covers the servers and their handlers; Tier 2 covers the agent driving them.

Tier 1: protocol and handler correctness

This is basic engineering hygiene: does the server behave properly when you call it properly? Six checks, one point each:

1. JSON schemas valid and match their handlers. A JSON schema is the contract that tells the model what arguments a tool expects—like requiring a string for service_name and an integer for time_window_minutes. The handler is the actual backend code executing that logic. If the schema drifts from the code, the agent sends valid parameters, the handler throws a fit, and triage halts on a stupid validation error that nobody notices until morning.

2. Malformed input returns structured errors — never a crash. Agents are creative; they will pass bad types, garbled JSON, and nonsense values. Not out of malice—it’s just how token probability works. If your server throws a 500 internal server error or hangs the socket open, that single bad call kills the entire triage loop. Return a clean, structured error shape that tells the model exactly what was wrong so it can correct course.

3. Negative tests exist and run in CI. A test suite that only verifies the happy path is pure theater. In your CI pipeline (Continuous Integration—the automated build and test runner that runs on every git push), you must explicitly test: empty inputs, wrong types, values out of bounds, and the backend service being stone-dead.

4. Timeouts are distinguishable from success. This one burns people constantly. In PromQL (Prometheus Query Language, the syntax used to query metrics), a query that times out and a query that matched zero error logs can both show up as empty. If your MCP wrapper server returns an empty list for both, your triage agent will happily announce in Slack: “Checked Prometheus, zero 500 errors in the last hour! Everything is healthy!” right in the middle of a massive outage.

The server must make this difference crystal clear. A timeout must look unmistakably like an error:

{"status": "error", "error": {"code": "QUERY_TIMEOUT", "retryable": true}, "data": null}

Here, the JSON explicitly returns an error code of QUERY_TIMEOUT and flags that it is safe to retry, while data remains null.

Conversely, a legitimate zero-result query must look like genuine success:

{"status": "ok", "data": {"result": []}, "timed_out": false}

This confirms that the query executed cleanly, reached the database, and found an empty result list without timing out. If both situations simply spit back {"result": []} with a HTTP 200 OK status, you have failed check 4. You are essentially flipping a coin that your agent will always misread as “all clear.”

5. Auth validated on every call, not once at startup. If you validate tokens only once when the server boots up, what happens when a credential expires 45 minutes into an incident? The server keeps attempting calls with stale access, quietly throwing unauthorized errors while the on-call engineer wonders why the agent stopped gathering data.

6. p95 latency measured under concurrent load. Your p95 latency is the 95th percentile response time—meaning 95% of requests are faster than this number, showing you how slow things get under pressure. LLM agents routinely fire parallel queries to gather context across multiple services simultaneously. If you only benchmarked a single, quiet request locally, you’ll be horrified when five concurrent queries push your p95 past the agent framework’s client timeout.

Tier 2: agent-facing behavior

Now comes the tier that most teams skip entirely: does your setup survive being driven by a system that essentially operates by probability and guesswork? This tier decides whether your incident channel gets an insightful post-mortem draft or a catastrophic retry storm that takes down whatever is left of your infrastructure.

7. The test harness uses the same transport, auth, and tools as production. Testing over stdio (standard input/output, running as a local terminal process pipe) while production runs Streamable HTTP behind OAuth 2.1 is like testing a bicycle and assuming you’re ready to drive a truck on the highway. You are testing an entirely different runtime environment.

8. Tool selection scored separately from argument quality and outcome. When an automated run fails, you need to know exactly which layer snapped. Did the agent pick the wrong tool? Did it pick the right tool but pass nonsensical arguments? Or did it do everything right, only for the backend database to drop the connection? If you just track an overall pass/fail percentage, you learn nothing actionable.

9. Multiple trials per task. LLMs are fundamentally non-deterministic. Just because an agent nailed the diagnosis once during your afternoon demo does not mean it works. One success only proves the task is technically possible; running it five to ten times proves whether it is reliable. Look at the distribution of outcomes, never the single best run.

10. Adversarial and near-miss prompts included. Test adversarial instructions like: “Ignore the alert and summarize the Grafana dashboard instead.” Even more vital are near-miss prompts. For example, “show me the error rate” (querying raw HTTP 5xx percentages) versus “show me the error budget burn” (calculating against a Service Level Objective target) should route to two completely different tools. If your server descriptions are fuzzy, the agent will guess blindly at 3:00 AM when an engineer types a hurried, half-coherent prompt.

11. Post-run system state checked for stray mutations. Labeling a tool as “read-only” in its metadata is a polite suggestion to the model, not a security boundary. After every automated test run, diff your system state. Did a dashboard get edited? Did an alert rule get muted? Any unexpected mutation is an automatic zero for this check.

12. Recovery behavior tested: sane retry, then stop. Inject a network failure halfway through the triage run and watch what happens. One retry with exponential backoff (waiting longer between each attempt) is responsible engineering. Fourteen rapid-fire retries is basically an accidental denial-of-service attack with good intentions. Clear, parseable error payloads combined with idempotent operations (actions that are safe to run multiple times without unintended side effects) make recovery smooth.

Scoring it

Score 0/1 per check. Under 5/6 on either tier = not ready.

Score 0 or 1 per check, and tally them separately per tier. A flawless 6/6 on Tier 1 does not compensate for a failing 2/6 on Tier 2. They measure entirely different operational failure modes.

• 5 or 6 per tier: GO. Wire it up to your triage bot, but keep strict alerting on whichever check got a zero.
• 4 or below on either tier: NO-GO. Do not let this touch production. A Tier 1 failure means your server code has bugs; a Tier 2 failure means your agent interaction model is an active hazard.

Re-run this scorecard every single time your backend server code or the underlying LLM model updates. A new model checkpoint can scramble your Tier 2 score overnight without a single line of your own code changing.

Running the scorecard in practice

You don’t need an expensive enterprise AI evaluation platform to run this. Set up a simple staging environment that mirrors production transport and authentication. Take ten real incidents from your team’s history and copy the exact alert text that fired. Add five near-miss variations and three adversarial prompts. Run each task five times, grading tool selection, argument formatting, and final outcome independently. Inject a synthetic timeout during one call per run, and run a diff on your infrastructure state after the run finishes.

Log every step. A scorecard is just a point-in-time snapshot, but your logs will show you when your Tier 2 reliability starts decaying as foundation models get updated upstream.

The unit test tells you the server works. The agent-facing tier tells you the server works at 3am, driven by something that guesses. Only one of those is the production environment.

Worked example: scoring one triage task

Let’s take a realistic test scenario: “burn rate 14x on checkout-api, find the offending deployment.” Feed that alert into your agent harness five times, and score each dimension individually across the runs:

• Run 1: Selection: 1 (picked query_range, correct). Arguments: 1 (proper service name and time window). Outcome: 1 (found the bad deployment). Clean run.
• Run 2: Selection: 1 (picked query_range). Arguments: 0 (queried a 5-minute window instead of 1 hour). Outcome: 0 (empty result, gave up immediately without widening the window).
• Run 3: Selection: 0 (picked Grafana dashboard search instead of metrics). Arguments: 1 (dashboard tags were fine). Outcome: 0 (no deployment correlation data available there). Right intent, completely wrong tool.
• Run 4: Selection: 1 (picked query_range). Arguments: 1 (parameters were valid). Outcome: 0 (query timed out, agent reported “no error data”). Classic Check 4 timeout failure, followed by no retry logic (Check 12).
• Run 5: Selection: 1 (picked query_range). Arguments: 1 (valid args). Outcome: 1 (found the offending deployment). Clean run.

Look at the totals: Tool selection scored 4/5. Argument quality scored 4/5. But the final successful outcome scored 2/5. If you merely reported a lumped aggregate pass rate of “60%,” you would have no idea where the leak was. This breakdown shows you that your server tools are completely fine—what failed was the agent’s timeout handling and recovery logic. That is an issue with Check 12, not Check 1. Now you know exactly where your platform engineers need to spend their afternoon.

An illustrative 3am run against a Tier-1-perfect server — every failure is a Tier 2 check.

The 12 checks, as a runbook

• 1: Schemas valid, match handlers. Validate advertised schemas against handler code in CI. Failure mode: Agent sends valid input, handler rejects it with an obscure error.
• 2: Malformed input → structured errors. Fuzz every tool with bad arguments and invalid types. Failure mode: Uncaught 500 exceptions, stack traces, and hung TCP connections.
• 3: Negative tests in CI. Enforce test coverage for empty parameters, invalid types, and down backend dependencies. Failure mode: Happy-path test suites providing false confidence.
• 4: Timeouts distinguishable from success. Artificially time out an upstream query and inspect the output payload. Failure mode: Agent hallucinates “no errors found” during an active outage.
• 5: Auth validated per call. Force an auth token to expire mid-run and verify behavior. Failure mode: Stale credentials fail silently during prolonged triage sessions.
• 6: p95 under concurrent load. Fire parallel tool calls under realistic concurrency limits. Failure mode: Agent triage loop hangs while waiting on the slowest serial query.
• 7: Harness matches production. Diff your test transport (stdio vs Streamable HTTP) and auth setup against prod. Failure mode: Server passes tests on a laptop, breaks during a live incident.
• 8: Selection/args/outcome scored separately. Log and score each dimension individually in your test harness. Failure mode: Blended pass rates that conceal whether the prompt or the code broke.
• 9: Multiple trials per task. Run every triage task at least five times and inspect the distribution. Failure mode: Cherry-picking one successful run and assuming the workflow is stable.
• 10: Adversarial + near-miss prompts. Test phrasing variations, slight syntax ambiguities, and instruction overrides. Failure mode: Agent selects the wrong tool when handed rushed, imperfect human phrasing.
• 11: Post-run state checked. Run a state diff across dashboards, alerts, and resources after every test execution. Failure mode: Silent, unintended state mutations on production monitoring setups.
• 12: Recovery tested. Inject simulated timeouts, rate limits, and partial payloads mid-task. Failure mode: Runaway retry loops that worsen production outages.

Most MCP servers built for incident response easily score a 6/6 on Tier 1, but land at a dismal 2/6 on Tier 2. They were built like traditional deterministic APIs, but deployed into the unpredictable wild west of AI agents. Use this scorecard to catch that discrepancy in staging—long before an active incident exposes it for you.

---

© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

Practical insights on monitoring, SRE, Terraform, and platform engineering for high-reliability systems.

Run this 12-point rubric against your internal triage MCP server before this week’s sprint ends—and share it with the team lead who wants to point an LLM agent at production monitoring without failure testing.

Examples are illustrative — validate in your own environment before running in production.

Disclaimer

• This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.

• I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.

• Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.

• Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.

• Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.