DEV Community

xbill
xbill

Posted on

My Demo Script Found a Production Bug on Its First Run: A Tiny Post-Mortem

#testing #ai #python #debugging

I built a demo script to show off a project. It failed on step 2 — and in doing so, it found a real bug that had been sitting in the codebase the whole time, invisible to a fully green test suite.

This is a short technical report on what happened, why the tests missed it, and what it taught me about validating against remote APIs. Total incident cost: about ten minutes. Lessons: worth writing down.

Context

The project is an MCP (Model Context Protocol) server that wraps Google's gemini-3.1-flash-lite-image model — image generation and stateful editing exposed as four tools that any MCP client (Claude Code, a Google ADK agent, a Rust CLI) can call. I've written up the architecture separately; this post is only about the bug.

The relevant detail: the server validates tool arguments locally before calling the API. One of those arguments is thinking_level, the model's latency-vs-quality dial:

# server.py — as originally written
SUPPORTED_THINKING_LEVELS = {"minimal", "low", "medium", "high"}

@mcp.tool()
def generate_image(
    prompt: str, aspect_ratio: str = "1:1", thinking_level: str = "medium"
) -> str:
    ...
Enter fullscreen mode Exit fullscreen mode

Four allowed values, defaulting to medium. The test suite — 10 unit tests, all mocked — passed. Lint passed. The server had been demoed through agents and worked. Ship it, right?

The incident

I wrote a demo.sh that walks through the stack live: list the tools, generate an image, then do a stateful edit. To keep the demo cheap I picked the lowest thinking level:

cargo run --quiet -- generate "a tiny robot chef cooking ramen" 16:9 minimal
Enter fullscreen mode Exit fullscreen mode

First run, step 2:

🔴 Image generation failed: Error code: 400 - {'error': {'message':
"'minimal' is not a supported thinking level for this model.
Allowed values are: low, high.", 'code': 'invalid_request'}}
Enter fullscreen mode Exit fullscreen mode

The live API accepts exactly two thinking levels for this model: low and high. Not four.

Read that against the code above and the real bug jumps out — and it's much worse than a demo flag being wrong:

The server's default was medium. Every live call that didn't explicitly override thinking_level was a guaranteed HTTP 400.

The local validation layer was happily approving values the API would reject, and rejecting nothing that mattered. It wasn't validating the contract; it was validating a memory of the contract.

Why 10 passing tests missed it

The test suite mocks the API client:

@patch("server._get_client")
def test_generate_image_success(self, mock_get_client):
    mock_client.interactions.create.return_value = mock_interaction
    ...
    result = generate_image(prompt="test", thinking_level="medium")
    self.assertIn("🟢 Image successfully saved!", result)
Enter fullscreen mode Exit fullscreen mode

This is a good test — it verifies the server's own logic: argument handling, base64 decoding, file naming, error shaping. Mocked tests are supposed to isolate you from the network, and they do.

But that isolation cuts both ways: a mocked test can never detect that the remote contract changed (or was never what you thought). The mock returns whatever you tell it to, including for requests the real API would reject. My suite effectively asserted "the server correctly forwards medium to the API" — which it did. Correctly forwarding an invalid value is still a bug, just not one visible from inside the mock boundary.

Two things had to line up for this to reach production:

  1. The allowlist duplicated a remote contract. SUPPORTED_THINKING_LEVELS is a local copy of a fact the API owns. Local copies drift — whether because the docs were wrong, the model changed, or the set was written for a different model.
  2. Every prior live test happened to pass a valid value. Agents calling the tools tended to say high (quality) — masking the broken default and the two phantom values.

The fix

Mechanically boring, which is the point — the hard part was knowing:

-SUPPORTED_THINKING_LEVELS = {"minimal", "low", "medium", "high"}
+SUPPORTED_THINKING_LEVELS = {"low", "high"}

-    prompt: str, aspect_ratio: str = "1:1", thinking_level: str = "medium"
+    prompt: str, aspect_ratio: str = "1:1", thinking_level: str = "low"
Enter fullscreen mode Exit fullscreen mode

Plus the sweep that actually takes the time: three tool signatures and docstrings, the server's self-describing get_help text, the README, two articles, the API cheat-sheet doc — and a regression test that locks the new knowledge in:

# The live API only accepts low/high for this model; medium must be rejected
result = generate_image(prompt="test", thinking_level="medium")
self.assertIn("Unsupported thinking level 'medium'", result)
Enter fullscreen mode Exit fullscreen mode

Then verification against the real thing: a live call with pure default parameters — the exact case that was broken — returning 🟢 Image successfully saved!. And because the server ships as a Docker image, a rebuild and push, since the published image contained the bug too.

One design decision earned its keep during all this: the server's tools never raise — they catch everything and return human-readable 🔴 ... strings. The 400 came back as legible text an agent (or a demo script, or me) could read and act on, instead of a stack trace tearing down the MCP session.

Lessons

1. Mocked tests verify your code. They cannot verify the contract. You need at least one test that touches the real API — even a single cheap smoke call. Mine now lives in the demo script and a /verify-stack routine: generate one tiny image with default parameters, because defaults are the values nobody passes explicitly and therefore nobody tests.

2. A local allowlist of remote-owned values is a drift time bomb. If you must pre-validate (it does give agents faster, clearer errors than a round-trip 400), treat the list as a cache of someone else's truth: comment where it came from, and pin a regression test to the values you've observed the API reject.

3. Test your defaults, specifically. The bug survived every live interaction before the demo because humans and agents kept overriding the broken default. f(x) being called a hundred times tells you nothing about f().

4. A demo script is the cheapest end-to-end test you'll ever write. It exercises the happy path a real user takes, with real credentials, against the real API — precisely the layer unit tests can't reach. Mine found a production bug on its first execution, before any audience did. Write the demo before you think you need one; run it in fast mode (DEMO_FAST=1) whenever the API-facing code changes.

5. Return errors agents can read. Tool calls that fail as readable text keep the conversation alive — the calling LLM can see Allowed values are: low, high and retry correctly on its own. That same property is what made this bug a ten-minute fix instead of a debugging session.

Timeline

T+0 demo.sh first run: step 2 fails with HTTP 400
T+1 min Root cause identified from the error body: API allows low/high only
T+4 min Server validation + defaults fixed; regression test added
T+6 min Docs swept (README, articles, cheat-sheet, skill); 10/10 tests green
T+8 min Live verification with default params: 🟢
T+10 min Fixed image rebuilt and pushed to Docker Hub

The demo, incidentally, works great now — the stateful edit produces the same cyberpunk kitchen with a new neon RAMEN sign, pixel-for-pixel continuity intact. But the bug report turned out to be the better story.

Have your own "the demo found it" story? I'd love to hear it in the comments. 👇

Top comments (0)