Your agent demo is rigged (mine was too), so I let the judges write the tests

Alex Meador — Mon, 31 Aug 2026 18:22:48 +0000

Built for the All Things Agentic Hackathon. The project: an autonomous
compliance agent for cannabis track-and-trace, running on Cloud Run with
Gemini 3.5 Flash. This post is about the test harness, because it turned out to be the most interesting part.

The objection every agent demo deserves

Every hackathon agent demo has the same silent flaw: the person who built the agent also built the test. Of course the agent passes. The demo world was constructed, consciously or not, around what the agent is good at.

My project had this problem in an acute form. It reconciles a cannabis
facility's inventory against METRC - the state seed-to-sale system that is
legally the record of what exists - and decides whether packages can be
released under Kentucky's testing regulation, 915 KAR 1:110. There's no way to demo against production METRC (vendor API access requires a
training-and-agreement process), so the whole thing runs against a fixture
service I wrote: a fake METRC that serves the documented v2 API shapes from Firestore.

So the pitch is "my agent correctly polices a world I invented." Rigged, by construction.

The fix: make the regulation the ground truth, then hand over the keys

Two design moves converted the objection into the project's strongest
property.

First: the agent is never the reference. The regulation is. The required test panels from 915 KAR 1:110 Section 2 are stored as structured data - ten analyte categories for finished product, three for an in-process batch, each with its citation. Whether a package's testing is complete is a set difference computed in code, before any model is consulted. Anyone holding the same METRC record and the same regulation text can check every decision.

Second: judges can construct worlds I never wrote. The fixture service
takes a compact spec:

{
  "packages": [
    { "item": "1oz THC Tincture", "category": "Tincture",
      "tests": "finished", "omit": ["mycotoxins"] },
    { "item": "Bulk Live Resin", "category": "Concentrate",
      "tests": "production", "fail": ["solvents"] }
  ]
}

One POST expands that into a full world - realistic METRC package records,
lab results with plausible levels, transfer manifests - served over the same API surface as the seeded demo. Then you trigger a cycle and grade the agent yourself, against the regulation, on a case whose answer I never knew in advance.

What happened the first time someone (me) actually did this

The first custom scenario ever run against the deployed system found three
bugs in an evening. None of them were the model being wrong. All of them were the model being right about something I got wrong.

Bug 1: my "passing" data was physically absurd. The generator gave every product the same passing values, so a concentrate carried 1.2% total THC and a solventless rosin carried a butane result. Gemini escalated both as implausible - correctly. A rosin with residual butane on file is what a wrong-matrix lab result actually looks like. Lesson: generated test data must be boring. If your fixtures trip the agent's plausibility instincts, every scenario drowns in false alarms and the agent looks paranoid instead of careful.

Bug 2: my records were internally contradictory. A spec that failed a
test still stamped the package TestPassed in the METRC state field. The
model refused to act on the contradiction and asked for a human. Right again. The state-vs-results contradiction is a legitimate scenario - but it should be constructed on purpose, not seeded by accident.

Bug 3 (the one that mattered): escalation left a failing package
movable. My rules layer had a principle I was proud of: escalation performs no write, because an agent that escalates to a human and then acts anyway hasn't escalated. The custom scenario exposed the flaw: the model escalated a failing-solvents record, no write happened, and a package with a failing required test stayed movable in the system of record. A real compliance officer would never do that - you freeze the package pending review. The fix distinguishes ambiguity (escalate, touch nothing) from hard evidence (a missing or failing required test: protective hold first, then escalate). Freezing state isn't deciding the outcome; it's making sure nothing moves while the human decides.

That distinction is now enforced in code, covered by tests, and I would not have found it by running my own scenarios - because my own scenarios were built around the rules I already believed in.

The generalizable bit

If you're building an agent that acts on a system of record, the harness
pattern that made this trustworthy:

Store the rules as data, diff in code, let the model explain. The safety-critical invariant should never depend on a language model's reading - models can be talked out of anything. Compute the finding deterministically; use the model for judgement, prose, and noticing what state fields can't express.
Publish the world-builder, not just the world. A demo world proves you can pass your own test. A generator proves you'll accept anyone's.
Make bad specs loud. In my generator, an omit pattern that matches no test is an error, not a silent no-op - a judge whose scenario quietly tests less than they intended has been misled by the harness itself.
Include the boring cases. A fully compliant world where the agent correctly finds nothing is a required fixture. An agent that only ever raises alarms has demonstrated nothing.

The repo - agent, MCP servers, fixture service, generator, and the findings log with everything above in more detail - is MIT-licensed. The agent runs hourly on Cloud Run with nobody watching; the demo buttons exist for judges, but last night's log is the real evidence.

Stack: TypeScript, LangGraph, Gemini 3.5 Flash via the Google Gen AI SDK on Vertex AI, MCP for all tool access, Cloud Run + Cloud Scheduler + Firestore.

DEV Community: Alex Meador

Your agent demo is rigged (mine was too), so I let the judges write the tests

The objection every agent demo deserves

The fix: make the regulation the ground truth, then hand over the keys

What happened the first time someone (me) actually did this

The generalizable bit