DEV Community: JWL AI

Agents for Humans: the fixture that hid four bugs from me

JWL AI — Mon, 14 Sep 2026 07:12:36 +0000

I built an enterprise agent for mine operations and gave it a deterministic model provider so the demo would replay identically. That decision was right, and it hid four bugs from me for the entire project.

This is about what happened the day I first pointed it at a real model.

The setup

CAIRN correlates fragmented signals from a mine shift into one incident, has four specialist agents analyse it in parallel through a bounded Strands Agents graph, and produces three recovery options. Then it refuses to act until a person signs.

To make it reproducible I wrote a Strands Model provider returning deterministic output from fixtures. No credentials, no network, byte-identical replay. CI ran against it. A thirteen-case evaluation suite ran against it. Everything was green.

Then Nova Lite ran the same graph

The graph passed. All five nodes satisfied their typed contracts. Every specialist called tools before answering. Thirteen seconds, 30,253 input tokens.

And the output was unusable.

key                    option_1             (the UI selects on recover_tonnes)
recommendedScenarioId  scenario_id_1        (matches none of its own options)
requiredApprovals      operations_manager   (not a role the policy table knows)
estimatedTonnesDelta   -1200, -1500, -1000  (every option a loss)

Pydantic validated all of it. Every field was present and correctly typed.

Why the fixture hid it

Look at what the contract actually said:

key: str  # protect_safety | recover_tonnes | preserve_equipment

The legal values were in a comment. The model never sees comments. It sees the JSON schema Strands generates for its forced tool call, and that schema said string.

requiredApprovals was list[str], so any plausible role name validated. recommendedScenarioId was str, so it could point at nothing — and did. The tonnes sign convention was written down nowhere at all, so the model read the delta as a loss against an undisrupted plan while the fixture meant recovery against doing nothing. Both readings are defensible. That is the problem.

Each of those was latent and green for weeks, because the fixture always wrote the right value. My tests asserted against a provider that happened to behave.

The fix, and the thing worth generalising

Four constraints moved out of comments and into the schema: two enums, a model validator for referential integrity, and a field description stating the sign convention. Then five tests that assert the schema rather than a provider, so they need no credentials and hold for any model. All five fail against the old contract.

Anything whose correctness depends on a provider must be asserted against the schema, not against the provider.

A comment is not a constraint. It documents intent to humans and enforces nothing on the machine that actually consumes your contract.

A second thing the fixture hid

While I was in there, I asked why the risk agent made two tool calls when the others made three. The answer was boring — its plan is two. But checking turned up something else: three of my prompts advertised tools that do not exist.

## Allowed tools
`get_weather_window`, `get_active_permits`, `get_evidence`, `get_site_context`

get_active_permits exists nowhere in the codebase. The scenario planner's prompt offered three tools while its allow-list is empty by design, and one of those three was fictional too.

The fixture provider follows a tool plan and never reads the prompt. Only a real model reads the prompt, tries to call a tool that is not there, and gets cancelled by the hook. The guard held — but the attempt is wasted and the trajectory polluted.

Two tests now assert that a prompt's advertised tools equal its node's allow-list, and that every advertised tool is registered.

What I would do differently

Keep the deterministic provider. It is the right call: anyone can clone the repository and get the same result with no account, and CI has a hermetic gate. I would not give that up.

But run a real provider early, and treat the first run as a contract review rather than a smoke test. Every defect it found was a place where I had written the rule down somewhere the machine could not read it.

Both modes now run the same graph and the same governance. Swap the brain; the safety envelope does not move. That is a stronger claim than either mode alone, and I only earned it by running both.

CAIRN is built with Python, the AWS Strands Agents SDK, and Amazon Bedrock. Everything in it is synthetic and simulation-only: no mine system, PLC, SCADA, dispatch, ERP or CMMS is connected, and there is no control path to operational technology by design.

Code: github.com/jwlai-cloud/cairn

The difference between "mentioned" and "answered"

JWL AI — Mon, 31 Aug 2026 07:58:08 +0000

I built this for the All Things Agentic Hackathon (August 2026). Code:
github.com/jwlai-cloud/intake.
Set published: true in the front matter when you post it.

A community nurse has ninety minutes and a form she is legally required to
complete. She asks about falls in the last twelve months. The answer is:

"Oh, I've had a couple of wobbles."

Every AI scribe on the market ticks that item. It was mentioned. It was never
answered. She finds the gap that evening, at her desk — and now it needs a phone
call, a guess, or a second visit. The person who knew the answer was sitting
three feet away an hour ago.

That gap is the entire product.

Why every competitor ticks the box

I looked at what's shipping. Microsoft Teams' Facilitator marks a topic covered
"once the discussion for that topic has started". Balto ticks when an item is
mentioned. Otter's Live Assist checks off objectives. None of them adjudicate
whether a required item actually received a real answer, and none of them gate
the output on it.

That's not laziness. Mention-detection is a much easier problem. Deciding
whether "a couple of wobbles" satisfies a form's requirement for "the number
of falls and the circumstances of the most recent" is a judgement call, and
judgement calls are where LLM products quietly go wrong.

So the whole project reduces to one question: can you make that judgement
reliably enough to gate a report on it?

Build the judge first, and measure it

The first thing I built wasn't the UI. It was an eval harness.

eval/ holds 47 labelled cases — for each required item, answers that should
count and answers that shouldn't. It scores the adjudicator against the live
service and exits non-zero if any answer labelled insufficient was marked
sufficient. That failure is the one that destroys the product, because it's
indistinguishable from the mention-level behaviour I'm claiming to beat.

The bar is asymmetric on purpose. A false insufficient costs one extra
question. A false sufficient is a silently blank field in a legal document. So
the prompt's first rule is: default to insufficient — if you're weighing it up,
it's insufficient.

Twelve of those cases are adversarial, and writing them was the single
highest-value hour of the build. They found a real bug immediately:

"Three falls, and the last was in May on the stairs."
"No, hang on — I'm thinking of my sister. I've not actually fallen myself,
not that I can bring to mind."

The adjudicator read the retraction as a clean nil return and ticked the
item. The record now held a contradiction settled only by a hedge, which is a
practitioner's call, not the agent's. The fix was a rule: the later turn governs
only when it is itself a clear answer or a clear refusal.

I'd never have found that by hand-testing. A set that scores 100% on its first
run can't tell you anything.

The architecture, and the two decisions worth defending

Google ADK 2.6.2 on Cloud Run, calling Gemini 3.6 Flash on Vertex AI,
with session state in Firestore. One audio chunk in, one bounded turn out:

transcribe → route → adjudicate (fanned out) → coach

The slot state is the state, not the transcript. The naive design
accumulates the conversation and re-asks the model "what's still missing?" every
turn. A forty-five minute interview becomes tens of thousands of growing tokens
— expensive, and degrading as it grows. Instead each call gets the open items, a
fixed-size struct of current values, and the new audio. A three-hour interview
costs the same per chunk as a ten-minute one.

Adjudication is one isolated call per open item, not one big prompt. Three
reasons, in order: a wrong verdict on one item can't corrupt another; each item
is separately scoreable by the eval; and k concurrent calls cost about one
call's wall time. That last one surprises people — fanning out is usually the
expensive choice, and here it's close to free.

Between them sits a router. Without it, every open item was adjudicated against
every chunk, and each independently decided a vague remark was relevant to it —
so "a couple of wobbles" attached itself to mobility, memory and low mood as
well as falls. One cheap classification call first cut items touched from seven
to two, and cut cost with it.

The rules that are enforced by types, not prompts

The agent never authors domain content. It tracks coverage against a
human-authored form and quotes the span it relied on. It says "item M14 has no
recorded answer", never "this may indicate falls risk". That's not a prompt
instruction — the coach's output schema has fields for a question and for quotes
and no field an answer could go into. It structurally cannot suggest one.

Behavioural evaluation found the hole in that anyway. The schema forbids an
answer field; it doesn't forbid an interpretive label. The agent had emitted
a highlight titled "Formal decline to answer alcohol question" — a
characterisation, not a quote. Titles are bare noun phrases now, and that exact
string is a test case.

No interviewee identity, ever. Sessions are scoped to a job, not a person.
Persistent memory is scoped to the practitioner.

The honest version of that claim is narrower than the slogan. Recorded answers
are verbatim quotes, and a real interviewee says "my daughter Sarah drives me on
Mondays". Redacting that would break adjudication, so identity is a
retention answer, not a redaction one — quotes live in the session document,
never in logs, and go when the session goes.

That last part was a real bug. A Vertex error response echoes the offending
request, and the adjudicator's request body is the transcript — so a malformed
chunk was writing interviewee speech into Cloud Logging, which outlives the
session document. Logs now carry the exception type only.

What the agent learns, and what it refuses to

It gets better at helping one practitioner across her interviews. It learns
nothing about the people she interviews.

Two things, both about the professional: a question phrasing that closed an item
on the first ask, and the item ids whose highlights she keeps dismissing. From
her second interview the coach offers back a wording that worked, and stops
proposing chips she's binned — while still asking the required question, because
muting a suggestion must never mute an obligation.

The version that would demo better is the one it refuses to build. "People like
this one usually under-report falls" would be useful and would permanently
break the privacy property. The line between them is the line between a question
and an answer: the agent's own composition may cross sessions, a person
describing their own health may not. A guard rejects any candidate that's
quoted, isn't a question, or has a first-person subject — and a test feeds it
five samples of real interviewee speech and fails if any survives.

Nothing is ever silently blank

Before a report is produced, every required item resolves into exactly one of
three states:


Answered	with the transcript span it was drawn from
Declined	formally recorded, and only where the form permits it
Escalated	the agent drafts the follow-up itself and routes it

The gate is a router, not a wall. A copilot that only says no is one
practitioners switch off. When it refuses, the response carries what's
outstanding, what's missing from each, and whether a decline is even permitted.

The escalation is the part I'd point at. Given an unresolved item, the agent
writes what's still not recorded in the form's own terms, why it couldn't be
closed, and picks a destination from a closed list. "Home access and hazards ·
not recorded during the visit → Occupational therapy queue." Unprompted
judgement, a real artifact, filed.

Things that cost me time

GOOGLE_GENAI_USE_VERTEXAI=TRUE is mandatory for ADK. Without it, ADK builds its own AI Studio client and dies with "No API key was provided" — even though a hand-built genai.Client(vertexai=True) in the same process works.
SequentialAgent is deprecated in 2.6.2, and it emits a container event with no content. agents-cli eval generate rejects any content-less event, so ADK's deprecated orchestrator is incompatible with ADK's current eval tooling. Then I measured the replacement: the graph Workflow emits Event(output=…, content=None) and fails identically. A four-line custom BaseAgent is the only one of the three that can be evaluated.
I had the reason backwards for three days. My notes said the graph migration was blocked because "Workflow cannot take an LlmAgent as a sub-agent". The warning says the reverse — a Workflow can't be nested inside an LlmAgent. Both are BaseNode subclasses and compose fine.

The general lesson, for an SDK this young: read signatures off the installed
package with inspect, not off documentation. Every API claim in this project
was verified that way, and the one time I trusted a note instead, it was wrong.

The bug I found by finally doing the obvious thing

Two days before the deadline I asked someone to try the deployed app with a real
microphone. Nothing visibly happened.

Chunks were arriving, HTTP 200, the ADK pipeline was running — and every turn
finished in about a tenth of a second having done nothing. A real turn takes six
seconds.

The transcriber labels each turn practitioner or interviewee, and
adjudication only looks at interviewee turns. One person testing alone is a
single voice, and the model reasonably labelled it practitioner. Every chunk
was discarded. The screen sat inert — which is exactly what a judge trying it
alone would have seen.

I'd tested the API with curl, the text path, and an automated browser capture.
But that capture runs Chromium with --use-fake-device-for-media-stream. I had
verified a proxy for the product and called it the product.

The fix is one instruction, and where it went matters. My first attempt
relaxed the filter in the adjudicator, and it immediately broke a test asserting
that a nurse restating an answer must never close an item. Same input, two
opposite correct answers — the adjudicator can't tell a lone tester from a
professional summarising. The transcriber can: it's the only stage that hears
how many people are in the room. So it labels a lone voice as the interviewee,
and the downstream guarantee is untouched.

Where it stands

47 labelled cases, 100% precision on sufficient — it has never once ticked
an answer a human labelled insufficient. Accuracy moves between 45 and 47 across
runs because the adjudicator is occasionally stricter than a labelled case;
every one of those misses is a false insufficient, which costs a question and
never a wrong tick. 133 backend tests. A behavioural eval over the pipeline
scoring 18/18, graded by deterministic code rather than an LLM judge — asking a
model "is this quote verbatim?" is slower, costs money, and is worse than in.

A second template — insurance loss adjusting — runs on the same engine with no
code change, which is the test of whether the vertical is really just config.

What I'd build next is per-user identity. Access control today is a capability
model: one shared key, and session ids that are 128 bits of randomness. It
holds, but it stops holding the moment an id reaches a log or a shared screen.

How a Negotiating Agent Society Out-Plans a Single Scheduler

JWL AI — Fri, 03 Jul 2026 08:44:01 +0000

Turn a five-way scheduling trade-off into a debate you can audit: five advocate agents argue, a charge-nurse referee rules, and the plan beats a single agent on the same scorer.

The scariest scheduling failures are the confident ones.

You hand one LLM a clinic's week — 56 patients, 43 slots, three nurses — and ask it to book the follow-ups. No errors. No complaints. It returns a clean, plausible schedule in a single pass. But is it a good plan, or did it quietly sacrifice something that mattered?

Score it and you find out. High-acuity patients: all seen. Continuity of care: collapsed — patients bounced to whichever nurse had a gap. Overdue follow-ups: slipping. The agent anchored on the one objective that's easy to name and let the messy ones rot — and it never told you it made that trade.

A quiet objective is not a satisfied objective. Usually it's the opposite.

This is a case study in fixing that with a multi-agent society — and it won't stay abstract. Every number comes from one real system: RehabPanel, a rehab-scheduling agent built for the Qwen Cloud hackathon (Track 3), running on real Qwen models and deployed live on Alibaba Cloud.

The problem: one agent collapses the trade-off

Five objectives fight for the same slots — clinical acuity, overdue windows, continuity with the primary nurse, hard capacity, patient preference. They genuinely conflict: seat the sickest patient and you may break someone's continuity; honor a preference and you may bump an overdue visit.

A single agent resolves that tension inside one prompt, invisibly. You get an answer, not an argument. And you can't audit an answer.

The fix: make the conflict explicit and negotiated

RehabPanel replaces the one planner with five advocate agents, each obsessed with exactly one objective, plus a charge-nurse referee who brokers between them. It's a LangGraph state machine — draft → critique → negotiate → arbitrate, looping until nobody's still objecting.

Each round:

All five advocates object to the current plan, in their own words — in parallel.
The loudest objection's advocate proposes a concrete swap.
The rest split into FOR / AGAINST coalitions by whose objective the swap helps or hurts.
The referee rules on a fixed priority ranking (capacity ≻ acuity ≻ overdue ≻ continuity ≻ preference) — and explains the ruling in plain language, appending it to a conflict ledger.

Nobody scripts that explanation. Mid-run, the referee writes things like: "approved because it improves preference without violating any higher-ranked objective, and no competing acuity, overdue, or continuity claim contests this slot." That ledger is the whole point — a single agent never shows this work.

One round: five objections, the coalition split, and the referee's prose ruling appended to the conflict ledger.

The whole thing is a LangGraph state machine with one explicit exit:

flowchart TB
    DR["node_draft<br/>acuity-first skeleton — no LLM (or a warm seed)"]
    CR["node_critique<br/>all 5 advocates object — in parallel"]
    CD{"hot objection<br/>&amp; round &lt; cap?"}
    NG["node_negotiate<br/>top objector proposes · FOR/AGAINST coalitions"]
    AR["node_arbitrate<br/>referee brokers on priority rank (deterministic)<br/>+ writes the ruling in prose · logs the ledger"]
    EN(["END"])
    DR --> CR --> CD
    CD -- yes --> NG --> AR --> CR
    CD -- "no · stalled" --> EN

Draft once, then critique → negotiate → arbitrate until no hot objection remains.

Rule 1: Parallelize the debate. Serialize the decision.

The expensive, creative part is the critique — five agents reading the whole caseload and reasoning about their objective. So run the five critiques concurrently: one round-trip, not five sequential ones. That single change is the difference between a demo and a coffee break.

# orchestrator.py — the five advocates critique the SAME draft concurrently
with ThreadPoolExecutor(max_workers=len(advocates)) as ex:
    objections = [o for group in ex.map(_one, advocates.items()) for o in group]

The part that has to be trustworthy — who wins — is the opposite. The referee's decision is deterministic Python; only its rationale is the LLM. Autonomy lives where it earns its keep (what to flag, how hard, why) and stays out of where it doesn't (the ruling itself). You get a negotiation that's alive but reproducible.

# orchestrator.py — the verdict is a fixed priority ranking, never an LLM
_RANK = {"capacity": 100, "priority": 40, "window": 30, "continuity": 20, "preference": 10}

def _decide(forc, against):
    if any(a["agent"] == "capacity" for a in against):
        return False                    # capacity veto — never break feasibility
    fr = max((_RANK.get(f["agent"], 0) for f in forc), default=0)
    ar = max((_RANK.get(a["agent"], 0) for a in against), default=0)
    return fr > 0 and fr >= ar           # approve iff FOR outranks AGAINST

Design heuristic: put the LLM on the reasoning, keep a deterministic rule on the verdict. It's testable, it's cheap, and it can't drift.

Rule 2: Keep the judge outside the system

Both planners — the single agent and the society — are scored by the same pure-Python function. No LLM anywhere near it. It's locked in CI.

That's not decoration. If the thing declaring the winner is an LLM, "the society is better" is a vibe. If it's forty deterministic lines that neither planner controls, it's a claim. On the same scorer, same week, the society reaches a plan the single agent can't.

# scorer.py — one pure-Python function, no LLM anywhere, CI-locked
def score(assignments, patients, clinicians, slots, weights=None, meta=None):
    ...   # multi-objective: acuity coverage, overdue days, continuity, capacity, preference

Rule 3: Show the number you can defend

Here's the honest part. The first time I ran the full protocol on real Qwen, the live society scored lower than my deterministic reference implementation of the same protocol.

For a day that looked like a bug. It wasn't. The rule-based reference runs its critique to exhaustion — it finds every fixable conflict. The live LLM critique is sharper about what it flags but flags fewer things per round, so it converges earlier. Same scorer, honest gap.

So I had a choice: quote the big reproducible ceiling, or show the real run for what it is. I showed the real run. On live Qwen the society climbs 160 → 181 (+21) over twelve rounds — continuity breaks 30 → 25, preference misses 26 → 17, high-acuity coverage held throughout.

Same week, same deterministic scorer — the society (+21) recovers the continuity and preference a single agent abandons.

Show the number you can defend on a machine a stranger controls, not the number you wish you had. A smaller live win beats a bigger offline one.

Rule 4: If it costs money per click, gate it

I wanted judges to run it live. Which means an open public URL that fires real Qwen — a way to donate your token voucher to the first crawler that finds it.

So the demo has two doors. ▶ Replay plays a recorded real negotiation for anyone, free, no key. ◉ Run live and ⟳ Re-plan stream a fresh real negotiation over SSE, token-gated — only the judge link fires the models. Same protocol; the cost is contained.

# api.py — the live SSE endpoint bills the voucher, so gate it (constant-time compare)
gate = os.environ.get("REHABPANEL_DEMO_TOKEN")
if gate and not hmac.compare_digest(token, gate):
    return JSONResponse({"error": "live negotiation is token-gated"}, status_code=401)

One more trick that keeps live runs watchable instead of expensive: an explicit context cache on the invariant caseload. The big patient-and-slot table is identical on every advocate call, every round, so it's cached once (~99% prefix hit) instead of re-billed forty times.

# advocates.py — mark the invariant caseload block so Qwen caches the prefix
sysblocks = [
    {"type": "text", "text": _caseload_ref(context),
     "cache_control": {"type": "ephemeral"}},   # DashScope context-cache marker
    {"type": "text", "text": self.system},        # + this advocate's role (changes)
]

What you take away

Make the conflict a first-class object. A society that argues produces an auditable ledger; a single agent produces an opaque answer.
Parallelize critique, serialize the verdict — LLM on the reasoning, deterministic rule on the decision.
Keep the scorer pure and external, or your "win" isn't measurable.
Deploy it, and gate what costs money. A live, clickable demo is worth ten screenshots — as long as a crawler can't run up the bill.

A single agent collapses the trade-off. A society forced to negotiate under a referee reaches what one agent can't — and shows you exactly why each patient sits where they do.

Code (MIT): github.com/jwlai-cloud/rehabpanel. Live on Alibaba Cloud — ▶ Replay is free; ◉ Run live is token-gated. Built entirely on Qwen Cloud (dashscope-intl), LangGraph, FastAPI. Decision support, fully synthetic data — no real patient records, anywhere.