Onmitsu Part 2: What a 70-Year-Old Persona Found That a Busy Parent Didn't

#ai #bedrock #aws #agentcore

Onmitsu Part 2: What a 70-Year-Old Persona Found That a Busy Parent Didn't

Last time, I built Onmitsu (隠密) — an AI mystery-shopper agent that role-plays a citizen filling out a government form, built on Strands Agents and Amazon Bedrock AgentCore, wired to match the async API spec that Japan's Digital Agency uses to let external tools plug into their internal AI platform, Genai. The post ended on an anticlimax: after a day of iterating — local runs, cloud smoke tests, two full-form investigations — I ran the account straight into Bedrock's daily token quota. ThrottlingException: Too many tokens per day, please wait before trying again. I left it there on purpose, because that's an honest ending: the system worked, the bugs were fixed, and the last mile hit a resource limit no amount of code can solve. It resets tomorrow, I wrote.

It did. Here's what happened when I actually got to run it.

The retry: same pipeline, zero code changes, one day later

Bedrock's per-model daily token quota resets on a rolling UTC window. I didn't touch a single line of code — the fixes from Part 1 (the BrowserWorker threading fix, the region pin, the cross-region inference-profile IAM policy) were already deployed and already correct. I just re-authenticated, confirmed the deployed API was still live, and sent the same request I'd sent the day before, at the exact same endpoint, against the exact same real deployed target — not the example.com throwaway I'd used to smoke-test the wiring, the actual mock-form on Vercel:

curl -X POST "$API_URL/requests" \
  -H "x-api-key: $API_KEY" \
  -d '{"inputs":{
    "target_url":"https://mock-form-sigma.vercel.app/apply/step1",
    "goal":"子育て応援臨時給付金の申請を完了する",
    "persona_key":"elderly"
  }}'

[1]  IN_PROGRESS
[2]  IN_PROGRESS
...
[11] COMPLETED   ← ~3.5 minutes later

PENDING → IN_PROGRESS → COMPLETED, no read timeouts, no throttling, no decoy errors. 17 findings, 11 screenshots, all pulled back through the /status endpoint exactly the way the architecture diagram in Part 1 says it should work. If you're building anything that runs multi-turn agentic sessions — many LLM calls per task, not one — budget for daily token quotas explicitly, especially on a shared or personal AWS account you're also using for other Bedrock work that same day. It's an easy thing to not think about until you're mid-demo.

The interesting part isn't that it worked — it's what changed

Part 1's completed run (local Playwright, not the cloud path) used a "busy parent, ten minutes at lunch" persona and logged 15 findings, weighted toward things that block or slow down someone in a hurry: ambiguous required-checkbox copy, a validation error with zero diagnostic detail, file-upload constraints hidden behind a toggle, a "proceed" button labeled with a redundant, confusing phrase instead of a clear call to action.

This run used a completely different persona — "a 70-year-old unfamiliar with smartphones and computers, easily confused by technical terms, small text, and ambiguous wording" — driven through the full cloud path via AgentCore Browser Tool instead of local Playwright. Same target, same code, same day-old deployment. Different eyes.

Some findings overlapped with Part 1 almost exactly — which turns out to be the most reassuring part of this whole exercise (more on that below). But several were new, and they're new in a specific, legible way: they're exactly the kind of thing a person unfamiliar with government forms and small UI text would notice, and a person in a hurry would not.

[Medium] Heavy use of administrative jargon with no furigana, ruby text, or plain-language alternatives. The page has an English toggle, but no furigana anywhere. Terms like 扶養 (dependent), 基準額 (threshold amount), 世帯 (household), and 所得証明書類 (proof-of-income documents) appear throughout with no reading aid or simplified explanation — difficult for elderly users or anyone unfamiliar with bureaucratic Japanese.

That's a real, specific accessibility gap, and it's not one I designed into the mock form on purpose — I built an English/Japanese toggle and considered that "internationalization done," which is exactly the blind spot a busy-parent persona (fluent, in a hurry, skimming) would never surface, because fluency isn't the axis that persona struggles on.

[Medium] No guidance on how to actually complete the file upload. The file-upload UI is a bare "choose file" button — no explanation of drag-and-drop, and critically, no guidance for someone uploading a photo taken on a smartphone. For an elderly applicant, the real task isn't clicking a button — it's photographing a paper document, saving it, and finding it again in a file picker, and nothing on the page acknowledges that workflow exists.

Also new — and also invisible to a persona that already knows how to scan a document, because that persona doesn't stop to ask "how would someone who doesn't know how to do this figure it out?"

[Medium] The current step in the step indicator isn't visually distinguished from the others. ① → ② → ③ → ④ renders as plain text with no color or weight difference marking which step you're on, making it hard to tell where you are in a four-step process.

A detail a fast, confident user glides past without noticing it's missing.

The elderly-persona run also independently flagged something Part 1's findings never called out as its own issue: step 1's validation error ("入力内容に誤りがあります" — "there is an error in the input") and step 2's validation error ("入力エラーです" — "there is an input error") use different wording for the same underlying kind of failure. Both are equally unhelpful on their own, which is presumably why the busy-parent run treated them as two instances of the same complaint rather than flagging the inconsistency between them. The elderly persona, moving more slowly and re-reading each screen, caught the mismatch as a distinct problem: two different vocabularies for "you made a mistake" reads as a system that doesn't fully hang together, which is its own small trust cost.

The part that actually matters: independent convergence

Here's the finding I think is more interesting than any individual bug report. Several defects were rediscovered independently — different day, different persona, different execution environment (local headless Chromium in Part 1 vs. AgentCore's managed, isolated Browser Tool session in Part 2), different underlying LLM context window with zero shared state between the two runs:

Required checkboxes are missing the HTML required attribute, in both runs, independently.
The contradictory "checking this isn't necessarily required, but it's labeled required" copy, in both runs.
The referenced-but-missing income-threshold PDF link, in both runs.
The English-only 404 page when a step is skipped, in both runs.

This matters because it's a real methodological question for any LLM-driven testing tool: how do you know a "finding" is a genuine, reproducible defect and not the model confabulating a plausible-sounding complaint under the pressure of a system prompt that explicitly asks it to find problems? A single run doesn't answer that question — a demand for findings can manufacture findings. Two runs, on different days, with different personas, through different code paths, that independently converge on the same underlying defects is a real (if informal) signal that those specific findings are load-bearing rather than noise. The findings that didn't repeat — the furigana gap, the smartphone-upload guidance, the step-indicator contrast — aren't automatically wrong just because only one run caught them; they're exactly the kind of persona-specific issue you'd expect a single run to catch and a differently-configured run to walk right past. That's not a flaw in the method. That's the entire argument for running more than one persona against the same target instead of treating "ran the agent once" as coverage.

Where this leaves the project

Two completed investigations, two different personas, two different execution paths, one real deployed target, zero manual QA — 15 findings and then 17, with meaningful overlap and meaningful divergence between them. That's a more convincing demonstration than either run alone: not just "an agent that finds bugs," but an agent whose bug reports hold up under a second, independently-run test with a different premise.

Code, mock form, and infrastructure are still the same repo as Part 1: github.com/yama3133/onmitsu-agent (MIT). The mock form — still live, still breakable, still bilingual — is at mock-form-sigma.vercel.app.