I Built an AI Mystery Shopper for Government Forms — Then Watched It Find Bugs I Didn't Plant

Japan's Digital Agency (デジタル庁) runs an internal generative-AI platform for government staff called Genai — written 源内, read Gennai, named after Hiraga Gennai, an Edo-period polymath. It's not just a chatbot portal. Buried in its architecture is a small, specific hook: agencies can register an external REST API as an "administrative-use AI app," and Genai's web frontend will auto-generate a form for it, call it, and render whatever comes back. I read that spec and had one thought: that's an integration point, not just a chat widget slot.

So I built something to plug into it — an AI agent that pretends to be a citizen filling out a government form, actually clicks and types through the UI like a person would, and files a structured bug report when it gets confused. Not a linter. Not an axe-core scan. An agent that has to want to finish the form, the way an actual applicant does, and tells you exactly where it gave up.

I call it 隠密 (Onmitsu) — "undercover agent," Edo period vocabulary again, matching Genai's own naming. It's open source on GitHub, built on Strands Agents and Amazon Bedrock AgentCore (Runtime + Browser Tool), and this post is the build log: the architecture, the bugs I hit shipping it to AWS, and the genuinely interesting bugs the agent itself found — including some I never designed into the test form on purpose.

The hook: Genai's external AI app API

Genai's web interface is Digital Agency's fork of AWS's open-source Generative AI Use Cases (GenU), released as MIT-licensed OSS in April 2026 with 500+ GitHub stars. Most of it is what you'd expect from a government ChatGPT wrapper: chat, summarization, translation. The part I cared about is documented in AIアプリAPI仕様.md (AI App API Spec):

源内 Web インターフェースは、行政実務用 AI アプリとして外部の REST API を呼び出すことが可能です。

Translation: the Genai web frontend can call an external REST API as an "administrative-use AI app." You define your input form as a JSON schema — text fields, selects, file uploads — and Genai auto-generates the UI. For long-running work, the spec has a built-in async contract:

POST /requests  →  202 Accepted, { request_id, status: "PENDING", status_url }
GET  /status/{request_id}  →  { status: "PENDING" | "IN_PROGRESS" | "COMPLETED" | "ERROR", outputs?, artifacts? }

That's not a generic webhook pattern — it's specifically designed for API Gateway's 29-second integration timeout, which is exactly the constraint you hit the moment an "AI app" means "an agent that has to actually operate a browser for two or three minutes." I don't have a government staff account, so I can't actually click the register button in Genai's team-management screen — but I built the whole pipeline to the letter of that spec anyway, because the interesting part isn't the button, it's the shape of API that a real government platform expects an agentic UX-testing tool to have.

What it tests: a mock form with traps I know I built, on purpose

Before pointing anything at a real government site — which the spec explicitly doesn't require, and which I have no intention of doing without authorization — I built a fictional target: a Next.js app called mock-form, styled like a real Japanese municipal benefit application ("子育て応援臨時給付金," a made-up child-rearing benefit), with UX anti-patterns wired in on purpose:

• A checkbox says 必須 ("required") in 10px gray text, right next to intro copy that says "checking this isn't necessarily required" — visually near-invisible, and internally contradictory.
• A furigana field silently requires full-width katakana input with zero explanation, and the error message for getting it wrong is the same generic string as every other validation failure: 「入力エラーです。もう一度ご確認ください。」 ("There is an input error. Please check again.") — no field name, no hint.
• File-upload constraints (PNG/JPEG/PDF, 2MB max) are hidden behind a low-contrast "notes" toggle instead of shown up front.
• On the confirmation screen, "Edit" is a solid navy button; "Submit" is a plain underlined text link — the visually weaker element is the one that actually completes the task.
• A 20-minute idle session timeout (configurable to 15 seconds via ?fast=1 for testing) that wipes the form silently.
• An application number on the success screen, in select-none CSS, so you can't even copy it.

It ships with a full Japanese/English toggle (React context + localStorage), and — deliberately — the furigana field keeps demanding katakana even in English mode, because that's a realistic accessibility failure: a form that translates its labels but not its actual constraints.

Architecture

The 29-second API Gateway integration timeout is why /requests returns immediately and /status is polled: an agentic browser session that clicks and types through a multi-step form, with an LLM turn behind every action, can legitimately take several minutes — nowhere close to fitting inside a synchronous HTTP request.

The agent itself is deliberately boring: a strands.Agent with eight tools (open_url, observe_page, read_page_text, click_element, fill_field, take_screenshot, record_finding, wait_seconds) and a system prompt that tells it to role-play a persona, try to complete the goal, and call record_finding — not just narrate — every time something trips it up. observe_page runs a small JS snippet that walks the DOM, tags every visible interactive element with a data-onmitsu-id, and returns label text resolved from <label for> / closest-label / aria-label, so the model reasons over a compact list of {id, tag, text, required, checked} instead of raw HTML.

@tool
def record_finding(severity: str, title: str, detail: str, screenshot_label: str = "") -> str:
    """Record a UX/accessibility issue. severity is 'high' | 'medium' | 'low'.
    Describing a problem in prose isn't enough — you must call this tool."""
    findings.append({"severity": severity, "title": title, "detail": detail,
                      "screenshot": screenshot_label, "url": current_url})
    return "Recorded."

For the actual browser, there are two code paths behind one interface: Playwright's local headless Chromium for development, and AgentCore Browser Tool — a managed, isolated Chromium session reachable over CDP — for production. Same Page object either way; the tool layer doesn't know which one it's talking to.

@contextmanager
def agentcore_browser_session(region: str):
    from bedrock_agentcore.tools.browser_client import browser_session
    with browser_session(region=region) as client:
        ws_url, headers = client.generate_ws_headers()
        with sync_playwright() as p:
            browser = p.chromium.connect_over_cdp(ws_url, headers=headers)
            ...

That's the whole trick: AgentCore's managed browser speaks CDP, Playwright speaks CDP, so a tool written against Playwright's sync API doesn't care whether it's driving a local process or a remote managed session in us-east-1. It made the "works on my machine → works in the cloud" gap much smaller than I expected — right up until it didn't, which is the next section.

Bug #1: Strands runs your tools on a thread Playwright didn't ask for

The first real run against the local mock form produced this, over and over, and completed zero navigation:

greenlet.error: cannot switch to a different thread (which happens to have exited)

Playwright's sync API is a wrapper around an async implementation that uses a background event-loop thread and greenlets to fake synchronous calls. The catch: every call has to originate from the same OS thread that created the Playwright instance. Strands Agents, by default, executes tool calls concurrently via a thread pool — reasonable for I/O-bound tools in general, fatal for a tool holding a Page object.

First fix attempt: pass tool_executor=SequentialToolExecutor() to Agent(...), so tools run one at a time instead of concurrently. That helped — the crash rate dropped — but didn't fully fix it, because "sequential" still doesn't mean "same thread as the one that opened the browser." Strands still runs each tool call inside its own event-loop-managed thread; sequential just means one at a time, not thread-pinned.

The actual fix: a tiny dedicated-thread proxy that owns the browser and marshals every call onto it, regardless of which thread Strands happens to invoke the tool from.

class BrowserWorker:
    """Serializes Playwright calls onto one dedicated thread."""
    def __init__(self, use_agentcore, region="us-east-1", headless=True):
        self._executor = ThreadPoolExecutor(max_workers=1)
        self._cm = open_browser_session(use_agentcore, region=region, headless=headless)
        self._page = self._executor.submit(self._cm.__enter__).result()

    def run(self, func):
        return self._executor.submit(func, self._page).result()

Every tool now does worker.run(lambda page: page.goto(url, ...)) instead of touching page directly. I verified it explicitly — spun up a BrowserWorker, called .run() from the main thread, then again from a manually spawned thread simulating Strands' scheduler — before trusting it with a real agent run. With BrowserWorker in place, the same investigation that produced zero results now ran clean end to end and generated 15 findings with 8 screenshots on the first real pass.

What it actually found

Some of the 15 were exactly the traps I'd planted — the ambiguous required-checkbox copy, the unhelpful validation error, the hidden file-upload constraints. Those are expected; I'm glad the agent noticed them, but they're not the interesting part.

The interesting part is what it found that I didn't plant:

[High] Required checkboxes are missing the HTML required attribute. All three checkboxes display "required" as a label, but required is false in the actual markup. If validation logic ever regresses, a user could proceed without meeting eligibility requirements.

That's true — and it's a real gap between my visual design intent and the DOM. I'd written client-side JS validation that blocks submission without checking, so functionally nothing broke, but the agent read the accessibility tree, not my intentions, and caught the discrepancy.

[High] Navigating directly to /apply/step4 returns a 404 page in English, on an otherwise entirely Japanese site, with no way back.

Also true, and also not something I'd thought about — there is no step4 route (the fourth step is /apply/confirm), and Next.js's default 404 page is English-only. An agent probing for "is there a way to skip the blocked step" found a genuine internationalization inconsistency as a side effect of trying to route around a dead end.

[High] Input fields have no aria-label and empty type attributes in the DOM snapshot — a screen reader would have nothing to announce.

Also correct, from the same observe_page tool used to navigate, repurposed by the model as an accessibility audit without being told to.

None of this came from a checklist. On the furigana field specifically, the agent's own transcript shows real trial and error, not pattern-matching against something I told it to look for: it filled in a full name with a space ("山田 花子"), got the same generic validation error as every other failure, hypothesized the date format was the problem, ruled that out, tried the contact field's hyphen format, ruled that out too, and only then tried the name field again without the space — noticed it worked, and that's when it logged the finding, correctly attributing the actual cause instead of just reporting "there was an error somewhere." That's closer to how an actual confused applicant behaves than a scripted test would ever get, because it isn't scripted: nothing in the prompt mentions spaces, names, or katakana.

Bug #2: the region default that silently wasn't what I wrote

The CDK stack has to live in us-east-1 — AgentCore Runtime and this account's Bedrock model access are pinned there — while my other infra defaults to ap-northeast-1 (Tokyo). I wrote this, thinking it was a safe fallback:

const region = process.env.CDK_DEFAULT_REGION || "us-east-1";

cdk diff showed every ARN in the plan resolving to ap-northeast-1. The CDK CLI populates CDK_DEFAULT_REGION from the active AWS CLI profile's default region before your app code ever runs — so on a machine where the AWS CLI default is Tokyo, that env var is never unset, and my us-east-1 fallback never fires. The fallback logic was backwards for a stack that has to be pinned, not defaulted:

const region = "us-east-1"; // no fallback — this stack does not follow the CLI profile

Bug #3: the ARN a cross-region inference profile actually needs

First real deploy, first real invocation, immediate AccessDeniedException:

User: .../assumed-role/OnmitsuStack-AgentRuntimeRole.../BedrockAgentCore-...
is not authorized to perform: bedrock:InvokeModelWithResponseStream
on resource: arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-sonnet-4-6

I'd granted InvokeModel* on two resources: the exact foundation-model ARN built from the model ID I was passing in (us.anthropic.claude-sonnet-4-6 — a cross-region inference profile ID, not a bare foundation-model ID), and a wildcard on inference-profile/*. Both looked reasonable. Neither matched what IAM actually checked.

The error message is the tell: the denied resource is foundation-model/anthropic.claude-sonnet-4-6 — no us. prefix, and critically, no account or region-specific scoping that would let a single-region ARN match. Cross-region inference profiles route the actual request to whichever underlying regional endpoint has capacity; Bedrock evaluates the caller's permissions against that underlying foundation-model resource, not the profile resource you invoked. Granting on the inference-profile ARN authorizes you to call the profile; it does nothing for the model the profile hands you off to.

resources: [
  "arn:aws:bedrock:*::foundation-model/*",  // region-wildcarded: the profile can route anywhere
  `arn:aws:bedrock:${this.region}:${this.account}:inference-profile/*`,
],

Both statements are necessary; neither alone is sufficient. This is the kind of thing you only find by actually invoking, because cdk diff, the IAM policy simulator, and reading the docs all suggested my original two-line policy was fine.

Bug #4: not a bug — a quota

The IAM fix above was verified with a deliberately trivial smoke test — target_url: https://example.com, a throwaway goal — which completed end to end in under a minute: PENDING → IN_PROGRESS → COMPLETED, with a real S3-backed report and screenshot round-tripped through the status API. Wiring confirmed. Then I pointed the same pipeline at the actual deployed mock-form, end to end, to have a clean final artifact for this post. It failed. So did the retry. CloudWatch had the actual root cause, and it wasn't code:

ThrottlingException: An error occurred (ThrottlingException) when calling the
ConverseStream operation (reached max retries: 4): Too many tokens per day,
please wait before trying again.

A day of iterating — local runs, cloud smoke tests, two full-form investigations, this article's own drafting pass reading through logs — had run the account into Bedrock's daily token quota for this model. The first time this happened, the symptom I actually saw in DynamoDB was a different error — a boto3 client-side read timeout — because my worker Lambda's bedrock-agentcore client didn't have an extended read_timeout configured, so the client gave up and reported a timeout before the server-side retries inside AgentCore Runtime had finished exhausting themselves against the real throttling error. I fixed the timeout (it's a legitimate latent bug — a multi-step agentic browser session can legitimately take several minutes, and a ~60-second default client timeout will always misreport slow-but-working sessions as broken), redeployed, and got the real error on the next attempt instead of a decoy one. Better error, same root cause: no more tokens today.

I'm leaving it there rather than working around it, because that's an honest ending for this kind of post: the system works, the pipeline works, the bugs are fixed and merged — and the last mile of "run it one more time for a screenshot" hit a real-world resource limit that no amount of code changes solves. It resets tomorrow.

What this pattern is actually good for

Stepping back from Genai specifically: "agent that role-plays a user and files structured findings, fronted by an async job API matching whatever platform you're plugging into" is a reusable shape. The genuinely useful property, demonstrated by the findings above, is that an LLM-driven UX agent operating through an accessibility-tree-shaped tool (observe_page) naturally catches accessibility and DOM-hygiene issues as a side effect of trying to navigate — it isn't running axe-core, it's just reading the same signal a screen reader would, because that's the only signal it has to work with. A scripted E2E test checks what you told it to check. This kind of agent finds what a confused person would actually run into, including the things you forgot to check for.

Code, mock form, Lambda handlers, and CDK stack are all on GitHub: github.com/yama3133/onmitsu-agent (MIT). The mock form is live at mock-form-sigma.vercel.app if you want to see (or try to break) the traps yourself — in Japanese or English, top right corner.

Sources

• デジタル庁 — ガバメントAI「源内」
• digital-go-jp/genai-web — MIT-licensed source, forked from AWS's Generative AI Use Cases (GenU)
• Strands Agents
• Amazon Bedrock AgentCore — Runtime, Browser Tool