Benchmark a Voice Agent Across Permission Loss and First Rollback

You sit down with a cloned notes app, a charged Pixel, and a Slack ping asking you to ship the voice agent. The README promises a wake word, a cloud planner, and a spoken recap after every meeting. Nobody documented what happens when you revoke the microphone and then lock the phone mid-sentence. Your first hour is not for polishing the greeting; it is for proving the agent can stop and roll back.

That gap is how junior engineers accidentally ship an agent that still assumes the world from tap-to-talk. The model still thinks it holds the mic, and the lock screen still leaves a session token alive. You do not need a lab study; you need one device, one permission toggle, and a rollback in the PR. Treat every cloud agent claim as a hypothesis until the OS has been allowed to disagree.

The first hour: map assumptions, not files

Skip the tour of every package on day one and walk the user flow that can hurt people. You will still clone, install, and boot the app, but you will annotate assumptions instead of architecture diagrams. Write each assumption as something the OS can steal from you without a crash.

On a typical voice notes screen the agent quietly assumes all of the following:

• RECORD_AUDIO or the iOS microphone permission stays granted for the whole session
• The process remains in the foreground while planner tokens stream back
• The network path that hosts the planner stays reachable and unchanged
• Partial transcripts are safe to retry if the RPC fails mid-sentence
• Session identifiers can live in memory until the user says goodbye

None of those survive a realistic commute between an office desk and a noisy train platform. Android and iOS will reclaim the mic, freeze sockets, and later restore your process with colder caches. If the repo already has an AI service singleton, read that file before you read the UI kit. Dangerous retries usually hide there, sitting behind a friendly "just call the planner again" helper.

Fill this block in your notes before you touch a line of product code:

Device:
OS / patch:
App build / feature flag:
Framework (RN / Flutter / native) and versions:
Permission state at launch:
Network (wifi / cellular / offline):
Power (plugged / battery saver):

You are not collecting vanity metrics for a launch blog, and you are not ranking phones by speed. You are creating the only context that makes a later rollback story reviewable by someone else. Without those fields, a passing emulator run will look identical to a failure on a battery-saver OEM build.

A decision table for the first rollback

Use this table in the PR description so reviewers argue about behavior instead of untested optimism. Mark each row as must-stop, must-prompt, or must-discard before you request a review.

OS event	Agent assumption that dies	Rollback you should prove	Sensitive leftover to inspect
Mic revoked in Settings	"I can still listen"	Move to permission_lost, stop capture, drop PCM	Audio ring buffer, cache file
App backgrounded or locked	"The user is still in the session"	Move to backgrounded, cancel TTS	Partial transcript, session cookie
Network loss mid-plan	"The planner will finish"	Fail closed, show local copy, do not auto-retry body audio	Queued multipart body
Process death then restore	"Memory still has the turn"	Cold start to idle, do not replay the last utterance	Encrypted store / MMKV / UserDefaults
Permission restored later	"Resume where we left off"	Explicit user confirm; do not auto-open the mic	Old session id

If a row cannot be tested on your loaner device, say so plainly in the PR body. A missing row is honest; a green check invented from a simulator happy path is not.

Session state you can actually roll back

Label this as a proposed module, not as production code measured on a device farm. Keep the agent in a small explicit machine so a junior can abort it from one function. The names matter more than the framework: Idle, Listening, Planning, Speaking, PermissionLost, Backgrounded.

// proposed: session.ts — wrap from RN, Flutter, or native
export type AgentPhase =
  | "idle"
  | "listening"
  | "planning"
  | "speaking"
  | "permission_lost"
  | "backgrounded";

export type AgentSession = {
  phase: AgentPhase;
  sessionId: string | null;
  transcriptDraft: string;
  audioBytesHeld: number;
};

export function createSession(): AgentSession {
  return { phase: "idle", sessionId: null, transcriptDraft: "", audioBytesHeld: 0 };
}

export function abortForOs(
  session: AgentSession,
  reason: "permission_lost" | "backgrounded"
): AgentSession {
  // Drop anything that could be retried as if the user were still talking.
  return {
    phase: reason,
    sessionId: null,
    transcriptDraft: "",
    audioBytesHeld: 0,
  };
}

export function mayCaptureAudio(session: AgentSession): boolean {
  return session.phase === "listening";
}

Wire abortForOs to the permission callback and to the app lifecycle owner on day one. Do that before you wire the greeting, the planner prompt, or any streaming UI chrome. If your platform kit already has a speech session object, wrap it rather than adding another retry queue. The first rollback is a delete of buffers, not a clever resume that keeps PCM "just in case."

On Android you can force the interesting transition with a real Settings change, then confirm the process observed it:

# proposed drill — replace the package name with yours
adb shell cmd appops set com.example.notes RECORD_AUDIO deny
adb shell am start -a android.settings.APPLICATION_DETAILS_SETTINGS \
  -d package:com.example.notes
adb logcat -d | grep -E "AgentPhase|RECORD_AUDIO|permission_lost"

On iOS Simulator the privacy tool is enough to practice the callback; still repeat the drill on a physical phone before you call the PR done:

xcrun simctl privacy booted revoke com.example.notes microphone
xcrun simctl privacy booted grant com.example.notes microphone

Capture should stop, spoken playback should stop, and sessionId should become null after the revoke. No PCM file should remain under the app sandbox cache or in an upload retry directory. If the agent restarts listening when you return from Settings, you have a resume bug. Call that a failed rollback in the PR, even if the greeting demo still sounds smooth.

First PR: ship the stop path

Your first pull request should not add an agent greeting and a planner prompt by themselves. Make the stop path visible in the diff so a tired reviewer can find it quickly. Include the following artifacts in this order, and keep the planner work for a later PR.

1. The state machine and the two OS hooks that call abortForOs.
2. A screen or log line that shows permission_lost and backgrounded as first-class UI, not a generic toast.
3. A test, even an instrumentation sketch, that revokes the mic and asserts mayCaptureAudio is false.
4. A short note of what you could not test: iOS background audio modes, OEM battery savers, Wear, cars.

// proposed: android instrumentation sketch
@Test
fun micRevokeStopsCapture() {
    val session = AgentStore.current()
    session.startListeningForTest()
    InstrumentationRegistry.getInstrumentation().uiAutomation
        .executeShellCommand("cmd appops set com.example.notes RECORD_AUDIO deny")
    Thread.sleep(1_000) // coarse; replace with an idling resource in a real suite
    assertFalse(session.mayCaptureAudio())
    assertEquals(0, session.audioBytesHeld)
    assertNull(session.sessionId)
}

Reviewers should reject auto-retry of audio bodies after the session has already been aborted. Text prompts can retry; microphone buffers should not travel with those retries under any flag. If the planner needs a replay, replay a redacted local summary the user already saw on screen. Never replay the PCM you were supposed to drop when the OS took the microphone away.

A throwaway inspector that never sees the phone mic

You will want another pair of eyes on the session dump after you finish the revoke drill. That is especially true when the PR includes prompt text that might echo user speech later. You may want a disposable inspector for those dumps while you learn the rollback path. MonkeyCode's free model access and free server option can summarize a synthetic session file without a long-lived backend. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the inspector on fake transcripts you generated for the drill, not on captured meeting audio. Do not upload contact names, production session tokens, or anything your privacy review has not cleared. Tear the server down when the afternoon ends so the dump does not become an unofficial log store. This is practice infrastructure for a junior's first rollback, not a substitute for your model vendor.

A minimal inspector contract looks like this:

# proposed: send only a redacted JSON dump you created by hand
curl -sS -X POST "$INSPECTOR_URL/summarize-rollback" \
  -H "Content-Type: application/json" \
  -d @synthetic-session.json

{
  "phase_after": "permission_lost",
  "audio_bytes_held": 0,
  "transcript_draft": "",
  "session_id": null,
  "notes": "synthetic commute revoke; no user audio"
}

Ask the free model only whether the dump still contains fields that should have been cleared. That question is cheaper than arguing about architecture on a whiteboard during onboarding hour one. It stays useful if you later swap the inspector for an internal tool your security team already likes.

Limitations

This drill does not prove battery cost, thermal throttling, or the quality of any planner model. It does not replace platform guides for background audio, CallKit, or Android foreground services. Simulator permission tools skip OEM overlays that real Android users live with every commuting week. A free server is the wrong place for real customer speech, and free model access is wrong for anything under an unread DPA.

Device timings are omitted here because this is a proposed onboarding workflow, not a lab report. If you publish numbers, publish the device, OS, app build, permission state, network, and power condition. Without those fields, other juniors cannot tell whether they reproduced your rollback or a different bug.

Who should not use this approach

Do not use this as production voice design if you ship regulated recording or medical dictation. Do not skip legal and privacy review because a junior completed a Settings revoke once on a desk phone. Do not point free model access at real user transcripts, even when the free server is empty after hours. Teams without even one physical phone should not claim a rollback works; file the gap and wait.

Skip the inspector entirely if your security team forbids any outbound dump, including synthetic ones. In that case keep the decision table and the state machine, and run the adb steps on an offline desk. The onboarding value is the failed-closed path, not the brand of inspector that read a JSON file.

After you merge, ask for the same boring facts

When a teammate repeats the drill, ask for the same boring facts you recorded in hour one. Collect the device, the OS, the exact transition, and whether the agent recovered, restarted, or silently disappeared. That sentence is more useful in a handoff doc than a demo video of a perfect spoken greeting.

If the agent talked after the mic was gone, you did not finish onboarding on that feature flag. Roll the flag back, keep the state machine, and fix the lifecycle hook before you write a smarter planner. Your first useful PR is the one that proves the agent can stop, not the one that makes it sound friendly.