Why We Check Twice Before a Risky Action: Defense-in-Depth in an Android AI Agent

#ai #android #kotlin #security

Introduction

When an AI agent is about to send a message, make a payment, or delete something, one layer of protection isn’t enough.

StareBrain uses two independent checks before any risky action reaches the user for confirmation: the LLM’s own requires_confirmation flag in the response, and a client-side keyword guard that runs regardless of what the LLM said.

Neither layer trusts the other. That’s the point.

Why Two Layers

The LLM decides requires_confirmation based on its understanding of the action. Most of the time it gets this right. But LLMs can be wrong — a prompt that’s slightly unusual, a new action type the model hasn’t seen in context, a subtle misclassification. If the only gate is the model’s own judgment, a missed classification means an unconfirmed destructive action goes through.

The keyword guard is dumb by design. It doesn’t understand the action — it just scans the button label or action text for known risky patterns. It can’t be confused by context. It doesn’t hallucinate. It catches what the LLM missed.

Together they form a defense-in-depth: the LLM catches the cases it understands, the keyword guard catches the cases it doesn’t.

The Confirmation Guard

`object ConfirmationGuard {

    private val RISKY_TEXT_HINTS = listOf(
        "send", "pay", "transfer", "purchase", "buy", "delete",
        "remove", "uninstall", "submit", "confirm payment",
        "place order", "checkout", "sign out", "log out",
        "deactivate", "cancel subscription", "share"
    )

    fun isRiskyActionText(text: String?): Boolean {
        if (text == null) return false
        val lower = text.lowercase()
        return RISKY_TEXT_HINTS.any { hint -> lower.contains(hint) }
    }
}`

Simple substring match on the lowercased text. No ML, no context, no state. If the button label contains “send”, “pay”, “delete”, or any other hint in the list, it’s flagged as risky.

Wiring It Into the Agent Loop

The agent loop checks both signals before dispatching:

`fun shouldRequireConfirmation(
    response: AgentStepResponse
): Boolean {
    // LLM explicitly flagged it
    if (response.requires_confirmation) return true

    // Keyword guard catches what the LLM missed
    if (ConfirmationGuard.isRiskyActionText(response.text_to_type)) return true
    if (ConfirmationGuard.isRiskyActionText(response.status_text)) return true

    return false
}`

Either signal is sufficient. The user sees the confirmation dialog if the LLM flagged it OR if the keyword guard flagged it. Both conditions don’t need to be true.

What the User Sees

When either layer triggers, the agent pauses and surfaces the action to the user before dispatching:

`data class ConfirmationRequest(
    val actionId: String,
    val actionSummary: String,
    val reason: String,
    val confirmedBy: ConfirmationSource
)

enum class ConfirmationSource {
    LLM_FLAGGED,
    KEYWORD_GUARD,
    BOTH
}`

The confirmedBy field is logged for analysis — if KEYWORD_GUARD appears frequently without LLM_FLAGGED, that’s a signal the LLM is systematically missing a class of risky actions and the prompt needs updating.

Keeping the List Current

The keyword list is a living document. New action types get added when:

A risky action slips through in testing
A new integration is added (new app, new capability)
Analysis shows KEYWORD_GUARD catching things LLM_FLAGGED missed

It’s not a replacement for improving the LLM’s classification — it’s the safety net that buys you time while you improve it.

The Rule

For any action that can’t be undone or involves real-world consequences — money, messages, deletions, account changes — confirmation should never depend on a single system getting it right. The keyword guard exists precisely for the cases where the smarter system fails.

Series so far:

Part 1: Two-axis state model + DENIED_UNRESOLVED
Part 2: Context propagation + confidence thresholds
Part 3: Late confirmation race + terminal state check
Part 4: Provider capability tiers
Part 5: Stale plan protection with FreshCtx
Part 6: Stateless backend architecture
Part 7: Defense-in-depth confirmation guard ← today

DEV Community