DEV Community

StareBrain
StareBrain

Posted on

Why We Check Twice Before a Risky Action: Defense-in-Depth in an Android AI Agent

Introduction

When an AI agent is about to send a message, make a payment, or delete something, one layer of protection isn’t enough.

StareBrain uses two independent checks before any risky action reaches the user for confirmation: the LLM’s own requires_confirmation flag in the response, and a client-side keyword guard that runs regardless of what the LLM said.

Neither layer trusts the other. That’s the point.

Why Two Layers

The LLM decides requires_confirmation based on its understanding of the action. Most of the time it gets this right. But LLMs can be wrong — a prompt that’s slightly unusual, a new action type the model hasn’t seen in context, a subtle misclassification. If the only gate is the model’s own judgment, a missed classification means an unconfirmed destructive action goes through.

The keyword guard is dumb by design. It doesn’t understand the action — it just scans the button label or action text for known risky patterns. It can’t be confused by context. It doesn’t hallucinate. It catches what the LLM missed.

Together they form a defense-in-depth: the LLM catches the cases it understands, the keyword guard catches the cases it doesn’t.

The Confirmation Guard

`object ConfirmationGuard {

    private val RISKY_TEXT_HINTS = listOf(
        "send", "pay", "transfer", "purchase", "buy", "delete",
        "remove", "uninstall", "submit", "confirm payment",
        "place order", "checkout", "sign out", "log out",
        "deactivate", "cancel subscription", "share"
    )

    fun isRiskyActionText(text: String?): Boolean {
        if (text == null) return false
        val lower = text.lowercase()
        return RISKY_TEXT_HINTS.any { hint -> lower.contains(hint) }
    }
}`
Enter fullscreen mode Exit fullscreen mode

Simple substring match on the lowercased text. No ML, no context, no state. If the button label contains “send”, “pay”, “delete”, or any other hint in the list, it’s flagged as risky.

Wiring It Into the Agent Loop

The agent loop checks both signals before dispatching:

`fun shouldRequireConfirmation(
    response: AgentStepResponse
): Boolean {
    // LLM explicitly flagged it
    if (response.requires_confirmation) return true

    // Keyword guard catches what the LLM missed
    if (ConfirmationGuard.isRiskyActionText(response.text_to_type)) return true
    if (ConfirmationGuard.isRiskyActionText(response.status_text)) return true

    return false
}`
Enter fullscreen mode Exit fullscreen mode

Either signal is sufficient. The user sees the confirmation dialog if the LLM flagged it OR if the keyword guard flagged it. Both conditions don’t need to be true.

What the User Sees

When either layer triggers, the agent pauses and surfaces the action to the user before dispatching:

`data class ConfirmationRequest(
    val actionId: String,
    val actionSummary: String,
    val reason: String,
    val confirmedBy: ConfirmationSource
)

enum class ConfirmationSource {
    LLM_FLAGGED,
    KEYWORD_GUARD,
    BOTH
}`
Enter fullscreen mode Exit fullscreen mode

The confirmedBy field is logged for analysis — if KEYWORD_GUARD appears frequently without LLM_FLAGGED, that’s a signal the LLM is systematically missing a class of risky actions and the prompt needs updating.

Keeping the List Current

The keyword list is a living document. New action types get added when:

  • A risky action slips through in testing
  • A new integration is added (new app, new capability)
  • Analysis shows KEYWORD_GUARD catching things LLM_FLAGGED missed

It’s not a replacement for improving the LLM’s classification — it’s the safety net that buys you time while you improve it.

The Rule

For any action that can’t be undone or involves real-world consequences — money, messages, deletions, account changes — confirmation should never depend on a single system getting it right. The keyword guard exists precisely for the cases where the smarter system fails.

Series so far:

Part 1: Two-axis state model + DENIED_UNRESOLVED
Part 2: Context propagation + confidence thresholds
Part 3: Late confirmation race + terminal state check
Part 4: Provider capability tiers
Part 5: Stale plan protection with FreshCtx
Part 6: Stateless backend architecture
Part 7: Defense-in-depth confirmation guard ← today

Top comments (4)

Collapse
 
ahmedznouda profile image
Ahmed ZNOUDA •

Logging confirmedBy is the part I like most. It turns the dumb guard into a measurement of where the model is weak, not just a backstop.

Two things I'd watch in the substring list. It only knows English, so on a phone set to French or Arabic the button reads "Envoyer" or "إرسال" and the guard never fires, which is probably also when the model is more likely to misjudge. The view's resource id or contentDescription might travel better than the visible label, since ids are often English even in a localized app. And substrings cut both ways: "pay" sits inside "payload", "send" inside "sender", "share" inside "shareholder". If confirmations fire too often, people learn to tap through them, and then the real one gets tapped through too.

Do you also take the app into account? "Send" in a banking app and "send" in a notes app probably deserve different treatment.

Collapse
 
starebrain profile image
StareBrain •

The localization gap is real — a guard that only reads English labels fails silently on a French or Arabic device, which is exactly the wrong place to fail. The resource ID approach is the right fix; IDs are set by the developer and almost always stay in English regardless of locale, so they travel across languages in a way visible labels don’t.

The false positive problem is the one I’d prioritize first though. “Pay” in “payload”, “send” in “sender” — if the guard fires too often on benign actions, users train themselves to confirm without reading, and the whole layer stops providing real protection. Whole-word matching over substring matching reduces that, and app context is the next layer: “send” in a banking app and “send” in a notes app are genuinely different risk levels. The current_package field in the AgentStepRequest already carries the app identity — it’s the right signal to weight the guard’s sensitivity by package.

Collapse
 
ahmedznouda profile image
Ahmed ZNOUDA •

Whole-word matching helps in English, but watch it in Arabic: the article ال and particles like و, ب and ل attach to the word, so a button reading "تأكيد الإرسال" (confirm sending) is not a whole-word match for "إرسال". Stripping those prefixes before matching, or matching on the stem, keeps the guard working there.

On the package signal, I'd make unknown packages default to the strict level and only relax it for apps you've actually classified. A banking app you've never seen before should get more checks, not fewer.

Thread Thread
 
starebrain profile image
StareBrain •

The Arabic morphology point is important — attaching particles mean whole-word matching breaks exactly where you need it most. Stemming or prefix-stripping before comparison is the right approach for Arabic, and probably the right model for any morphologically complex language. Treating it as a localization problem rather than just a string-matching problem is the correct frame.

The unknown-package default is the call I’d make too. Classifying every app upfront isn’t realistic — new apps install, enterprise apps have internal package names that won’t be in any list. Defaulting unknown to strict and relaxing only for explicitly classified LOW packages means the guard degrades safely rather than silently. The cost is occasional over-confirmation on genuinely low-risk apps the classifier hasn’t seen; that’s the right tradeoff.