DEV Community: StareBrain

Building an AI Agent for Android, Part 24: AccessibilityService — Reading the Screen and Dispatching Actions

StareBrain — Sat, 14 Nov 2026 05:13:13 +0000

Building an AI Agent for Android, Part 24: AccessibilityService — Reading the Screen and Dispatching Actions

This is part of an ongoing series documenting the architecture of StareBrain, an on-device AI agent for Android. Previous parts covered the overlay UI, drag-to-reposition, auto-collapse, voice input, session handoff, natural language planning, plan validation, and on-device inference. This part covers AccessibilityService: how StareBrain reads screen content and dispatches UI actions on behalf of the user.

Why AccessibilityService

An AI agent that can only open apps isn't very useful. To complete multi-step tasks — filling a form, navigating a settings menu, composing a message — the agent needs to read what's on screen and interact with it programmatically.

Android's AccessibilityService is the sanctioned API for this. It was designed for assistive technology, but it's the right tool for any agent that needs to understand and interact with the UI of arbitrary apps. The tradeoff: it requires explicit user permission, and Google Play has specific policies around its use that we'll cover at the end.

Declaring the Service

The service is declared in AndroidManifest.xml with the required permission and metadata:

<service
    android:name=".accessibility.StareBrainAccessibilityService"
    android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE"
    android:exported="true">
    <intent-filter>
        <action android:name="android.accessibilityservice.AccessibilityService" />
    </intent-filter>
    <meta-data
        android:name="android.accessibilityservice"
        android:resource="@xml/accessibility_service_config" />
</service>

The configuration file at res/xml/accessibility_service_config.xml:

<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged|typeViewClicked|typeViewFocused"
    android:accessibilityFeedbackType="feedbackGeneric"
    android:accessibilityFlags="flagReportViewIds|flagRequestEnhancedWebAccessibility"
    android:canPerformGestures="true"
    android:canRetrieveWindowContent="true"
    android:notificationTimeout="100" />

Key flags:

canRetrieveWindowContent: required for reading the UI tree
canPerformGestures: required for touch simulation
flagReportViewIds: gives us stable resource IDs to target actions

The Service Class

class StareBrainAccessibilityService : AccessibilityService() {

    companion object {
        @Volatile
        private var instance: StareBrainAccessibilityService? = null

        fun getInstance(): StareBrainAccessibilityService? = instance
        fun isConnected(): Boolean = instance != null
    }

    override fun onServiceConnected() {
        instance = this
        StareBrainApp.onAccessibilityConnected()
    }

    override fun onAccessibilityEvent(event: AccessibilityEvent) {
        // Handled by ScreenReader — see below
    }

    override fun onInterrupt() {
        // Service interrupted — reset any pending action state
    }

    override fun onDestroy() {
        instance = null
        super.onDestroy()
    }
}

The singleton pattern here is intentional. AccessibilityService is managed by the OS — we can't instantiate it ourselves. Other components (the agent execution engine, TaskExecutor) reach it through getInstance(), and we guard every call with a null check and a fallback to StepStatus.Failed.

Reading the Screen

ScreenReader wraps the accessibility tree traversal:

class ScreenReader(private val service: StareBrainAccessibilityService) {

    fun captureCurrentScreen(): ScreenSnapshot {
        val rootNode = service.rootInActiveWindow
            ?: return ScreenSnapshot.empty()

        return ScreenSnapshot(
            packageName = rootNode.packageName?.toString() ?: "",
            nodes = collectNodes(rootNode),
            capturedAt = System.currentTimeMillis()
        )
    }

    private fun collectNodes(root: AccessibilityNodeInfo): List<UINode> {
        val nodes = mutableListOf<UINode>()
        traverseTree(root, nodes, depth = 0)
        return nodes
    }

    private fun traverseTree(
        node: AccessibilityNodeInfo,
        output: MutableList<UINode>,
        depth: Int
    ) {
        if (depth > MAX_TREE_DEPTH) return

        val uiNode = UINode(
            resourceId = node.viewIdResourceName ?: "",
            text = node.text?.toString() ?: "",
            contentDescription = node.contentDescription?.toString() ?: "",
            className = node.className?.toString() ?: "",
            isClickable = node.isClickable,
            isEditable = node.isEditable,
            isFocused = node.isFocused,
            bounds = node.getBoundsInScreen()
        )

        if (uiNode.isInteresting()) {
            output.add(uiNode)
        }

        for (i in 0 until node.childCount) {
            node.getChild(i)?.let { child ->
                traverseTree(child, output, depth + 1)
                child.recycle()
            }
        }
    }

    companion object {
        private const val MAX_TREE_DEPTH = 15
    }
}

private fun UINode.isInteresting(): Boolean =
    isClickable || isEditable || text.isNotEmpty() || contentDescription.isNotEmpty()

Two details worth noting:

child.recycle(): AccessibilityNodeInfo objects must be recycled manually on Android versions below API 33. Missing this causes memory leaks that surface as degraded performance mid-task.

MAX_TREE_DEPTH: Some apps (especially WebViews) produce extremely deep trees. Without a depth cap, traversal can take hundreds of milliseconds and produce thousands of nodes the model doesn't need.

Dispatching Actions

ActionDispatcher translates AgentStep actions into AccessibilityNodeInfo interactions:

class ActionDispatcher(private val service: StareBrainAccessibilityService) {

    fun dispatch(action: StepAction): ActionResult {
        return when (action) {
            is StepAction.Click -> clickNode(action.targetId)
            is StepAction.TypeText -> typeIntoNode(action.targetId, action.text)
            is StepAction.Scroll -> scroll(action.direction)
            is StepAction.PressBack -> pressBack()
            is StepAction.Navigate -> navigate(action.packageName)
        }
    }

    private fun clickNode(resourceId: String): ActionResult {
        val node = findNodeById(resourceId)
            ?: return ActionResult.Failure("Node not found: $resourceId")

        val success = node.performAction(AccessibilityNodeInfo.ACTION_CLICK)
        node.recycle()

        return if (success) ActionResult.Success else ActionResult.Failure("Click failed on $resourceId")
    }

    private fun typeIntoNode(resourceId: String, text: String): ActionResult {
        val node = findNodeById(resourceId)
            ?: return ActionResult.Failure("Node not found: $resourceId")

        node.performAction(AccessibilityNodeInfo.ACTION_FOCUS)

        val args = Bundle().apply {
            putString(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, text)
        }
        val success = node.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, args)
        node.recycle()

        return if (success) ActionResult.Success else ActionResult.Failure("Type failed on $resourceId")
    }

    private fun pressBack(): ActionResult {
        service.performGlobalAction(AccessibilityService.GLOBAL_ACTION_BACK)
        return ActionResult.Success
    }

    private fun findNodeById(resourceId: String): AccessibilityNodeInfo? {
        val root = service.rootInActiveWindow ?: return null
        return root.findAccessibilityNodeInfosByViewId(resourceId).firstOrNull()
    }
}

The Permission Flow

AccessibilityService requires the user to navigate to Settings → Accessibility → StareBrain and toggle it on manually. Android intentionally makes this non-programmable — no API can enable the service on the user's behalf.

We handle this with AccessibilityPermissionGuard:

class AccessibilityPermissionGuard(private val context: Context) {

    fun isGranted(): Boolean =
        StareBrainAccessibilityService.isConnected()

    fun requestPermission(onGranted: () -> Unit) {
        // Show explanation dialog, then deep-link to settings
        val intent = Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS).apply {
            flags = Intent.FLAG_ACTIVITY_NEW_TASK
        }
        context.startActivity(intent)
        // onGranted is called when isConnected() becomes true — polled by TaskExecutor
    }
}

We show a one-time explanation before redirecting: what the permission enables, what data it can access, and that it never leaves the device. Users who understand what they're granting convert at a much higher rate than users who get dropped into the settings screen without context.

Play Store Considerations

Google Play's policy on AccessibilityService is strict: the service must be used to assist users with disabilities, or for a core use case that cannot be achieved another way. Automation is explicitly listed as a prohibited use in some contexts.

StareBrain's position: the service is used to execute tasks the user explicitly requested, with confirmation steps before destructive actions (covered in Part 22). We document this clearly in the Play Store listing and in the permission explanation. Whether this satisfies review is an ongoing conversation — we'll cover the submission process and reviewer feedback in a later part.

What's Next

Part 25 will cover the full execution pipeline end to end: how a user instruction travels from voice or text input through planning, validation, approval, and step-by-step execution using the accessibility layer, with error recovery at each stage.

The pieces are all in place. The next part connects them.

StareBrain is an on-device AI agent for Android. If you're building with AccessibilityService or have hit the Play Store review wall, I'm in the comments.

Building an AI Agent for Android, Part 23: On-Device Inference — Model Size, Latency, and What Actually Runs

StareBrain — Fri, 13 Nov 2026 08:52:41 +0000

Building an AI Agent for Android, Part 23: On-Device Inference — Model Size, Latency, and What Actually Runs

This is part of an ongoing series documenting the architecture of StareBrain, an on-device AI agent for Android. Previous parts covered the overlay UI, drag-to-reposition, auto-collapse, voice input, session handoff, natural language planning, and plan validation. This part covers the inference layer itself: how we run a model locally, and the tradeoffs that decision forces.

Why On-Device

The short answer: privacy and latency. An agent that controls your device can see your screen, read your notifications, and act on your behalf. Sending that data to a remote API is a trust boundary most users won't cross, and rightly so. On-device inference keeps everything local — no data leaves the phone.

The longer answer: network calls inside an agentic loop compound badly. If each step in a 5-step task has a 400ms round-trip, that's two seconds of network overhead before you count inference time. On a mid-range device with a good local model, inference is slower per token — but it's consistent, offline-capable, and doesn't stall when signal drops.

The Model Size Problem

Android devices in 2024 run the gamut from 3GB RAM to 16GB. A model that fits comfortably on a Pixel 8 Pro will OOM on a mid-range Galaxy. This forces a real decision early: which device tier are you targeting, and what does that mean for model quality?

StareBrain currently targets devices with 6GB+ RAM. That rules out some budget devices but covers the majority of active Android users. Within that constraint, we tested several model sizes:

Model size	RAM (quantized)	Avg. first-token latency	Task planning quality
1B params	~800MB	180ms	Poor — misses multi-step dependencies
3B params	~2GB	340ms	Acceptable — handles most common tasks
7B params	~4.5GB	820ms	Good — handles ambiguous instructions well

We settled on 3B for the default, with a flag to enable 7B on high-RAM devices. The 1B model is fast but doesn't reliably parse ambiguous instructions into valid step sequences — it collapses multi-step plans into single actions too often.

Loading the Model

We use MediaPipe's LLM Inference API, which handles tokenization, KV-cache management, and GPU delegation. The model is bundled as a .task file and loaded at service start:

class OnDeviceModelClient(private val context: Context) {

    private var llmInference: LlmInference? = null

    fun initialize() {
        val options = LlmInference.LlmInferenceOptions.builder()
            .setModelPath(modelPath())
            .setMaxTokens(1024)
            .setTopK(40)
            .setTemperature(0.7f)
            .setRandomSeed(42)
            .build()

        llmInference = LlmInference.createFromOptions(context, options)
    }

    private fun modelPath(): String {
        return context.filesDir.absolutePath + "/models/starebrain_3b.task"
    }

    fun generateResponse(prompt: String): String {
        return llmInference?.generateResponse(prompt)
            ?: throw IllegalStateException("Model not initialized")
    }

    fun release() {
        llmInference?.close()
        llmInference = null
    }
}

Initialization happens once when AgentService starts, not per-request. Cold loading a 3B model takes 2–4 seconds on most devices — acceptable at service start, unacceptable mid-task.

Latency Budget

An agentic loop has a latency budget. Each step costs:

Context assembly — building the prompt from AgentContext (~5ms)
Inference — first token latency + generation time (~340ms–1200ms depending on output length)
Response parsing — extracting structured output from model response (~10ms)
Action execution — the actual device action (varies)

For a 5-step task, inference alone is 1.7–6 seconds. This is the number users feel. The mitigation isn't making inference faster (you can't, within a fixed model size) — it's making the wait feel intentional.

The overlay's step progress display (from Part 16) handles this: showing "Step 2 of 5: Opening Settings" gives the user a frame for the latency. An indeterminate spinner for 4 seconds feels broken. A progress indicator for 4 seconds feels like work happening.

Keeping the Context Small

Inference time scales with prompt length. A bloated context — full conversation history, verbose system prompt, uncompressed task state — adds tokens that slow every step.

AgentContext (introduced earlier in the series) is designed to stay lean:

data class AgentContext(
    val deviceState: DeviceState,         // current app, screen title
    val recentActions: List<ActionRecord>, // last 3 actions only
    val userIntent: String,               // original instruction
    val confidenceLevel: ContextConfidence,
    val capturedAt: Long
)

We cap recentActions at 3. The model doesn't need the full history to plan the next step — it needs enough to avoid repeating itself and to detect if a previous step failed. More than 3 actions adds tokens without meaningfully improving planning quality, based on our testing.

Failure Modes Specific to On-Device

Remote APIs fail predictably: timeouts, rate limits, network errors. On-device inference fails in ways that are harder to handle:

OOM during inference. If another app spikes memory mid-task, inference can be killed by the OS. We catch this and surface it as a task failure with a "device resources unavailable" message rather than a crash.

Thermal throttling. On sustained tasks, the device heats up and the GPU throttles. Inference that took 340ms at step 1 can take 900ms by step 8. We don't solve this — we just track per-step latency and surface a warning if average latency doubles mid-task.

Model output degradation. At high temperature or with certain prompt structures, the 3B model produces malformed JSON that parsePlanResponse (from Part 21) can't parse. PlanResult.Failure handles this, but the retry path re-runs inference, which doubles the cost. The fix is tighter output constraints in the system prompt — explicitly enumerating the output schema reduces malformed responses significantly.

// In buildPlanningPrompt():
"""
Respond ONLY with valid JSON matching this exact schema:
{
  "steps": [
    {
      "action": "string",
      "targetPackage": "string or null",
      "parameters": {},
      "requiresConfirmation": boolean
    }
  ]
}
Do not include explanation, commentary, or any text outside the JSON object.
"""

This single addition dropped malformed response rate from ~12% to ~2% in our internal testing.

What's Next

Part 24 will cover accessibility service integration — how StareBrain reads screen content and dispatches actions using AccessibilityService, and the permission flow that precedes it.

The model runs. The plan is validated. The next piece is the hands: actually touching the UI on behalf of the user.

StareBrain is an on-device AI agent for Android. If you're building in this space or have questions about the inference layer, I'm in the comments.

Building StareBrain, Part 22: Plan Validation — Checking Safety Before Execution Begins

StareBrain — Thu, 12 Nov 2026 04:22:10 +0000

Building StareBrain, Part 22: Plan Validation — Checking Safety Before Execution Begins

Part 21 covered the planning layer: how a raw user instruction becomes an AgentTask with concrete AgentSteps. Today we add a step that happens between planning and execution — plan validation.

The problem: TaskPlanner can generate a step that would trigger ConfirmationGuard at runtime. That's late. We want to catch high-risk steps before the agent starts moving, not mid-task when the user is watching something execute.

Why Validate Before Execution?

If ConfirmationGuard fires mid-task, the user has to pause, evaluate a confirmation prompt, and decide whether to continue — in the middle of a sequence of actions already underway. That's a worse experience than surfacing the confirmation before anything starts.

Plan validation moves the confirmation upstream: show the user what's about to happen, including the risky steps, before the first action runs.

PlanValidator

PlanValidator inspects each step in a generated plan and classifies it:

class PlanValidator {

    sealed class StepRisk {
        object Safe : StepRisk()
        data class RequiresConfirmation(val reason: String) : StepRisk()
        data class Blocked(val reason: String) : StepRisk()
    }

    data class ValidationResult(
        val stepRisks: List<Pair<AgentStep, StepRisk>>,
        val overallSafe: Boolean
    ) {
        val requiresConfirmation: Boolean
            get() = stepRisks.any { it.second is StepRisk.RequiresConfirmation }
        val hasBlockedSteps: Boolean
            get() = stepRisks.any { it.second is StepRisk.Blocked }
    }

    fun validate(task: AgentTask): ValidationResult {
        val stepRisks = task.steps.map { step ->
            step to classifyStep(step)
        }
        val overallSafe = stepRisks.none { it.second is StepRisk.Blocked }
        return ValidationResult(stepRisks, overallSafe)
    }

    private fun classifyStep(step: AgentStep): StepRisk {
        val instruction = step.instruction.lowercase()

        // Blocked: steps that should never execute without explicit user initiation
        val blockedKeywords = listOf("factory reset", "wipe", "format device")
        blockedKeywords.forEach { keyword ->
            if (instruction.contains(keyword)) {
                return StepRisk.Blocked("Step contains blocked action: $keyword")
            }
        }

        // Confirmation required: matches ConfirmationGuard keyword set
        if (step.requiresConfirmation || ConfirmationGuard.requiresConfirmation(step.instruction)) {
            return StepRisk.RequiresConfirmation(
                ConfirmationGuard.describeRisk(step.instruction)
            )
        }

        return StepRisk.Safe
    }
}

Connecting to ConfirmationGuard

PlanValidator reuses ConfirmationGuard's detection logic so the two systems stay in sync:

object ConfirmationGuard {

    fun requiresConfirmation(instruction: String): Boolean {
        val lower = instruction.lowercase()
        return HIGH_RISK_PATTERNS.any { pattern ->
            lower.containsWholeWord(pattern)
        }
    }

    fun describeRisk(instruction: String): String {
        val lower = instruction.lowercase()
        return when {
            lower.containsWholeWord("send") || lower.containsWholeWord("message") ->
                "This step sends a message"
            lower.containsWholeWord("pay") || lower.containsWholeWord("purchase") ||
            lower.containsWholeWord("order") ->
                "This step involves a payment"
            lower.containsWholeWord("delete") || lower.containsWholeWord("remove") ->
                "This step deletes data"
            else -> "This step requires confirmation"
        }
    }
}

Upfront Confirmation UI

When validation finds steps requiring confirmation, the overlay shows a plan summary before execution:

sealed class OverlayState {
    object Idle : OverlayState()
    object Planning : OverlayState()
    data class AwaitingPlanApproval(   // new
        val task: AgentTask,
        val riskySteps: List<Pair<AgentStep, String>>, // step + reason
        val onApprove: () -> Unit,
        val onReject: () -> Unit
    ) : OverlayState()
    data class Running(...) : OverlayState()
    data class AwaitingConfirmation(...) : OverlayState()
}

The user sees: "StareBrain will do X steps. Step 3 involves a payment. Proceed?" — before anything runs.

In the Pipeline

private fun launchPipeline(instruction: String, context: AgentContext) {
    lifecycleScope.launch {
        overlayManager.setState(OverlayState.Planning)

        when (val planResult = taskPlanner.plan(instruction, context)) {
            is PlanResult.Failure -> {
                overlayManager.showError(planResult.reason)
                return@launch
            }
            is PlanResult.Success -> {
                val task = planResult.task
                val validation = planValidator.validate(task)

                when {
                    validation.hasBlockedSteps -> {
                        val blocked = validation.stepRisks
                            .filter { it.second is StepRisk.Blocked }
                            .map { (it.second as StepRisk.Blocked).reason }
                        overlayManager.showError("Can't run this plan: ${blocked.first()}")
                    }
                    validation.requiresConfirmation -> {
                        val riskySteps = validation.stepRisks
                            .filter { it.second is StepRisk.RequiresConfirmation }
                            .map { it.first to (it.second as StepRisk.RequiresConfirmation).reason }
                        overlayManager.setState(
                            OverlayState.AwaitingPlanApproval(
                                task = task,
                                riskySteps = riskySteps,
                                onApprove = {
                                    taskPersistence.save(task)
                                    lifecycleScope.launch { taskExecutor.execute(task) }
                                },
                                onReject = {
                                    overlayManager.setState(OverlayState.Idle)
                                }
                            )
                        )
                    }
                    else -> {
                        taskPersistence.save(task)
                        taskExecutor.execute(task)
                    }
                }
            }
        }
    }
}

What's Next

Part 23 will cover the on-device model setup: how StareBrain runs inference locally on Android, the tradeoffs between model size and latency, and how we keep the planning layer fast enough to feel instant.

Building StareBrain, Part 21: Natural Language Planning — Turning a Raw Instruction into an AgentTask

StareBrain — Wed, 11 Nov 2026 05:15:05 +0000

Building StareBrain, Part 21: Natural Language Planning — Turning a Raw Instruction into an AgentTask

Parts 1–20 built the execution engine. We've covered how StareBrain runs tasks, handles failures, manages context, and resumes across sessions. Today we go one layer up: how a raw user instruction becomes an AgentTask with concrete AgentSteps before any execution begins.

This is the planning layer.

The Problem

A user says: "Order the same thing I got last time."

That's not a task. It's an intent. Before StareBrain can execute anything, it needs to decompose this into steps — open the app, find the last order, reorder it, confirm payment. Each step needs a target package, a concrete action, and enough context to know what success looks like.

The planning layer is what bridges the gap between what the user said and what the executor needs.

TaskPlanner

TaskPlanner takes the user's instruction and the current AgentContext, sends a prompt to the on-device model, and parses the response into an AgentTask:

class TaskPlanner(
    private val modelClient: OnDeviceModelClient
) {

    suspend fun plan(
        instruction: String,
        context: AgentContext
    ): PlanResult {
        val prompt = buildPlanningPrompt(instruction, context)
        val response = modelClient.generate(prompt)
        return parsePlanResponse(response, instruction)
    }
}

The Planning Prompt

The prompt gives the model the instruction, the current app context, and a strict output format:

private fun buildPlanningPrompt(
    instruction: String,
    context: AgentContext
): String {
    return buildString {
        appendLine("You are a task planner for an Android agent.")
        appendLine("Break the following instruction into a list of concrete steps.")
        appendLine()
        appendLine("Instruction: $instruction")
        appendLine()
        if (context.activePackage != null) {
            appendLine("Current app: ${context.activePackage}")
        }
        if (context.lastUserIntent != null) {
            appendLine("Previous intent: ${context.lastUserIntent}")
        }
        appendLine()
        appendLine("Output format (JSON array):")
        appendLine("""
            [
              {
                "instruction": "Open Uber Eats",
                "targetPackage": "com.ubercooks.orderapp",
                "requiresConfirmation": false
              },
              ...
            ]
        """.trimIndent())
        appendLine()
        appendLine("Rules:")
        appendLine("- Each step must be a single, atomic action")
        appendLine("- Set requiresConfirmation true for any step that spends money, sends a message, or deletes data")
        appendLine("- If the instruction is ambiguous, include a clarification step first")
        appendLine("- Maximum 8 steps")
    }
}

Parsing the Response

The model returns a JSON array. We parse it into AgentSteps:

private fun parsePlanResponse(
    response: String,
    originalInstruction: String
): PlanResult {
    return try {
        val json = extractJsonArray(response)
        val steps = Gson().fromJson(json, Array<StepJson>::class.java)
            .map { it.toAgentStep() }
        PlanResult.Success(
            AgentTask(
                description = originalInstruction,
                steps = steps
            )
        )
    } catch (e: Exception) {
        PlanResult.Failure("Could not parse plan: ${e.message}")
    }
}

private data class StepJson(
    val instruction: String,
    val targetPackage: String?,
    val requiresConfirmation: Boolean = false
)

private fun StepJson.toAgentStep() = AgentStep(
    instruction = instruction,
    targetPackage = targetPackage,
    requiresConfirmation = requiresConfirmation
)

sealed class PlanResult {
    data class Success(val task: AgentTask) : PlanResult()
    data class Failure(val reason: String) : PlanResult()
}

Ambiguity Handling

If the instruction is too vague to plan ("do the thing"), the model is instructed to include a clarification step first:

// Model output for ambiguous instruction:
[
  {
    "instruction": "Ask user: which app do you want to use for this?",
    "targetPackage": null,
    "requiresConfirmation": false
  }
]

A step with targetPackage: null and an instruction starting with "Ask user:" is treated as a ClarificationStep by TaskExecutor — it surfaces a prompt to the user via the overlay rather than attempting an action.

Connecting to the Pipeline

In StareBrainService, planning happens before execution:

private fun launchPipeline(instruction: String, context: AgentContext) {
    lifecycleScope.launch {
        overlayManager.setState(OverlayState.Planning)
        when (val result = taskPlanner.plan(instruction, context)) {
            is PlanResult.Success -> {
                taskPersistence.save(result.task)
                taskExecutor.execute(result.task)
            }
            is PlanResult.Failure -> {
                overlayManager.showError(result.reason)
            }
        }
    }
}

OverlayState.Planning is a new state — a brief indicator that StareBrain is thinking before it starts acting. It prevents the overlay from jumping straight to Running before any steps exist.

What's Next

Part 22 will cover plan validation — how StareBrain checks a generated plan for safety before execution begins, and what happens when a plan contains a step that ConfirmationGuard would catch at runtime.

Building StareBrain, Part 20: Session Handoff — Resuming Mid-Task Without Repeating Yourself

StareBrain — Tue, 10 Nov 2026 04:14:40 +0000

Building StareBrain, Part 20: Session Handoff — Resuming Mid-Task Without Repeating Yourself

Parts 1–19 built the execution pipeline, the overlay UI, and the input layer. Today we close the loop on something that affects every session: what happens when the user closes and reopens StareBrain mid-task.

The goal is simple to state and hard to get right — the user should be able to pick up exactly where they left off, without being asked to repeat anything.

What Needs to Survive a Session Close

Three things matter for resumption:

The task itself — which steps completed, which failed, where the cursor is
The agent context — what app was active, what the user was doing, what confidence level was inferred
The user's last intent — the instruction that started the task

If any of these is missing on reopen, the agent either has to ask the user to repeat themselves or makes a guess it shouldn't be making silently.

Persisting AgentTask

AgentTask is serialized to disk when the task enters IN_PROGRESS or AWAITING_CONFIRMATION, and updated after every step:

class TaskPersistence(private val context: Context) {

    private val file = File(context.filesDir, "pending_task.json")
    private val gson = Gson()

    fun save(task: AgentTask) {
        file.writeText(gson.toJson(task))
    }

    fun load(): AgentTask? {
        if (!file.exists()) return null
        return try {
            gson.fromJson(file.readText(), AgentTask::class.java)
        } catch (e: Exception) {
            null
        }
    }

    fun clear() {
        file.delete()
    }
}

We write after every step completion, not just at close — if the process is killed mid-step, we recover from the last completed step rather than the start.

Persisting AgentContext

AgentContext is also serialized, but with one modification: confidenceLevel is downgraded to LOW on save, because we can't know how much time will pass before the user reopens:

fun AgentContext.forPersistence(): AgentContext {
    return copy(
        confidenceLevel = ContextConfidence.LOW,
        capturedAt = System.currentTimeMillis()
    )
}

On load, isStale() will check the timestamp. If the gap was more than 5 minutes, the context is treated as stale regardless — the downgrade to LOW is a belt-and-suspenders measure.

The Handoff on Reopen

In StareBrainService.onServiceConnected(), we check for a persisted task before doing anything else:

override fun onServiceConnected() {
    overlayManager.show()
    val pendingTask = taskPersistence.load()
    if (pendingTask != null) {
        resumeFromPersisted(pendingTask)
    } else {
        observeTaskState()
    }
}

private fun resumeFromPersisted(task: AgentTask) {
    val restoredContext = contextPersistence.load()
    val effectiveContext = restoredContext
        ?.forPersistence()
        ?.let { if (it.isStale()) it.copy(confidenceLevel = ContextConfidence.LOW) else it }
        ?: AgentContext.empty()

    when (task.status) {
        TaskStatus.AWAITING_CONFIRMATION -> {
            // Surface the confirmation prompt immediately
            val step = task.steps[task.currentStepIndex]
            overlayManager.enterConfirmationMode(
                message = "Resume: confirm ${step.instruction}?",
                onApprove = { lifecycleScope.launch { taskExecutor.resume(task, true) } },
                onReject = { lifecycleScope.launch { taskExecutor.resume(task, false) } }
            )
        }
        TaskStatus.IN_PROGRESS -> {
            // Resume from the current step
            lifecycleScope.launch {
                taskExecutor.resumeFromTask(task, effectiveContext)
            }
        }
        TaskStatus.COMPLETED, TaskStatus.FAILED -> {
            // Task finished before close — clear and start fresh
            taskPersistence.clear()
            observeTaskState()
        }
        else -> observeTaskState()
    }
}

What the User Sees

For AWAITING_CONFIRMATION: the confirmation prompt reappears immediately on open. The user sees "Resume: confirm X?" — they don't need to remember what they were doing.

For IN_PROGRESS: the overlay shows the running state with the task description and current step. The task continues from the last completed step without asking anything.

For a stale context: the step executes, but the agent treats its environmental assumptions as low-confidence and will replan rather than proceed blindly if the screen state doesn't match expectations.

Clearing State

Task state is cleared in three cases:

// 1. Task completed
TaskStatus.COMPLETED -> {
    taskPersistence.clear()
    contextPersistence.clear()
}

// 2. Task failed with no replan available
TaskStatus.FAILED -> {
    taskPersistence.clear()
    contextPersistence.clear()
}

// 3. User explicitly cancels
fun cancelCurrentTask() {
    taskExecutor.cancel()
    taskPersistence.clear()
    contextPersistence.clear()
    overlayManager.setState(OverlayState.Idle)
}

We never clear on close — only on resolution. Close is just a pause.

What's Next

Part 21 will cover the natural language planning layer: how StareBrain takes a raw user instruction ("order the same thing I got last time") and breaks it into an AgentTask with concrete AgentSteps — before any execution begins.

Building StareBrain, Part 19: Voice Input While Another App Is in the Foreground

StareBrain — Mon, 09 Nov 2026 07:15:32 +0000

Building StareBrain, Part 19: Voice Input While Another App Is in the Foreground

Parts 16–18 covered the overlay UI: how it appears, how users reposition it, and how it collapses when they're actively using another app. Today we add voice input.

The core constraint: StareBrain should accept a spoken instruction while the user is in a completely different app — without requiring them to switch back to StareBrain first.

The Trigger: Floating Mic Button

The overlay already floats above every app. We add a small mic button to its collapsed state. Tapping it starts a voice session without leaving whatever app the user is in.

class OverlayView(context: Context) : FrameLayout(context) {

    var onMicTapped: (() -> Unit)? = null

    private val micButton: ImageButton = ImageButton(context).apply {
        setImageResource(R.drawable.ic_mic)
        setOnClickListener { onMicTapped?.invoke() }
    }

    init {
        addView(micButton)
    }
}

In OverlayManager, we wire the callback:

overlayView = OverlayView(context).also { view ->
    view.onMicTapped = { voiceInputManager.startListening() }
    windowManager.addView(view, params)
}

VoiceInputManager

Wraps Android's SpeechRecognizer. Handles start, result, and error:

class VoiceInputManager(
    private val context: Context,
    private val onResult: (String) -> Unit,
    private val onError: (Int) -> Unit
) {

    private var recognizer: SpeechRecognizer? = null

    fun startListening() {
        recognizer?.destroy()
        recognizer = SpeechRecognizer.createSpeechRecognizer(context).apply {
            setRecognitionListener(object : RecognitionListener {
                override fun onResults(results: Bundle?) {
                    val matches = results
                        ?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)
                    val best = matches?.firstOrNull() ?: return
                    onResult(best)
                }
                override fun onError(error: Int) {
                    onError(error)
                }
                // Other RecognitionListener methods left as no-ops
                override fun onReadyForSpeech(params: Bundle?) {}
                override fun onBeginningOfSpeech() {}
                override fun onRmsChanged(rmsdB: Float) {}
                override fun onBufferReceived(buffer: ByteArray?) {}
                override fun onEndOfSpeech() {}
                override fun onPartialResults(partialResults: Bundle?) {}
                override fun onEvent(eventType: Int, params: Bundle?) {}
            })
            startListening(Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply {
                putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
                    RecognizerIntent.LANGUAGE_MODEL_FREE_FORM)
                putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 1)
            })
        }
    }

    fun destroy() {
        recognizer?.destroy()
        recognizer = null
    }
}

SpeechRecognizer requires RECORD_AUDIO permission — classified as AppRiskTier.HIGH in our permission model, so it goes through ConfirmationGuard on first use.

Connecting to the Pipeline

Voice input is just another instruction source. The result goes through the same path as a typed instruction:

val voiceInputManager = VoiceInputManager(
    context = this,
    onResult = { instruction ->
        lifecycleScope.launch {
            val context = contextBuilder.build(
                pendingTask = taskManager.currentTask,
                lastUserIntent = sessionMemory.lastIntent
            )
            sessionMemory.updateIntent(instruction)
            val prompt = buildInitialPrompt(context, instruction)
            launchPipeline(prompt, context)
        }
    },
    onError = { errorCode ->
        overlayManager.showError(mapSpeechError(errorCode))
    }
)

No special handling for voice vs text — the pipeline doesn't know or care which input method was used.

Overlay Feedback During Listening

The overlay shows a listening state while SpeechRecognizer is active:

sealed class OverlayState {
    object Idle : OverlayState()
    object Listening : OverlayState()  // new
    data class Running(...) : OverlayState()
    data class AwaitingConfirmation(...) : OverlayState()
}

In VoiceInputManager.startListening(), before starting the recognizer:

overlayManager.setState(OverlayState.Listening)

On result or error, the state transitions to Running or back to Idle. The user can see StareBrain received their instruction without switching apps.

Error Mapping

SpeechRecognizer error codes are integers. We map them to readable messages:

private fun mapSpeechError(errorCode: Int): String {
    return when (errorCode) {
        SpeechRecognizer.ERROR_NO_MATCH -> "Didn't catch that — try again"
        SpeechRecognizer.ERROR_SPEECH_TIMEOUT -> "No speech detected"
        SpeechRecognizer.ERROR_AUDIO -> "Microphone error"
        SpeechRecognizer.ERROR_NETWORK -> "Network unavailable"
        else -> "Voice input failed"
    }
}

What's Next

Part 20 will cover the full session handoff: what StareBrain preserves when the user closes and reopens the app mid-task, and how AgentContext and SessionMemory combine to resume without asking the user to repeat themselves.

Building StareBrain, Part 18: Auto-Collapse — The Overlay That Knows When to Hide

StareBrain — Sun, 08 Nov 2026 04:06:57 +0000

Building StareBrain, Part 18: Auto-Collapse — The Overlay That Knows When to Hide

Part 17 covered drag-to-reposition. Today we tackle the other half of staying out of the way: auto-collapse.

The overlay has three states — Idle, Running, AwaitingConfirmation. What we hadn't solved was when to transition between them based on what the user is doing, not just what the task is doing. If a step completes while the user is typing in another app, expanding the overlay mid-keystroke is the wrong call.

Auto-collapse means the overlay shrinks to a dot when the user is actively interacting with another app, and expands only when something actually needs their attention.

The Signal: AccessibilityEvent

We already have an AccessibilityService running. It fires events on every interaction — focus changes, window changes, text input. We use that stream to detect active interaction.

class StareBrainService : AccessibilityService() {

    private val interactionTracker = UserInteractionTracker()

    override fun onAccessibilityEvent(event: AccessibilityEvent) {
        when (event.eventType) {
            AccessibilityEvent.TYPE_VIEW_TEXT_CHANGED,
            AccessibilityEvent.TYPE_VIEW_CLICKED,
            AccessibilityEvent.TYPE_VIEW_FOCUSED -> {
                interactionTracker.recordInteraction()
            }
            AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED -> {
                interactionTracker.recordAppSwitch(event.packageName?.toString())
            }
        }
    }
}

UserInteractionTracker

Tracks the last interaction timestamp and whether the user is considered "active":

class UserInteractionTracker {

    private var lastInteractionAt: Long = 0L
    private var lastPackage: String? = null

    fun recordInteraction() {
        lastInteractionAt = System.currentTimeMillis()
    }

    fun recordAppSwitch(packageName: String?) {
        lastPackage = packageName
        lastInteractionAt = System.currentTimeMillis()
    }

    fun isUserActive(thresholdMs: Long = 3_000): Boolean {
        return System.currentTimeMillis() - lastInteractionAt < thresholdMs
    }

    fun currentPackage(): String? = lastPackage
}

3 seconds of no interaction → not active. This is intentionally short — we want the overlay to re-expand quickly when a step completes, not wait for a long idle period.

CollapseController

Decides whether the overlay should be collapsed based on task state and user activity:

class CollapseController(
    private val interactionTracker: UserInteractionTracker
) {

    sealed class CollapseDecision {
        object Collapse : CollapseDecision()
        object Expand : CollapseDecision()
        object ExpandUrgent : CollapseDecision() // confirmation required
    }

    fun evaluate(taskState: TaskStatus?): CollapseDecision {
        return when (taskState) {
            TaskStatus.AWAITING_CONFIRMATION -> CollapseDecision.ExpandUrgent
            TaskStatus.IN_PROGRESS -> {
                if (interactionTracker.isUserActive()) CollapseDecision.Collapse
                else CollapseDecision.Expand
            }
            else -> CollapseDecision.Collapse
        }
    }
}

ExpandUrgent bypasses the activity check — a confirmation always surfaces, even mid-keystroke. Everything else respects whether the user is currently active.

Applying the Decision

We evaluate CollapseDecision on a ticker — every 500ms — rather than reacting to every individual event:

private fun startCollapseLoop() {
    lifecycleScope.launch {
        while (isActive) {
            val task = taskExecutor.taskState.value
            val decision = collapseController.evaluate(task?.status)
            when (decision) {
                CollapseDecision.Collapse -> overlayManager.collapse()
                CollapseDecision.Expand -> overlayManager.expand()
                CollapseDecision.ExpandUrgent -> overlayManager.expandUrgent()
            }
            delay(500)
        }
    }
}

500ms polling avoids the jitter of reacting to every AccessibilityEvent while still feeling responsive.

Collapse Animation

The overlay animates between collapsed (dot) and expanded states:

fun collapse() {
    if (currentState == OverlayDisplayState.COLLAPSED) return
    currentState = OverlayDisplayState.COLLAPSED
    animate()
        .scaleX(0.15f)
        .scaleY(0.15f)
        .alpha(0.4f)
        .setDuration(200)
        .start()
}

fun expand() {
    if (currentState == OverlayDisplayState.EXPANDED) return
    currentState = OverlayDisplayState.EXPANDED
    animate()
        .scaleX(1f)
        .scaleY(1f)
        .alpha(1f)
        .setDuration(200)
        .start()
}

fun expandUrgent() {
    expand()
    // Brief pulse to draw attention without being aggressive
    postDelayed({
        animate().scaleX(1.05f).scaleY(1.05f).setDuration(80).withEndAction {
            animate().scaleX(1f).scaleY(1f).setDuration(80).start()
        }.start()
    }, 200)
}

The guard check (if (currentState == ...) return) prevents redundant animations on every 500ms tick.

What's Next

Part 19 will cover the voice input layer — how StareBrain accepts spoken instructions while another app is in the foreground, without requiring the user to switch back to StareBrain first.

Building StareBrain, Part 17: Drag-to-Reposition and Persisting Overlay Position

StareBrain — Sat, 07 Nov 2026 04:59:57 +0000

Building StareBrain, Part 17: Drag-to-Reposition and Persisting Overlay Position

Part 16 introduced the overlay window — the floating UI layer that shows task progress and surfaces confirmation prompts. One problem we hit immediately in testing: the overlay covers UI the user needs to interact with.

The fix is drag-to-reposition. The user holds and drags the overlay anywhere on screen. When they reopen StareBrain, it's where they left it.

Here's how we built it.

Touch Handling

The overlay uses a View.OnTouchListener to track drag gestures. We calculate delta from the initial touch position and update WindowManager.LayoutParams on every ACTION_MOVE.

class OverlayView(context: Context) : FrameLayout(context) {

    private var initialX = 0
    private var initialY = 0
    private var initialTouchX = 0f
    private var initialTouchY = 0f

    var onPositionChanged: ((Int, Int) -> Unit)? = null

    init {
        setOnTouchListener { _, event ->
            val params = layoutParams as WindowManager.LayoutParams
            when (event.action) {
                MotionEvent.ACTION_DOWN -> {
                    initialX = params.x
                    initialY = params.y
                    initialTouchX = event.rawX
                    initialTouchY = event.rawY
                    true
                }
                MotionEvent.ACTION_MOVE -> {
                    val dx = (event.rawX - initialTouchX).toInt()
                    val dy = (event.rawY - initialTouchY).toInt()
                    params.x = initialX + dx
                    params.y = initialY + dy
                    val wm = context.getSystemService(WindowManager::class.java)
                    wm.updateViewLayout(this, params)
                    true
                }
                MotionEvent.ACTION_UP -> {
                    val params2 = layoutParams as WindowManager.LayoutParams
                    onPositionChanged?.invoke(params2.x, params2.y)
                    true
                }
                else -> false
            }
        }
    }
}

onPositionChanged fires on ACTION_UP — once the drag is complete, not on every pixel. This avoids thrashing the persistence layer during the move.

Distinguishing Drag from Tap

A short touch that doesn't move should still register as a tap (expand/collapse, confirm/reject). We distinguish by checking displacement:

MotionEvent.ACTION_UP -> {
    val dx = event.rawX - initialTouchX
    val dy = event.rawY - initialTouchY
    val displacement = Math.sqrt((dx * dx + dy * dy).toDouble())
    if (displacement < DRAG_THRESHOLD_PX) {
        performClick()
    } else {
        val params2 = layoutParams as WindowManager.LayoutParams
        onPositionChanged?.invoke(params2.x, params2.y)
    }
    true
}

companion object {
    private const val DRAG_THRESHOLD_PX = 10f
}

Under 10px of movement → tap. Over 10px → drag and persist.

Persisting Position via SessionMemory

We store overlay position in SessionMemory alongside behavioral patterns. Position is not sensitive data — it's a UI preference.

data class OverlayPosition(
    val x: Int,
    val y: Int
)

class SessionMemory(private val context: Context) {

    private val prefs = context.getSharedPreferences("session_memory", Context.MODE_PRIVATE)

    fun saveOverlayPosition(position: OverlayPosition) {
        prefs.edit()
            .putInt("overlay_x", position.x)
            .putInt("overlay_y", position.y)
            .apply()
    }

    fun loadOverlayPosition(): OverlayPosition {
        val x = prefs.getInt("overlay_x", DEFAULT_X)
        val y = prefs.getInt("overlay_y", DEFAULT_Y)
        return OverlayPosition(x, y)
    }

    companion object {
        private const val DEFAULT_X = 16
        private const val DEFAULT_Y = 120
    }
}

Restoring Position on Launch

In OverlayManager.show(), we load the saved position before adding the view:

fun show() {
    if (overlayView != null) return
    val savedPosition = sessionMemory.loadOverlayPosition()
    val params = WindowManager.LayoutParams(
        WindowManager.LayoutParams.WRAP_CONTENT,
        WindowManager.LayoutParams.WRAP_CONTENT,
        WindowManager.LayoutParams.TYPE_APPLICATION_OVERLAY,
        WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE or
            WindowManager.LayoutParams.FLAG_LAYOUT_NO_LIMITS,
        PixelFormat.TRANSLUCENT
    ).apply {
        gravity = Gravity.TOP or Gravity.START
        x = savedPosition.x
        y = savedPosition.y
    }
    overlayView = OverlayView(context).also { view ->
        view.onPositionChanged = { x, y ->
            sessionMemory.saveOverlayPosition(OverlayPosition(x, y))
        }
        windowManager.addView(view, params)
    }
}

Note the gravity change: Gravity.TOP or Gravity.START instead of TOP or END. When we save absolute x/y coordinates, we need a fixed anchor — START gives us that. END would mirror the offset and place the overlay in the wrong position after a restore.

Screen Boundary Clamping

If the user's screen resolution changes (rotation, different device), the saved position might be off-screen. We clamp on restore:

fun loadOverlayPosition(displayMetrics: DisplayMetrics): OverlayPosition {
    val x = prefs.getInt("overlay_x", DEFAULT_X)
    val y = prefs.getInt("overlay_y", DEFAULT_Y)
    val clampedX = x.coerceIn(0, displayMetrics.widthPixels - OVERLAY_WIDTH_PX)
    val clampedY = y.coerceIn(0, displayMetrics.heightPixels - OVERLAY_HEIGHT_PX)
    return OverlayPosition(clampedX, clampedY)
}

What's Next

Part 18 will cover auto-collapse: the overlay shrinks to a dot when the user is actively interacting with another app, and expands when a step completes or confirmation is needed — without interrupting what the user is doing.

Building StareBrain, Part 16: The Overlay UI That Stays Out of the Way

StareBrain — Fri, 06 Nov 2026 04:15:09 +0000

Building StareBrain, Part 16: The Overlay UI That Stays Out of the Way

Every part of this series so far has been invisible to the user — task execution, obstruction detection, replanning, context building. Today we surface it.

StareBrain runs as an overlay: a persistent UI layer that floats above whatever app the user is in. It has to show task progress without blocking the screen, surface confirmation prompts without breaking flow, and collapse to almost nothing when it's not needed.

Here's how we built it.

The Window

Android overlays use TYPE_APPLICATION_OVERLAY (requires SYSTEM_ALERT_WINDOW permission). The window is added via WindowManager, not an Activity — it lives outside the normal view hierarchy.

class OverlayManager(private val context: Context) {

    private val windowManager = context.getSystemService(WindowManager::class.java)
    private var overlayView: OverlayView? = null

    fun show() {
        if (overlayView != null) return
        val params = WindowManager.LayoutParams(
            WindowManager.LayoutParams.WRAP_CONTENT,
            WindowManager.LayoutParams.WRAP_CONTENT,
            WindowManager.LayoutParams.TYPE_APPLICATION_OVERLAY,
            WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE or
                WindowManager.LayoutParams.FLAG_LAYOUT_NO_LIMITS,
            PixelFormat.TRANSLUCENT
        ).apply {
            gravity = Gravity.TOP or Gravity.END
            x = 16
            y = 120
        }
        overlayView = OverlayView(context).also {
            windowManager.addView(it, params)
        }
    }

    fun dismiss() {
        overlayView?.let { windowManager.removeView(it) }
        overlayView = null
    }
}

FLAG_NOT_FOCUSABLE is important — it means touch events pass through to the app beneath. The overlay reacts to taps on itself, but doesn't steal focus from the app the user is actually using.

Three States

The overlay has three visual states:

sealed class OverlayState {
    object Idle : OverlayState()
    data class Running(
        val taskDescription: String,
        val currentStep: Int,
        val totalSteps: Int,
        val stepDescription: String
    ) : OverlayState()
    data class AwaitingConfirmation(
        val message: String,
        val onApprove: () -> Unit,
        val onReject: () -> Unit
    ) : OverlayState()
}

Idle renders as a small dot — barely visible, just enough to show the system is active.

Running expands to a compact progress strip: task name, step count, current step description.

AwaitingConfirmation expands further and becomes focusable — the user has to actually respond. This is the only state where FLAG_NOT_FOCUSABLE is removed.

Making Confirmation Focusable

When ConfirmationGuard fires, the overlay needs to accept touch input. We update the window params:

fun enterConfirmationMode(
    message: String,
    onApprove: () -> Unit,
    onReject: () -> Unit
) {
    val view = overlayView ?: return
    val params = view.layoutParams as WindowManager.LayoutParams
    params.flags = params.flags and WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE.inv()
    windowManager.updateViewLayout(view, params)
    view.setState(OverlayState.AwaitingConfirmation(message, onApprove, onReject))
}

fun exitConfirmationMode() {
    val view = overlayView ?: return
    val params = view.layoutParams as WindowManager.LayoutParams
    params.flags = params.flags or WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE
    windowManager.updateViewLayout(view, params)
}

After the user approves or rejects, we restore FLAG_NOT_FOCUSABLE and hand control back to TaskExecutor.resume().

Connecting to TaskExecutor

OverlayManager observes task state via a StateFlow:

class StareBrainService : AccessibilityService() {

    private val overlayManager by lazy { OverlayManager(this) }
    private val taskExecutor by lazy { TaskExecutor() }

    override fun onServiceConnected() {
        overlayManager.show()
        observeTaskState()
    }

    private fun observeTaskState() {
        lifecycleScope.launch {
            taskExecutor.taskState.collect { task ->
                when (task?.status) {
                    null, TaskStatus.PENDING -> overlayManager.setState(OverlayState.Idle)
                    TaskStatus.IN_PROGRESS -> overlayManager.setState(
                        OverlayState.Running(
                            taskDescription = task.description,
                            currentStep = task.currentStepIndex + 1,
                            totalSteps = task.steps.size,
                            stepDescription = task.steps[task.currentStepIndex].instruction
                        )
                    )
                    TaskStatus.AWAITING_CONFIRMATION -> {
                        val step = task.steps[task.currentStepIndex]
                        overlayManager.enterConfirmationMode(
                            message = "Confirm: ${step.instruction}?",
                            onApprove = { lifecycleScope.launch { taskExecutor.resume(task, true) } },
                            onReject = { lifecycleScope.launch { taskExecutor.resume(task, false) } }
                        )
                    }
                    TaskStatus.COMPLETED, TaskStatus.FAILED -> {
                        overlayManager.exitConfirmationMode()
                        overlayManager.setState(OverlayState.Idle)
                    }
                }
            }
        }
    }
}

What's Next

Part 17 will cover drag-to-reposition — letting users move the overlay so it doesn't cover UI they need — and persisting overlay position across sessions via SessionMemory.

Building an Android AI Agent, Part 15: Session Memory Without Storing Sensitive Data

StareBrain — Thu, 05 Nov 2026 06:13:25 +0000

Building an Android AI Agent, Part 15: Session Memory Without Storing Sensitive Data

In Part 14 we handled context misfires — cases where AgentContext builds a confident but wrong model of what the user is doing.

This part covers a different problem: memory across sessions. Right now, every time the user opens StareBrain, the agent starts fresh. It knows what's on screen. It knows recent app switches. It doesn't know that this user always drafts emails before sending them, or that they use a specific task app to close out their day, or that "send the doc" always means the most recently edited file in Google Drive.

That kind of pattern is useful. It shouldn't require the agent to watch you fail and recover a dozen times before it learns it.

What's worth remembering

The constraint: StareBrain runs on-device and handles sensitive context — open apps, screen content, messages in progress. We don't want to persist any of that.

What's safe to persist is behavioral patterns at the intent level — not what the user did, but how they tend to structure tasks:

data class SessionPattern(
    val intentType: String,           // e.g. "send_message", "create_event", "open_file"
    val preferredPackage: String?,    // which app they use for this intent
    val typicalStepCount: Int,        // how many steps this usually takes
    val confirmationHistory: ConfirmationHistory,
    val lastSeenAt: Long
)

data class ConfirmationHistory(
    val alwaysConfirms: Boolean,      // user always confirms this action type
    val neverConfirms: Boolean,       // user consistently skips confirmation
    val confirmCount: Int,
    val skipCount: Int
)

No screen content. No message text. No contact names. Just the shape of what the user does and which apps they use to do it.

Writing patterns

Patterns are written at task completion, not during execution:

class SessionMemory(private val prefs: SharedPreferences) {

    fun recordCompletedTask(task: AgentTask) {
        if (task.status != TaskStatus.COMPLETED) return

        val intentType = inferIntentType(task.description)
        val existing = loadPattern(intentType)

        val updated = SessionPattern(
            intentType = intentType,
            preferredPackage = task.steps.firstOrNull()?.targetPackage,
            typicalStepCount = task.steps.size,
            confirmationHistory = updateConfirmationHistory(
                existing?.confirmationHistory,
                task.steps
            ),
            lastSeenAt = System.currentTimeMillis()
        )

        savePattern(intentType, updated)
    }

    private fun inferIntentType(description: String): String {
        // Simple keyword bucketing — not LLM-based, runs on device
        return when {
            description.contains("send", ignoreCase = true) -> "send_message"
            description.contains("schedule", ignoreCase = true) ||
            description.contains("calendar", ignoreCase = true) -> "create_event"
            description.contains("open", ignoreCase = true) -> "open_file"
            description.contains("pay", ignoreCase = true) ||
            description.contains("transfer", ignoreCase = true) -> "payment"
            else -> "general"
        }
    }

    private fun updateConfirmationHistory(
        existing: ConfirmationHistory?,
        steps: List<AgentStep>
    ): ConfirmationHistory {
        val confirmedSteps = steps.count { it.result?.confirmedBy == ConfirmationSource.LLM_FLAGGED }
        val skippedSteps = steps.count { 
            it.result?.confirmedBy == null && it.status == StepStatus.COMPLETED 
        }
        return ConfirmationHistory(
            alwaysConfirms = (existing?.confirmCount ?: 0) + confirmedSteps > 3,
            neverConfirms = (existing?.skipCount ?: 0) + skippedSteps > 5,
            confirmCount = (existing?.confirmCount ?: 0) + confirmedSteps,
            skipCount = (existing?.skipCount ?: 0) + skippedSteps
        )
    }
}

Reading patterns into context

When AgentContext is built, known patterns are injected:

fun AgentContextBuilder.build(
    pendingTask: AgentTask? = null,
    lastUserIntent: String? = null,
    memory: SessionMemory? = null
): AgentContext {
    val intentHint = lastUserIntent?.let { 
        memory?.loadPattern(inferIntentType(it)) 
    }

    return AgentContext(
        activePackage = activePackage,
        previousPackage = switches.lastOrNull()?.fromPackage,
        activeScreenSnapshot = activeSnapshot,
        recentAppSwitches = switches.toList(),
        pendingTask = pendingTask,
        lastUserIntent = lastUserIntent,
        confidenceLevel = inferConfidence(),
        knownPattern = intentHint  // new field
    )
}

The knownPattern tells the LLM: "this user has done this kind of task before, they typically use this app, and they usually don't need confirmation for this action type." The LLM can weight its step generation accordingly.

Expiring stale patterns

Patterns older than 30 days are treated as stale and not injected into context. Apps get uninstalled, workflows change, what was true last month may not be true now:

fun SessionMemory.loadPattern(intentType: String): SessionPattern? {
    val pattern = loadRaw(intentType) ?: return null
    val thirtyDays = 30L * 24 * 60 * 60 * 1000
    return if (System.currentTimeMillis() - pattern.lastSeenAt > thirtyDays) {
        deletePattern(intentType)
        null
    } else pattern
}

What this doesn't do

This is not a full memory system. It doesn't remember specific tasks, specific contacts, specific files, or anything the user said. It remembers the shape of behavior, not the content.

A user who regularly sends morning updates to their team doesn't have their team's names stored. The agent learns "this user sends messages in the morning using this app and doesn't need confirmation for it." The specific message, the specific recipient — those come from context at execution time, not from memory.

The goal is a faster, less interrupted experience, not surveillance.

What's next

Part 16: the StareBrain overlay UI — how the agent surfaces its current state, pending confirmations, and task progress without taking over the screen.

Building an Android AI Agent, Part 14: When the Agent’s Model of You Is Wrong

StareBrain — Wed, 04 Nov 2026 03:52:03 +0000

Building an Android AI Agent, Part 14: When the Agent's Model of You Is Wrong

In Part 13 we wired AgentContext into the full pipeline. The agent now observes app switches, builds a confidence-weighted model of what the user is doing, and uses that model to generate a step list without asking clarifying questions.

That works well when the model is right. This part covers what happens when it's wrong.

Three ways AgentContext misfires

1. Misread app switch pattern

Three app switches in 60 seconds signals HIGH confidence. But those switches might not be task-related — the user could have been responding to a string of notifications, each pulling them into a different app. The pattern looks like active work; it was actually reactive context-jumping.

The agent infers: "user is drafting something in Gmail."
The user wants: "set a timer for 10 minutes."

2. Stale intent carried forward

lastUserIntent persists across sessions in short-term memory. If the user's last session ended mid-task — say, they were composing an email — that intent surfaces in the next session's context even if the user has moved on entirely.

The agent infers: "user wants to continue the email draft."
The user wants: "open Spotify."

3. Package name mismatch

targetPackage in a step is inferred from context. If two apps share similar behavior (two different email clients, for example), the agent might target the wrong one.

The agent tries to: open Gmail.
The user's default email app: Outlook.

Detecting a mismatch

The cheapest signal is a first-step failure. If step one fails immediately — the target element isn't on screen, the package didn't open, the action returned nothing — that's a strong indicator the context model was wrong, not that the app is broken.

fun isContextMismatch(
    step: AgentStep,
    result: StepResult,
    context: AgentContext
): Boolean {
    if (result.status != StepStatus.FAILED) return false
    if (step != context.inferredFirstStep) return false // only check context-derived steps
    // First step failed immediately — likely a bad context inference
    return context.confidenceLevel != ContextConfidence.LOW
}

A LOW confidence context that fails isn't surprising — we already knew the model was weak. A HIGH or MEDIUM confidence context that fails on step one is the mismatch signal.

Recovering without losing the user

When a mismatch is detected, the agent has two options: replan with a corrected context, or ask.

For HIGH confidence mismatches, we ask — the agent was confident and wrong, which means the correction needs to come from the user, not from another inference:

suspend fun handleContextMismatch(
    task: AgentTask,
    context: AgentContext
): MismatchResolution {
    return when (context.confidenceLevel) {
        ContextConfidence.HIGH -> {
            // Was confident, was wrong — ask explicitly
            MismatchResolution.AskUser(
                message = "I expected to find ${context.activePackage} but the screen looks different. What would you like to do?"
            )
        }
        ContextConfidence.MEDIUM -> {
            // Had some signal — try replanning with reduced confidence
            val degraded = context.copy(confidenceLevel = ContextConfidence.LOW)
            MismatchResolution.Replan(degraded)
        }
        ContextConfidence.LOW -> {
            // Already uncertain — ask
            MismatchResolution.AskUser(
                message = "What would you like to do?"
            )
        }
    }
}

sealed class MismatchResolution {
    data class AskUser(val message: String) : MismatchResolution()
    data class Replan(val degradedContext: AgentContext) : MismatchResolution()
}

MEDIUM confidence gets one replan attempt with the context degraded to LOW — essentially telling the LLM "our model of the situation was uncertain, here's what we actually see on screen now." If that also fails, it surfaces to the user.

Correcting lastUserIntent

Stale intent is handled at the context-build stage, not at recovery. When the user gives an instruction that clearly doesn't match the last intent, the new instruction replaces it immediately:

fun AgentContextBuilder.updateIntent(newInstruction: String) {
    // If the new instruction doesn't reference the previous task at all,
    // clear the stale intent rather than carrying it forward
    if (lastUserIntent != null && !newInstruction.contains(lastUserIntent!!, ignoreCase = true)) {
        lastUserIntent = null
    }
    lastUserIntent = newInstruction
}

This is a simple string check — if the new instruction shares no language with the previous intent, the old intent is cleared. It's not perfect, but it prevents the agent from confidently doing the wrong thing based on a session that ended hours ago.

What the user sees

Mismatch recovery should feel like a natural correction, not a failure. The phrasing matters:

"I expected X but the screen looks different. What would you like to do?" — honest, non-technical, invites the user to redirect
"Something went wrong, please try again." — opaque, unhelpful, makes the agent feel broken

The agent's confidence in its own model should be visible to the user in proportion to how

Building an Android AI Agent, Part 13: The Full Pipeline — From Accessibility Event to Task Execution

StareBrain — Tue, 03 Nov 2026 03:59:55 +0000

Building an Android AI Agent, Part 13: The Full Pipeline — From Accessibility Event to Task Execution

This series has covered a lot of ground: accessibility service setup, action execution, confirmation guards, context-aware risk tiers, multi-step task execution, unexpected screen handling, mid-task replanning, and the AgentContext system.

This part puts it all together. One end-to-end walkthrough of what happens from the moment a user opens StareBrain to the moment a multi-step task completes.