We Tried to Make a Voice Agent Control a Real Device. Here's What Broke

Most voice-agent demos have a simple loop:

1. The user speaks.
2. The model thinks.
3. The agent replies.

That loop works well when the agent only needs to answer a question.

It becomes much harder when the agent is also controlling a real device.

Aiden can read a phone’s screen and operate it through an external hardware control path. A task might involve several screenshots, tool calls, taps, swipes, and waits for the device to respond.

While that is happening, the person may still want to speak:

• “Stop.”
• “Actually, open the other app.”
• “Wait, I’ll handle this part.”
• “What is it doing right now?”

Our first instinct was to treat voice as another input method around the existing Agent loop. That turned out to be the wrong abstraction.

The difficult part was not streaming audio. It was deciding who owns the task, when messages should be delivered, and how to keep two different kinds of Agent work from interfering with each other.

Lesson 1: Don’t make the conversation own the device task

A long-running GUI task and a real-time conversation have different timing requirements.

The device task needs to:

• inspect the screen;
• call tools;
• send input;
• wait for the device;
• inspect the new screen;
• continue or recover.

The voice interaction needs to respond quickly and remain available for interruptions.

Putting both inside one loop created predictable problems. The conversation had to wait for device work, and interruptions arrived too late to be useful.

The solution was to split the responsibilities:

• The foreground Realtime Agent handles listening, short conversational responses, and interruption.
• The backend Agent handles screen understanding, tool calls, and device actions.
• An asynchronous task queue connects the two.

The foreground Agent is not a second GUI operator. It is an interaction layer that can start, cancel, query, or redirect work handled by the backend.

This separation also means that the existing backend execution path can be reused across different voice modes instead of implementing device control twice.

Lesson 2: “Completed” does not mean “Successful”

This sounds obvious, but it matters a lot in an Agent system.

A task can finish because the execution loop reached its end state. That does not necessarily mean the intended result happened.

For example, a task might:

• open an app but fail to find the expected screen;
• send an input event but receive an unexpected result;
• stop after hitting a safety or loop guard;
• wait for a human action and never receive it.

We therefore keep execution state and outcome separate:

• Completed means execution ended.
• Failed means an abnormal condition interrupted execution.
• The task result explains whether the intended action was actually achieved.

Without this distinction, the conversational Agent is encouraged to turn “the loop stopped” into “the task succeeded.” That is a small wording problem in a demo and a serious reliability problem in a physical system.

The distinction also appears directly in the task model. Execution status, result, error, and pending human action are stored separately:

type Status string

const (
    StatusCreated    Status = "created"
    StatusQueued     Status = "queued"
    StatusRunning    Status = "running"
    StatusCancelling Status = "cancelling"
    StatusCancelled  Status = "cancelled"
    StatusFailed     Status = "failed"
    StatusCompleted  Status = "completed"
)

type Task struct {
    ID                string      `json:"id"`
    Prompt            string      `json:"prompt"`
    Status            Status      `json:"status"`
    Result            string      `json:"result,omitempty"`
    Error             string      `json:"error,omitempty"`
    CreatedAt         time.Time   `json:"created_at"`
    UpdatedAt         time.Time   `json:"updated_at"`
    StartedAt         *time.Time  `json:"started_at,omitempty"`
    CompletedAt       *time.Time  `json:"completed_at,omitempty"`
    PendingUserAction *UserAction `json:"pending_user_action,omitempty"`
}

The important part is that StatusCompleted only describes the execution lifecycle. The actual outcome still has to be understood from Result and the observed device state.

View the task model on GitHub

Lesson 3: Notifications need backpressure

The backend Agent can produce several events in a short period: tool results, screen changes, task updates, or requests for human input.

If every event is immediately injected into the foreground conversation, the model receives a noisy stream of updates. Worse, a background result can arrive in the middle of an answer and compete with the person’s current request.

We added a notification queue with a short aggregation window. When several task results arrive close together, the runtime waits 500 milliseconds, extending the window if another result arrives, and then delivers the combined update.

We also avoid injecting a background task message while the foreground Agent is actively answering.

This is not only a queueing detail. It changes how the interaction feels. A real-time Agent should not sound as if it is being interrupted by its own internal logs.

Here is the less glamorous—but surprisingly important—part of the implementation. Instead of forwarding each completed task immediately, the runtime resets a short debounce timer whenever another result arrives:

case <-agentTaskNotifications(tasks):
    pendingTaskUpdates = append(
        pendingTaskUpdates,
        tasks.DrainTerminalTasks()...,
    )
    taskUpdatesReady = false

    if taskDebounceTimer == nil {
        taskDebounceTimer = time.NewTimer(
            realtimeTaskResultDebounce,
        )
    } else {
        if !taskDebounceTimer.Stop() {
            select {
            case <-taskDebounceTimer.C:
            default:
            }
        }
        taskDebounceTimer.Reset(realtimeTaskResultDebounce)
    }

    taskDebounce = taskDebounceTimer.C

realtimeTaskResultDebounce is currently set to 500 milliseconds. It is a small delay, but it prevents several closely spaced backend events from turning into several competing voice responses.

View the debounce interval and the notification queue logic.

Lesson 4: Keep the two contexts separate

The foreground and backend Agents should not share one constantly changing transcript.

The backend needs detailed device state and tool history. The foreground needs enough information to explain what is happening and respond to the person. Mixing everything together makes both contexts harder to manage.

Aiden uses two runtime message types to communicate across the boundary:

• StateMessage carries current device and runtime state.
• NoticeMessage carries events generated by the runtime, such as task results, loop-guard corrections, or requests for human action.

For example, a new screen observation can be added as state without rewriting the system prompt. A completed backend task can be delivered as a notice without pretending that the person said it.

At the model boundary, these runtime messages are converted into ordinary user-message content. The runtime still controls where the information came from and when it should be delivered.

Lesson 5: Human handoff is part of the normal lifecycle

A physical-device task will sometimes need a person.

Maybe the device requires authorization. Maybe the user needs to choose an account. Maybe the Agent reaches an action that should not be automated without confirmation.

Instead of treating this as an exception outside the task system, Aiden gives it an explicit lifecycle:

• The backend calls request_user_action.
• The foreground Agent asks the person.
• The foreground Agent returns the answer through response_user_action.
• The backend continues with the same task identity and device context.

The task remains managed while it waits.

We also keep device execution serial for now. Multiple tasks competing for one screen and one input path create ambiguous ownership: which task owns the next screenshot, or the next tap? Serial execution is less exciting than parallelism, but it keeps observation and action understandable.

When the person provides the missing information, we do not create an unrelated replacement task. The existing task is returned to the queue with a continuation message:

if item.task.Status != StatusRunning ||
    item.task.PendingUserAction == nil {
    m.mu.Unlock()
    return Task{}, errors.New(
        "agent task is not waiting for user action",
    )
}

select {
case m.queue <- taskID:
    item.task.PendingUserAction = nil
    item.actionNotified = false
    item.nextPrompt = userMessage
    item.resumeQueued = true
    item.task.UpdatedAt = m.now().UTC()

    task := item.task
    m.mu.Unlock()
    return task, nil

default:
    m.mu.Unlock()
    return Task{}, errors.New("agent task queue is full")
}

This lets the workflow preserve its task identity while making the pause and resume behavior explicit.

View the human-handoff continuation code on GitHub

What we are still validating

This implementation is part of Aiden’s open-source development-board firmware. It is not a claim that every board, phone, operating-system version, or audio setup behaves identically.

The actual experience still depends on:

• the target device and OS;
• the audio and trigger path;
• the configured model endpoint;
• screen-capture and input hardware;
• the specific task being tested.

Useful tests include speaking while audio is playing, interrupting a running task, canceling and querying task state, waiting for human input, and checking that only one device task owns the control path at a time.

The full-duplex architecture is still evolving, but the main lesson has held up:

When an Agent talks to a person and operates a real device at the same time, the core problem is not just voice latency. It is ownership, scheduling, context, and honest task state.

The implementation is available in the Aiden firmware repository. The deeper architectural write-up is available on the Aiden blog.

If you are building a voice Agent that can also take real-world actions, I’d be interested to hear how you handle interruptions, task ownership, and human handoff.

Comments

[Baptiste Le Bouquin]

Lesson 1 is the one every agent framework eventually rediscovers the hard way. We run headless browser agents with the same shape — a long tool loop on one side, a chat surface on the other — and as long as one asyncio task owned both, interruptions arrived at whatever point the loop happened to yield, which for us meant "stop" landing in the middle of a form submission we really did not want to complete. Splitting into an interaction layer that can only issue start/cancel/query/redirect against a queue is exactly where we landed too.

The Completed-versus-Failed status axis maps 1:1 onto our problem. When a browser agent's loop ends, the question "did the page actually change the way the user wanted" is a different fact from "the loop exited", and conflating them is how you get an assistant cheerfully reporting success after a failed login that returned a 200 with an error banner.

Two questions from the trenches. How do you handle cancel granularity — when a cancel lands while the backend is mid-action (tap dispatched, device not yet settled), do you let the in-flight step finish and refuse the next one, or is there an explicit "abort at next safe point" policy in the task model? And on barge-in: does the Realtime Agent get any live view of the queue (pending + current step description) so it can answer "what is it doing right now" without poking the backend agent and derailing its loop? That's the spot where our design still feels held together with tape.

[Dan]

This is a very fair read, and “held together with tape” is probably the right description of the remaining edge cases.

On cancellation, our current contract is cooperative and task-level. Queued work is cancelled immediately. Running work moves to cancelling, then becomes cancelled after the underlying runtime returns from context cancellation.

We cannot retract a HID report once it has been dispatched, and we do not yet claim transactional rollback or a formally specified “abort at next safe point” policy for every device action. The behavior we want is to stop scheduling subsequent actions, treat the in-flight effect as potentially unsettled, and re-observe the device before resuming. That boundary should be made more explicit in the task model.

On barge-in, the Realtime Agent has query_agent_task, and terminal updates are aggregated and delivered only when the foreground session is idle, so background results are designed not to cut through live speech or an active response.

But you’re right that this is not yet a full live queue view. The current task snapshot exposes state, terminal result/error, and pending user action; it does not provide a continuously updated queue plus a “current tool/action” description. That is the observability layer we still need.

So I think we got the ownership split right, but not all the control semantics around it. For the next iteration, would you favor explicit safe-point checkpoints around device actions, or a lease/owner model for the HID path?