DEV Community: Dan

We Kept Rebuilding Aiden for Six Months: From a HID Demo to a Physical AI Agent

Dan — Wed, 09 Sep 2026 08:34:26 +0000

Six months ago, Aiden was much closer to a hardware experiment than an Agent platform.

The first public version proved a simple idea:

Capture the screen of a phone.
Send that image to a multimodal model.
Turn the model’s decision into keyboard, pointer, or touch input.
Observe the result and repeat.

That was enough for a demo.

It was not enough for a system that had to survive USB reconnects, stale screenshots, context limits, interrupted voice conversations, different operating systems, and tasks that occasionally needed a human.

During roughly six months of development—the public repository history begins in late April—we kept rebuilding Aiden around those failures.

This post is a tour of the current architecture, but more importantly, it is a record of the assumptions we had to replace along the way.

What Aiden actually is

Aiden is a physical AI Agent project for operating phones and computers through an external hardware control path.

The current development-board implementation connects to a target device through a USB-C hub:

the target display is captured through HDMI;
an RK628D or TC358743 bridge converts the video signal to CSI;
a C++ service captures frames from the Linux video device;
the Go Agent sends visual observations to a configured multimodal model;
the model returns structured tool calls;
the Agent writes keyboard, pointer, touch, or auxiliary-control reports through Linux USB HID gadget devices.

The basic loop looks like this:

Target display
    |
    v
HDMI -> RK628D or TC358743 -> CSI
    |
    v
/dev/video0
    |
    v
frame_service
    |
    v
Screenshot observation
    |
    v
Go Agent -> configured model endpoint
    |
    v
Structured tool call
    |
    v
/dev/hidg0, /dev/hidg1, /dev/hidg2
    |
    v
Target device input

Voice adds another path:

Board microphone
    |
    v
audio_service
    |
    v
VAD -> STT or audio attachment
    |
    v
Go Agent -> LLM
    |
    v
TTS
    |
    v
audio_service playback

The firmware and device-side Agent Runtime are open source. The current repository includes the firmware overlay, C++ hardware services, Go Agent, OTA tooling, tests, and benchmark infrastructure.

It is not the final integrated Aiden hardware product.

There is another boundary worth making clear:

The Agent Runtime runs on the device. Model inference does not have to.

Screenshots, audio, and text are sent to the model, STT, TTS, and search endpoints configured by the owner. Those endpoints can be hosted or self-hosted. Aiden does not require an Aiden-hosted inference backend.

You can inspect the current architecture in the Aiden documentation and the open-source firmware repository.

The architecture we ended up with

The current system has several distinct layers:

Layer	Responsibility
Hardware abstraction	Low-level GPIO, audio, video, and HID capabilities
Hardware services	Own shared resources such as video capture and audio
Cross-language transport	Connect Go and C++ through Unix domain sockets
Go Agent	Model requests, tools, memory, tasks, voice, and Web APIs
USB gadget layer	Keyboard, pointer/touch, auxiliary control, and ECM networking
Firmware integration	Startup scripts, configuration, watchdogs, OTA, and image generation
Evaluation	Environment bridges, benchmarks, traces, screenshots, and reports

Four principles now shape most of the codebase:

A hardware resource should have one owner.
Go and C++ should communicate through explicit protocols rather than a shared ABI.
Agent reasoning should be decoupled from hardware implementation details.
Every important action should leave enough state and evidence to explain what happened.

We did not begin with all four principles. We reached them by repeatedly breaking the earlier architecture.

Refactor 1: From opening hardware everywhere to one resource owner

Screen capture was one of the first architectural problems.

A simple prototype can open /dev/video0, capture a frame, and close it. That becomes unreliable when several tools or services believe they can access the same device.

Multiple consumers introduce questions that the model should never have to answer:

Which process owns the capture stream?
Who configures the sensor?
What happens if HDMI disconnects?
Which frame is fresh?
Who restarts the pipeline after failure?

We moved that responsibility into frame_service.

frame_service is the single owner of the video device. Other processes request screenshots through a Unix domain socket rather than opening the hardware themselves.

The audio path follows the same principle. audio_service owns recording, playback, and volume state.

The UDS protocol uses a JSON header with an optional binary payload. That gives us a language-neutral boundary: C++ can own the hardware while Go consumes the capability without depending on a C++ ABI.

Go Agent
   |
   | screenshot request over UDS
   v
frame_service
   |
   | fresh frame
   v
/dev/video0

This separation also made failures easier to locate.

If the Agent cannot get a screenshot, we can ask whether the failure is in:

the model;
the Go screenshot client;
the UDS protocol;
frame_service;
the HDMI bridge;
or the underlying video device.

Before the boundary existed, all of those failures looked like “the screenshot tool failed.”

The first major frame-service work appeared in PR #17. Later changes moved screenshots behind a Screen Provider, changed capture to happen on demand, and improved freshness and latency in PR #519, PR #521, and PR #559.

The reusable lesson was:

Don’t let every part of an Agent application become a hardware driver.

Refactor 2: From platform branches to explicit action semantics

Our early HID code was full of apparently small platform decisions.

Should a pointer be absolute or relative? Is a swipe a mouse movement or a touchscreen gesture? Which coordinate space does a screenshot use? Should text entry use HID, ADB, or a Phone Bridge?

These decisions leaked into tool descriptions, prompts, configuration, and benchmark code.

That made the Agent’s behavior depend on several sources of truth at once.

We eventually separated two layers:

what the Agent intends to do;
how the configured device performs it.

The intention layer contains operations such as:

click;
swipe;
drag;
keyboard shortcut;
text entry;
auxiliary media or Android controls.

The execution layer selects a configured HID, ADB, or HTTP input path and applies the appropriate coordinate and keyboard rules.

This became the MNK Provider abstraction.

We also removed reverse platform inference. The configured device_type became the authority instead of asking multiple parts of the runtime to guess the target platform.

That made the system less magical and more predictable.

Aiden still does not have one universal input transport. HID, ADB, and HTTP remain different paths with different constraints. The abstraction unifies the Agent-facing action semantics, not the underlying operating systems.

We then changed what happens after an action.

A click or swipe should not be considered successful merely because writing the input report returned no error. The device may ignore the event, display another page, or route the input somewhere unexpected.

The runtime began attaching post-action visual observations and touch markers so the model could reason about where the action landed and what changed afterward.

A drag could also be separated into:

drag_start
    |
    v
observe the screen
    |
    v
drag_release

That lets the Agent inspect the intermediate state before committing to the release.

The central input refactors are visible in PR #539, PR #543, PR #588, and PR #595.

The lesson:

A shared action interface is useful only if the platform-specific boundary underneath it remains explicit.

Refactor 3: From scattered settings to a configuration control plane

As Aiden added more model, STT, and TTS providers, configuration became a source of architectural debt.

Each integration brought slightly different assumptions:

provider-specific base URLs;
different model identifiers;
different reasoning controls;
credentials that must survive unrelated edits;
STT and TTS settings with their own defaults;
fields understood by the runtime but not yet understood by the UI.

A naive configuration page reads a file, edits the fields it knows, and writes the whole file back.

That is dangerous. Unknown fields, comments, ordering, and credentials can disappear because the UI did not understand them.

We gradually moved provider and configuration ownership into the Agent Runtime.

The current direction includes:

unified model, STT, and TTS provider records;
provider-owned endpoint configuration;
metadata-driven validation;
native Anthropic and Responses API paths;
provider-aware reasoning controls;
JSON Merge Patch updates;
atomic TOML writes;
preservation of comments, formatting, unknown fields, and write-only credentials.

Config Web also moved toward using the Go Agent as its management API instead of maintaining a second implementation of configuration behavior.

That is more than a frontend refactor.

It establishes one authority for resolving, validating, and applying runtime configuration.

Relevant changes include PR #484, PR #504, PR #542, PR #551, and PR #631.

The lesson:

A configuration UI is part of your runtime contract, not just a form that edits a file.

Refactor 4: From one growing transcript to several kinds of memory

Our early mental model of context was straightforward: keep the conversation and send the relevant history back to the model.

That stops being straightforward when the history contains:

user messages;
assistant responses;
screenshots;
tool calls;
large tool results;
device state;
runtime warnings;
learned procedures;
user preferences;
task evidence.

Putting all of this into one transcript creates both token pressure and semantic confusion.

We started separating the information by purpose.

The current memory architecture distinguishes:

session memory, which keeps the active conversation and compressed history;
long-term memory, which stores explicitly saved user facts, preferences, and rules;
device memory, which stores reusable device, application, navigation, procedure, calibration, failure, and fact knowledge;
task episodes, which preserve execution evidence for audit and background learning.

Device memory is no longer automatically injected into every prompt. The model calls the recall tool when a task materially depends on previous device knowledge.

The runtime message model also distinguishes ordinary conversation from state generated by the system:

const (
    MessageRoleUser       MessageRole = "user"
    MessageRoleAssistant  MessageRole = "assistant"
    MessageRoleToolCall   MessageRole = "tool_call"
    MessageRoleToolResult MessageRole = "tool_result"
    MessageRoleState      MessageRole = "state"
    MessageRoleSystem     MessageRole = "system"
    MessageRoleNotice     MessageRole = "notice"
)

A State message can describe the current device or screen environment.

A Notice message can carry a task result, loop-guard correction, or human-action request without pretending that the person wrote it.

Large results and old context introduced another set of refactors:

large tool results can be stored as recoverable artifacts;
historical tool results and state can be pruned before compaction;
a context session can rotate after truncation;
parent-child lineage records how the new session was created;
persisted session events became the recovery source instead of redundant history mirrors.

Relevant changes include PR #475, PR #497, PR #524, PR #607, PR #609, and PR #618.

The current design is documented in the Aiden memory plane.

The lesson:

Context management is part of Agent architecture, not cleanup performed after the architecture is finished.

Refactor 5: From voice input to foreground and background Agents

Adding speech to an Agent looks easy when voice is treated as another input method:

speech -> STT -> Agent -> TTS

That works until the Agent starts a multi-step device task.

A device task may need to capture several screens, call tools, wait for UI changes, and recover from unexpected states. Meanwhile, the person may want to interrupt:

“Stop.”
“Use the other account.”
“What is happening?”
“I’ll enter the password myself.”

If conversation and execution share one loop, the conversation waits for the task. If every backend event is immediately sent to the voice model, the Agent can interrupt itself.

We rebuilt realtime voice around two cooperating Agents:

a foreground Realtime Agent that owns the live conversation;
a backend Agent that executes device tasks;
an asynchronous task queue connecting them.

The task manager uses an explicit lifecycle:

type Status string

const (
    StatusCreated    Status = "created"
    StatusQueued     Status = "queued"
    StatusRunning    Status = "running"
    StatusCancelling Status = "cancelling"
    StatusCancelled  Status = "cancelled"
    StatusFailed     Status = "failed"
    StatusCompleted  Status = "completed"
)

The status describes execution state. It does not prove that the user’s intended goal was achieved.

The foreground Agent can create, query, or cancel background tasks. It does not need to wait for the backend Agent to finish before continuing the conversation.

Human handoff also became part of the task lifecycle.

If a password, CAPTCHA, payment confirmation, permission dialog, or ambiguous decision requires a person, the backend task can publish a pending action and remain in the running state.

After the person finishes, the foreground Agent returns a continuation message. The manager resumes the same task identity and backend context instead of starting an unrelated task.

We also added backpressure to result delivery. Terminal task updates use a 500-millisecond sliding debounce window, and the runtime injects them only when the foreground is idle.

This prevents several nearby results from creating several competing spoken responses.

The first full-duplex implementation landed in PR #566. Realtime provider adapters and the expanded foreground tool surface followed in PR #620 and PR #623.

The architecture is described in Foreground and Background Agents.

The lesson:

A voice Agent operating a physical device needs ownership and scheduling rules, not only lower audio latency.

Refactor 6: From successful demos to saved execution evidence

A demo video tells you that one task worked once.

It usually does not tell you:

which tools were called;
whether the Agent used the intended input path;
whether the screenshot was fresh;
what the screen looked like before and after;
whether the final response matched the physical outcome;
whether the result can be reproduced.

We built the benchmark system around an Environment Bridge contract:

Runner
   |
   v
Go Agent
   |
   v
Environment Bridge
   |
   +-- physical device
   +-- MobileGym / ADB
   +-- desktop environment
   +-- custom bridge

For each task, the runner can:

prepare an isolated environment;
capture a pre-task screenshot;
send the task to the real Agent;
capture a post-task screenshot;
extract the structured tool trace;
run deterministic assertions;
optionally evaluate the saved evidence with a judge;
retain the artifacts for later analysis.

Execution and scoring are separate. A rubric or judge can change without forcing the physical task to run again.

We also added:

a hardware-free Docker sandbox;
centralized benchmark configuration;
local low-level smoke tests;
macOS, Linux, and Windows desktop bridges;
environment health checks;
isolated Agent workers for concurrent environments;
separate pre/post screenshots and structured traces.

The benchmark now asks a better question than “Did the demo look good?”

It asks:

What did the Agent observe, what action did it take, what changed, and what evidence supports the reported result?

Relevant changes include PR #523, PR #533, PR #569, PR #593, and PR #612.

The current flow is documented in the benchmark architecture.

Some refactors made Aiden smaller

Not every important change added a feature.

We removed several ideas after they stopped matching the system we were building:

reverse platform inference;
automatic upfront memory injection;
unused chat-history mirrors;
duplicated configuration paths;
obsolete script and image-diff tools;
tool descriptions containing platform policy that belonged in configuration;
assumptions that a completed loop implied a successful physical outcome.

Removing a tool can look like lost capability.

For an Agent, it can be the opposite. Every exposed tool expands the model’s decision space. An ambiguous or obsolete tool makes the runtime harder to understand and the Agent harder to control.

One example is PR #601, which removed script and image-diff tools while preserving post-action screen-change detection as a narrower capability.

The lesson:

A smaller, explicit tool surface can be more capable than a larger, ambiguous one.

What the six months changed

The visible result is a larger system, but the more important change is in where responsibilities now live.

Earlier assumption	Current direction
Any process can capture a frame	One service owns each hardware resource
Tools decide platform behavior	Configuration selects an explicit provider
A written HID report means success	Observe and verify the resulting screen
Configuration is a file-editing problem	The Agent owns the runtime contract
Context is one transcript	Conversation, state, memory, and evidence are separate
Voice is another input mode	Foreground conversation and backend work are decoupled
A stopped loop means success	Execution state and physical outcome are different
A demo is enough evidence	Save screenshots, traces, assertions, and reports
More tools mean more capability	Smaller boundaries can improve reliability

None of these principles is unique by itself.

The difficulty is applying all of them at once while a model is interacting with a real screen, an operating system, a USB stack, and hardware that can disconnect.

A physical Agent is not only an LLM with tools.

It is a distributed system squeezed onto a small device.

Where the project stands

The current Aiden development board can combine:

HDMI screen observation;
keyboard, pointer, touch, and auxiliary USB HID control;
USB ECM networking;
configurable model, STT, and TTS endpoints;
voice interaction;
skills and several kinds of memory;
foreground and background task execution;
human handoff;
OTA and diagnostics;
device and desktop benchmark environments.

There are still real boundaries.

The basic control path requires a target capable of video output and USB HID input. iOS requires AssistiveTouch. Optional Phone Bridge, notification, BLE, and ADB paths have their own setup requirements.

Compatibility still depends on the target device, operating-system version, capture bridge, USB behavior, audio path, permissions, and configured inference endpoints.

The current repository should be read as a development-board implementation and an invitation to inspect the engineering—not as a claim that every supported-looking combination has already been validated.

If you are building voice Agents, GUI Agents, embedded controllers, or hardware automation systems, we would especially value reports that include:

the board and firmware revision;
the target device and OS version;
the task prompt;
the configured model endpoint;
screenshots before and after;
the structured tool trace;
interruption or handoff behavior;
what you expected and what actually happened.

You can explore the project through:

We are still refactoring Aiden.

That is not because the original idea stopped working.

It is because “the Agent moved the cursor” and “the system can explain what it observed, what it did, what happened next, and why it stopped” are two very different engineering milestones.

We Tried to Make a Voice Agent Control a Real Device. Here's What Broke

Dan — Wed, 09 Sep 2026 07:47:45 +0000

Most voice-agent demos have a simple loop:

The user speaks.
The model thinks.
The agent replies.

That loop works well when the agent only needs to answer a question.

It becomes much harder when the agent is also controlling a real device.

Aiden can read a phone’s screen and operate it through an external hardware control path. A task might involve several screenshots, tool calls, taps, swipes, and waits for the device to respond.

While that is happening, the person may still want to speak:

“Stop.”
“Actually, open the other app.”
“Wait, I’ll handle this part.”
“What is it doing right now?”

Our first instinct was to treat voice as another input method around the existing Agent loop. That turned out to be the wrong abstraction.

The difficult part was not streaming audio. It was deciding who owns the task, when messages should be delivered, and how to keep two different kinds of Agent work from interfering with each other.

Lesson 1: Don’t make the conversation own the device task

A long-running GUI task and a real-time conversation have different timing requirements.

The device task needs to:

inspect the screen;
call tools;
send input;
wait for the device;
inspect the new screen;
continue or recover.

The voice interaction needs to respond quickly and remain available for interruptions.

Putting both inside one loop created predictable problems. The conversation had to wait for device work, and interruptions arrived too late to be useful.

The solution was to split the responsibilities:

The foreground Realtime Agent handles listening, short conversational responses, and interruption.
The backend Agent handles screen understanding, tool calls, and device actions.
An asynchronous task queue connects the two.

The foreground Agent is not a second GUI operator. It is an interaction layer that can start, cancel, query, or redirect work handled by the backend.

This separation also means that the existing backend execution path can be reused across different voice modes instead of implementing device control twice.

Lesson 2: “Completed” does not mean “Successful”

This sounds obvious, but it matters a lot in an Agent system.

A task can finish because the execution loop reached its end state. That does not necessarily mean the intended result happened.

For example, a task might:

open an app but fail to find the expected screen;
send an input event but receive an unexpected result;
stop after hitting a safety or loop guard;
wait for a human action and never receive it.

We therefore keep execution state and outcome separate:

Completed means execution ended.
Failed means an abnormal condition interrupted execution.
The task result explains whether the intended action was actually achieved.

Without this distinction, the conversational Agent is encouraged to turn “the loop stopped” into “the task succeeded.” That is a small wording problem in a demo and a serious reliability problem in a physical system.

The distinction also appears directly in the task model. Execution status, result, error, and pending human action are stored separately:

type Status string

const (
    StatusCreated    Status = "created"
    StatusQueued     Status = "queued"
    StatusRunning    Status = "running"
    StatusCancelling Status = "cancelling"
    StatusCancelled  Status = "cancelled"
    StatusFailed     Status = "failed"
    StatusCompleted  Status = "completed"
)

type Task struct {
    ID                string      `json:"id"`
    Prompt            string      `json:"prompt"`
    Status            Status      `json:"status"`
    Result            string      `json:"result,omitempty"`
    Error             string      `json:"error,omitempty"`
    CreatedAt         time.Time   `json:"created_at"`
    UpdatedAt         time.Time   `json:"updated_at"`
    StartedAt         *time.Time  `json:"started_at,omitempty"`
    CompletedAt       *time.Time  `json:"completed_at,omitempty"`
    PendingUserAction *UserAction `json:"pending_user_action,omitempty"`
}

The important part is that StatusCompleted only describes the execution lifecycle. The actual outcome still has to be understood from Result and the observed device state.

View the task model on GitHub

Lesson 3: Notifications need backpressure

The backend Agent can produce several events in a short period: tool results, screen changes, task updates, or requests for human input.

If every event is immediately injected into the foreground conversation, the model receives a noisy stream of updates. Worse, a background result can arrive in the middle of an answer and compete with the person’s current request.

We added a notification queue with a short aggregation window. When several task results arrive close together, the runtime waits 500 milliseconds, extending the window if another result arrives, and then delivers the combined update.

We also avoid injecting a background task message while the foreground Agent is actively answering.

This is not only a queueing detail. It changes how the interaction feels. A real-time Agent should not sound as if it is being interrupted by its own internal logs.

Here is the less glamorous—but surprisingly important—part of the implementation. Instead of forwarding each completed task immediately, the runtime resets a short debounce timer whenever another result arrives:

case <-agentTaskNotifications(tasks):
    pendingTaskUpdates = append(
        pendingTaskUpdates,
        tasks.DrainTerminalTasks()...,
    )
    taskUpdatesReady = false

    if taskDebounceTimer == nil {
        taskDebounceTimer = time.NewTimer(
            realtimeTaskResultDebounce,
        )
    } else {
        if !taskDebounceTimer.Stop() {
            select {
            case <-taskDebounceTimer.C:
            default:
            }
        }
        taskDebounceTimer.Reset(realtimeTaskResultDebounce)
    }

    taskDebounce = taskDebounceTimer.C

realtimeTaskResultDebounce is currently set to 500 milliseconds. It is a small delay, but it prevents several closely spaced backend events from turning into several competing voice responses.

View the debounce interval and the notification queue logic.

Lesson 4: Keep the two contexts separate

The foreground and backend Agents should not share one constantly changing transcript.

The backend needs detailed device state and tool history. The foreground needs enough information to explain what is happening and respond to the person. Mixing everything together makes both contexts harder to manage.

Aiden uses two runtime message types to communicate across the boundary:

StateMessage carries current device and runtime state.
NoticeMessage carries events generated by the runtime, such as task results, loop-guard corrections, or requests for human action.

For example, a new screen observation can be added as state without rewriting the system prompt. A completed backend task can be delivered as a notice without pretending that the person said it.

At the model boundary, these runtime messages are converted into ordinary user-message content. The runtime still controls where the information came from and when it should be delivered.

Lesson 5: Human handoff is part of the normal lifecycle

A physical-device task will sometimes need a person.

Maybe the device requires authorization. Maybe the user needs to choose an account. Maybe the Agent reaches an action that should not be automated without confirmation.

Instead of treating this as an exception outside the task system, Aiden gives it an explicit lifecycle:

The backend calls request_user_action.
The foreground Agent asks the person.
The foreground Agent returns the answer through response_user_action.
The backend continues with the same task identity and device context.

The task remains managed while it waits.

We also keep device execution serial for now. Multiple tasks competing for one screen and one input path create ambiguous ownership: which task owns the next screenshot, or the next tap? Serial execution is less exciting than parallelism, but it keeps observation and action understandable.

When the person provides the missing information, we do not create an unrelated replacement task. The existing task is returned to the queue with a continuation message:

if item.task.Status != StatusRunning ||
    item.task.PendingUserAction == nil {
    m.mu.Unlock()
    return Task{}, errors.New(
        "agent task is not waiting for user action",
    )
}

select {
case m.queue <- taskID:
    item.task.PendingUserAction = nil
    item.actionNotified = false
    item.nextPrompt = userMessage
    item.resumeQueued = true
    item.task.UpdatedAt = m.now().UTC()

    task := item.task
    m.mu.Unlock()
    return task, nil

default:
    m.mu.Unlock()
    return Task{}, errors.New("agent task queue is full")
}

This lets the workflow preserve its task identity while making the pause and resume behavior explicit.

View the human-handoff continuation code on GitHub

What we are still validating

This implementation is part of Aiden’s open-source development-board firmware. It is not a claim that every board, phone, operating-system version, or audio setup behaves identically.

The actual experience still depends on:

the target device and OS;
the audio and trigger path;
the configured model endpoint;
screen-capture and input hardware;
the specific task being tested.

Useful tests include speaking while audio is playing, interrupting a running task, canceling and querying task state, waiting for human input, and checking that only one device task owns the control path at a time.

The full-duplex architecture is still evolving, but the main lesson has held up:

When an Agent talks to a person and operates a real device at the same time, the core problem is not just voice latency. It is ownership, scheduling, context, and honest task state.

The implementation is available in the Aiden firmware repository. The deeper architectural write-up is available on the Aiden blog.

If you are building a voice Agent that can also take real-world actions, I’d be interested to hear how you handle interruptions, task ownership, and human handoff.