Building a Flutter client for Hermes Agent on a foldable

One of us runs about a dozen Hermes Agent profiles on a home server: a researcher, a marketing bot, a health bot and more. Hermes can reach you through Telegram, Discord and other messaging gateways, and that works. We wanted something else on the phone: every bot in one list, the agent's approvals and questions as real controls, and a way to see what a bot is doing on its own desktop.

So we built Pomelu, an open-source (Apache-2.0) Flutter app that connects to your own Hermes gateway. Each Hermes profile becomes a bot in a sidebar with one chat. This post covers the parts that took the most work. The main test device was a Galaxy Z Fold.

The app is a thin client

Pomelu has no agent logic and no server of its own. Hermes Agent ships a gateway (hermes dashboard) with a WebSocket at /api/ws that speaks JSON-RPC, plus REST routes for bots, skills and cron jobs. The app uses those and nothing else. There is no account, no analytics and no Pomelu backend; a message goes from the phone to your gateway and back.

The code has four layers, and dependencies point down:

presentation   widgets, screens, Riverpod notifiers   lib/features/*/presentation
domain         pure rules and models (no I/O)          lib/features/*/domain
data           stores, platform bridges, file work     lib/features/*/data
hermes SDK     transport, JSON-RPC, typed API, events  lib/hermes

lib/hermes/ is pure Dart with no Flutter import, so the protocol code is tested without a device. Interfaces exist only where something real is swapped: the WebSocket (WsConnection, a fake socket in tests), key-value and secret storage, the share bridge, notifications, the screen's web view.

Sign-in uses the dashboard's own password login. The gateway returns a short-lived access token and a refresh token. For the socket, the client asks for a single-use ticket right before every dial (it expires after 30 seconds), so a reconnect never reuses one. The credential is bound to the origin that issued it, and the app refuses user:pass@ URLs. If you point it at plain http:// on a public host, it shows "Not encrypted" and asks before a password goes out. Home users often reach their gateway by IP over http on the LAN or over Tailscale, so the app allows cleartext but says so.

A turn must never be sent twice

A phone loses its connection all the time: lifts, tunnels, Wi-Fi to mobile data, the OS killing the app. For a chat app that means a resend button. For an agent that can run shell commands, sending a prompt twice can mean doing the work twice.

So every message goes through an outbox that is stored on the device:

1. send writes an entry to the store and shows the row at once.
2. Entries go out oldest first, one at a time, and only while no reply streams.
3. Files are read and uploaded first; a restart at this point re-queues safely.
4. The entry is marked submitted before prompt.submit is written to the socket.

The interesting case is step 4. If the frame never left the phone, the error is a NetworkFailure and the entry goes back in the queue (up to five tries). If the frame was written and then the socket dropped, the outcome is unknown: the gateway may have the prompt. That error has its own type, OutcomeUnknownFailure, and a non-idempotent call is never resent blindly. After the next connect, the app compares the entry with the session history and the gateway's in-flight and queued user messages. Found means delivered. Not found means the row says "delivery unconfirmed" and waits for you. A failed turn also holds the queue, because order is part of a conversation.

The queue survives process death. On start and on every reconnect, a small drainer wakes the chats that still have work, so a message you wrote in a tunnel goes out even if you closed that chat.

Streaming Markdown without re-laying out the whole reply

Hermes streams message.delta events while the model writes, plus tool events and the agent's todo plan. Events flow from the socket through the JSON-RPC client (with sequence de-duplication and replay after a reconnect) into a pure reducer that builds the chat state.

Long agent answers were the performance problem. Agents write long replies with tables and code, and re-parsing and re-laying out the whole reply on every delta gets slower as the reply grows. The fix: a reply longer than 800 characters is cut into several list rows at Markdown block boundaries. While the reply streams, only the last row rebuilds; the rows above it are finished blocks that never change. The list itself is a reversed lazy list, and large JSON payloads are decoded on another isolate.

Two smaller details: reply direction is read once from the prose, so a Persian or Arabic answer is laid out right to left as a whole while code blocks stay left to right. And while a turn runs with a plan, a row above the composer says "Plan · 2 of 5" with the step in progress; a tap shows every step.

Questions from the agent (clarify, approval, sudo, secret input) are JSON-RPC requests from the server, not events. A router parks them per session and the chat shows the oldest one as a card above the composer. The router, not the chat state, is the source of truth, so a question survives a closed chat and comes back after a reconnect.

One layout for folded, unfolded and tablet

We did not want a "foldable mode". The layout is chosen by the window's size alone:

• Below 640 dp the chat is full screen and the bot list is a drawer.
• From 640 dp there are two panes: a sidebar (or a rail of avatars) next to the chat.
• From 1024 dp the sidebar gets wider; the chat column keeps a reading width and stays centred.

Hinges were the subtle part. Flutter exposes them in MediaQuery.displayFeatures. Only an occluding vertical hinge moves the split between the panes, so the hinge becomes dead space between sidebar and chat. A seamless fold, like the Z Fold's inner screen, is ignored; snapping to it would only make the sidebar too wide.

Folding and unfolding must not cost you anything. On Android the activity declares configChanges for fold, resize and density, so a posture change never recreates it. In Flutter, each chat has one GlobalKey, so when the layout switches between drawer and two panes the same chat widget moves instead of being rebuilt: scroll position, the half-typed message and attached files stay.

On iOS the iPhone Duo was a surprise: iOS gives Flutter no fold at all (displayFeatures is empty), so width alone decides there, and its safe-area insets are not symmetric. We test at 360, 640 and 840 dp, at iPad and Android-tablet sizes, and on the Duo closed and open.

With a hardware keyboard, Return sends, Ctrl+Tab steps through bots, Cmd or Ctrl with 1 to 9 opens a bot, and a right-click opens a message's actions at the pointer.

Notifications without a push server

Push notifications normally need a server that holds device tokens. Pomelu has no server, and we did not want to add one, so every notification is local:

• With the app open or connected, the app listens for gateway signals (a reply landed, a background job finished, a question arrived) and shows a notification unless you are looking at that chat.
• On Android, an optional "Stay connected" foreground service keeps the socket alive. Without it, Android 15 and later cut the app's network a few seconds after it leaves the screen. After 10 minutes unreachable in the background, it stops redialling until the network changes or you open the app.
• A background check runs every 15 minutes (WorkManager on Android, background app refresh on iOS when iOS allows). It dials with the stored credential, compares each bot's message count with what it saw when the app left the screen, and rings once per bot with the newest message.

On iOS this is best effort: the system suspends the app soon after it leaves the screen. That is the honest trade-off of having no push server.

A bot's desktop on a phone

Hermes has a feature called Bot Screen: on a Linux gateway host, each bot gets its own Xfce desktop that its computer_use and browser tools act on. When a bot hits a login, a 2FA prompt or a captcha, a person can take over, finish the step, and hand control back; the bot continues in the signed-in session.

Pomelu shows that desktop in a web view. The page loads under the gateway's origin and runs noVNC 1.7.0, which dials the stream itself with a single-use display ticket. The app keeps the bot's status and the control lease from display.status and display.lease events, and ignores an older lease epoch.

Making a remote desktop usable on a 6-inch screen took most of the effort:

• On a tall window (portrait, a folded cover screen) the desktop fills the height, and the view follows the regions noVNC redraws, which is where the bot is working. A pinch stops following; a double tap goes back.
• In control, one finger is the mouse and two fingers scroll. Drag and drop has its own mode: tap what to move, pan or zoom freely, tap where it goes, and the pointer moves there in steps before the release, because desktops only start a drag on motion.
• Typing goes through a hidden field with no autocorrect, and its changes become key presses. A key row adds Esc, Tab, Ctrl, Alt and arrows. Paste puts the phone's clipboard on the desktop's.
• Control goes back to the bot when you close the page. If you switch apps to copy a 2FA code, the app keeps control for a short while, then hands it back only if a fresh status shows it still holds it, never over someone who took over in the meantime.

What it needs, and what is missing

• Your own Hermes Agent 0.21.5 or newer, started with hermes dashboard --host 0.0.0.0 --no-open and a dashboard username and password, reachable from the phone over the LAN, Tailscale or HTTPS.
• Bot Screen needs a Linux gateway host with TigerVNC and Xfce; macOS and Windows hosts do not offer it.
• On Android (10+) it is on Google Play, and a signed APK is on GitHub. The two builds have different signatures, so uninstall one before you install the other.
• The App Store build is coming soon; until then iPhone and iPad users can build from source.

Try it

• Google Play: play.google.com/store/apps/details?id=se.royan.pomelu
• Android APK (arm64-v8a or universal, with SHA-256 sums): github.com/ateymoori/pomelu/releases/latest
• Source, architecture and protocol notes: github.com/ateymoori/pomelu (docs/ARCHITECTURE.md, docs/PROTOCOL.md)

Pomelu is a community project by Royan AB in Stockholm, not affiliated with Nous Research. Bug reports from phones other than a Samsung foldable are the most useful thing you can send us.

Written by the Royan Labs team with help from an AI assistant.