We Sent 10 AI Agents After Bugs That Take Weeks to Appear. 26 Were Real

• Hermes IDE: Download free · Star it on GitHub — one console for every coding agent, on macOS, Windows and Linux
• Book: AI Agents Pocket Guide · AI in TypeScript — the 5-book series
• Me: xgabriel.com | GitHub

---

Some bugs need three weeks of uptime to show up. We didn't want to wait three weeks.

You know the shape. The app passes every test and survives a day of use. A month later an issue arrives: the window hangs on resize, the disk is full, a cost figure makes no sense. Nobody can reproduce it in a ten-minute session.

I build Hermes IDE, a desktop app on Tauri 2, Rust and React where coding agents like Claude Code, Codex and Antigravity CLI run side by side, each in its own terminal and usually its own git worktree. It is built to stay open all day while agents work. Hermes 2.0 shipped on October 2, 2026, and 14 more releases followed by October 11. At that pace, waiting for weeks-of-uptime bugs to surface by themselves was never going to work.

So for 2.2.9 we went looking for those bugs on purpose, by reading the code.

Hunters and skeptics

Five hunter agents read the codebase, and five skeptic agents tried to knock down what they found. A finding only counted if a skeptic confirmed it in the code and wrote down a fix. The two briefs, condensed:

HUNTER
Question: what goes wrong if this app stays open
for days or weeks?
Report: file and line, the mechanism, and a
concrete scenario.

SKEPTIC
Trace the code yourself and try to refute the claim.
Confirm only what the code shows, trim what was
overstated, and write down a fix.

34 findings went in. 26 came out confirmed.

The skeptics worked in both directions. They cut claims that had grown in the telling. One hunter predicted launches 15 to 40 seconds slower; the skeptic traced the code and put it at about 100 to 300 ms per leftover folder. Another warned that turn snapshots would pin gigabytes of build output in .git; the skeptic noted the snapshot uses git add -A, which respects .gitignore. They also pushed findings up. Four hunters reported the same worktree watcher bug, and a skeptic still said they had undersold it. Most of the gap between 34 and 26 is duplicates like that one, several hunters reporting the same bug, rather than findings the skeptics threw out.

Of the 26, 24 are fixed in 2.2.9. Six fix agents wrote them in parallel, each on its own branch, and the fixes were cherry-picked onto the release branch. One finding, Agent view's memory growth, is deferred. One fix, for a Windows CPU drain, was reverted before release, so that finding is still open. The 24 fixes fall into six patterns, and a ten-minute test run hides every one of them.

1. One lock for three jobs

Symptom. Paste a long log into a terminal pane running tail -f or a dev server, then resize the window. The whole app freezes and never comes back.

Mechanism. On macOS and Linux, Hermes runs terminals in a separate host process, so agents keep running when the app quits. Each session had one connection to that host, guarded by one mutex. Input, resize and kill all used it. A program that doesn't read stdin stops draining its input. The PTY buffer fills (the code notes about 1 KB on macOS). The host's write blocks, so the host stops reading frames. The socket buffer fills, and the app's input thread blocks while holding the mutex. Then you resize. Resize is a synchronous Tauri command, so it runs on the main thread. It takes the session manager lock and waits on that mutex for good. Close and quit wait the same way.

Why short tests miss it. Nobody pastes an 80 KB log into tail -f during a test. It's an accident that happens late in a long day.

Fix. The host now writes each session's input on a thread of its own, fed by a queue. A blocked write stalls only that thread, and the host keeps reading resize and kill frames. On the app side, when input holds the connection, resize and kill travel over a separate connection. The host half, simplified:

use std::io::Write;
use std::sync::mpsc::{channel, Sender};

fn input_thread(
    mut pty: Box<dyn Write + Send>,
) -> Sender<Vec<u8>> {
    let (tx, rx) = channel::<Vec<u8>>();
    std::thread::spawn(move || {
        for bytes in rx {
            // Blocks while the program ignores stdin.
            // Only this thread waits.
            let _ = pty.write_all(&bytes)
                .and_then(|_| pty.flush());
        }
    });
    tx
}

Check this in your app. List every lock that a blocking write holds. If a control path like cancel, resize or shutdown shares that lock, one slow consumer can freeze the control path.

2. Rows that multiply

Symptom. Cost totals keep climbing while a session sits open, and the database grows with them.

Mechanism. Hermes auto-saves the workspace every 10 seconds while anything changes, and a terminal printing output counts as a change. Each save wrote every live session's token usage into a token_usage table with a plain INSERT. Those values were running totals. The cost views then summed the rows. Five busy sessions over an eight-hour day add 14,400 rows, and each one repeats usage that was already counted.

Real Hermes window, test build with stand-in agents. Cost totals like these are summed from stored usage rows, which is where the duplicates crept in.

Why short tests miss it. One save writes a correct row. Ten minutes write sixty, and the total looks a little high, if anyone checks. The bug lives in the sum over time.

Fix. Each session now has one row per provider, enforced by a unique index, and each save adds only what was used since the previous save. A migration first folds the old duplicates into one row per key, keeping the largest totals each session reached, and then creates the index. Simplified:

CREATE UNIQUE INDEX IF NOT EXISTS idx_token_usage_session_provider
  ON token_usage(session_id, provider);

-- ?3..?5 are deltas since the last save, not running totals
INSERT INTO token_usage
  (session_id, provider, input_tokens, output_tokens, cost_usd)
VALUES (?1, ?2, ?3, ?4, ?5)
ON CONFLICT(session_id, provider) DO UPDATE SET
  input_tokens  = input_tokens  + excluded.input_tokens,
  output_tokens = output_tokens + excluded.output_tokens,
  cost_usd      = cost_usd      + excluded.cost_usd;

Check this in your app. For every table written on a timer, ask what makes a row unique. If the answer is "nothing", the table grows forever.

3. Cleanup that doesn't reach everything

The audit turned up several "the cleanup exists, but" bugs. The worst involved turn history. Hermes saves every agent turn as a git ref under refs/hermes/<session>/, and a 14-day collector is supposed to delete those refs. Refs in that namespace are shared, so for a worktree session they live in the main repository's .git. The collector looked for the repository in the session's worktree folder, which had been deleted when the session closed. It found no repository and deleted no refs. Then it deleted the database rows anyway. With the rows gone, nothing would ever try again. Sessions that ran in the project folder itself were collected correctly, which hid the bug in casual testing.

The fix also looks in the session's projects and in the repository the worktree came from. It keeps the rows until it has reached a repository and deleted the refs, so a later run retries.

Smaller versions of the same pattern, all fixed in 2.2.9:

• Review desk state was saved in the webview's local storage under each worktree's path and never removed. Once the storage quota filled, review progress and preferences would have stopped saving, with no error. It is bounded now.
• The background terminal service's log only grew. It rotates at 1 MB.
• Closed sessions kept their saved scrollback forever. It is dropped after 30 days, except for the 20 most recently closed sessions and anything in the saved workspace.
• Pasted screenshots were pruned only at startup, so an app that never restarted kept them all. Pruning now runs daily.

Why short tests miss it. A cleanup that runs at startup, or after 14 days, never runs inside a test session.

Check this in your app. Does your cleanup delete its bookkeeping before it confirms the thing itself is gone? Does it ever run if nobody restarts the app?

4. Watchers that watch too much

Symptom. On Linux, editors and dev servers start failing with "System limit for number of file watchers reached". On every platform, memory creeps up after each closed task.

Mechanism. Hermes watches its worktrees folder so it notices when you delete a worktree by hand. It watched recursively. On Linux, every directory inside every worktree cost an inotify watch, node_modules and target included, and those watches come from a per-user limit that every other tool on the machine shares. The audit put one worktree with node_modules and a Rust target at roughly 18,000 folders, and many distros still default to 8,192 watches. Removals were worse. Every removed folder anywhere in the tree went into a debounce map that was never pruned, and it triggered a database query under the global DB lock. Closing a task deletes its whole worktree, so each close added thousands of entries that stayed until restart.

From hermes-ide.com: a recreation of the 2.0 app with demo data. Four agents, each on its own hermes/… branch in its own worktree folder. Before 2.2.9, the watcher followed every one of those folders down into node_modules.

Fix. Hermes now watches the base folder and each per-repo folder without recursion. It reacts only to removals of exact worktree roots, and it prunes the debounce map to its 500 ms window. One detail from the real change: adding a watch from inside notify's callback can deadlock its event loop, so events now go to a thread of ours over a channel. The filter and the debounce, simplified:

use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant};

// A worktree root is exactly {base}/{repo_hash}/{name}.
fn is_worktree_root(base: &Path, path: &Path) -> bool {
    path.parent().and_then(Path::parent) == Some(base)
}

struct Debounce {
    window: Duration,
    seen: HashMap<PathBuf, Instant>,
}

impl Debounce {
    fn first_in_window(&mut self, p: &Path, now: Instant) -> bool {
        let window = self.window;
        self.seen.retain(|_, t| now.duration_since(*t) < window);
        if self.seen.contains_key(p) {
            return false;
        }
        self.seen.insert(p.to_path_buf(), now);
        true
    }
}

Check this in your app. On Linux, compare these two numbers:

cat /proc/sys/fs/inotify/max_user_watches
find node_modules -type d | wc -l

If the second is a real fraction of the first, a recursive watcher on that folder is a problem waiting for a few more checkouts.

5. Polling that scales with sessions

Symptom. The fans spin and the battery drains with Hermes in the background, and it gets worse with every open session.

Mechanism. Every session row polled git status every 5 seconds. Nothing checked whether the previous walk was still running, and nothing paused while the window was hidden. Each worktree is its own folder, so 25 sessions meant 25 full status walks every 5 seconds, all day. A 30-second timeout wrapped each walk. In tokio, though, a timeout around spawn_blocking only stops the waiting. The blocking walk keeps going. On a big repository the walks overlapped and slowed each other down.

Two more pollers had the same shape. SSH sessions ran blocking ssh processes every 8 seconds from synchronous Tauri commands, which run on the main thread, so after sleep or a network change a dead link could freeze the window for up to about 45 seconds. A memory check read the whole OS process table on the main thread every 5 seconds.

Fix. Git status now runs one walk per folder at a time, pauses while the window is hidden, and catches up once when the window comes back. SSH queries moved off the main thread, with a timeout that kills ssh and keepalive options so a dead connection is noticed quickly. The memory check reads the process table off the main thread, and not at all while hidden. The git status guard, simplified:

const hidden = () => document.visibilityState === "hidden";

function poll(entry: Entry, force = false) {
  if (!force && hidden()) return;
  if (entry.inFlight) return; // the last walk is still running
  entry.inFlight = true;
  gitStatus(entry.sessionId)
    .then((status) => publish(entry, status))
    .catch(() => {})
    .finally(() => {
      entry.inFlight = false;
    });
}

document.addEventListener("visibilitychange", () => {
  if (!hidden()) entries.forEach((e) => poll(e, true));
});

Check this in your app. For every timer, ask three questions. Does it run while nobody is looking? Can it overlap itself? Does its timeout stop the work, or only the waiting?

6. Lifecycle leaks

These need a specific sequence of actions spread over time, which is why no test session hits them.

• A host that never exits. On Linux, close a terminal that left ssh -fN or cmd & disown running, and the host service never forgot that session. Linux doesn't revoke the terminal when the shell exits, so the session's output never ended. The host never emptied out, so it never exited, and it kept running its old binary across app updates. A session closed on request is now forgotten once its program has exited.
• A kept folder that disappeared. Close a session, choose "Keep worktree", keep working in that folder from another editor, commit and push. At the next launch Hermes saw a clean orphan and removed it, ignored files like .env included, with no backup. Kept worktrees are now marked, and every automatic removal path leaves them alone.
• A promised retry that never came. When a worktree removal failed on close (a locked file on Windows, say), a toast promised a retry at the next startup. Startup never retried. Now it does.

Check this in your app. For every "close", list what "open" started: threads, file handles, child processes, timers. Each one needs a matching stop on the close path.

The one we deferred

Agent view, the optional view that renders a Claude conversation instead of the terminal, keeps the whole conversation in memory, every tool result included. It recomputes over all of it on each streamed event. The skeptic confirmed the growth inside a live session and refuted part of the claim: restoring a session reads at most the last 8 MB of its transcript, so a restart does not reload everything. Agent view is opt-in, and terminal mode is the default. There was no small, safe fix for 2.2.9, so this one waits.

Then our fixes broke two things

We brought the six branches together on the release branch and ran the full CI, which includes end-to-end tests that drive the real app. It caught two regressions before anything shipped.

The status that froze. One confirmed finding was a Windows CPU drain: the focused terminal's foreground check runs every 300 ms, and on Windows each check reads the whole process table, even with Hermes in the background. The fix paused the check while the Hermes window had no focus. The pause also froze background status. You switch to your browser and wait for an agent to finish, and that is the moment you need status to keep updating. We reverted the change. That finding is one of the 26, and it stays open in 2.2.9 until it gets a different fix.

The cleanup that could have deleted work. This one came from the fix for the promised retry in the lifecycle section. It made startup retry failed worktree removals for closed sessions, with --force. Earlier versions had kept the record tying a closed session to its worktree, though, and some of those folders still held uncommitted work. On the first launch of 2.2.9, the retry could have force-deleted a closed session's worktree with that work inside. Startup now removes such a folder only when it has no uncommitted files and no commits of its own. Otherwise it leaves the folder and its record as they are.

The retry was correct for the state the new code creates, and wrong for the state older versions had left on disk. That kind of mismatch is easy to miss in tests written alongside the change. The full suite, run before every release, caught it.

A checklist for your long-running app

Copy this into your next review. Every item comes from a bug above.

LOCKS
[ ] No blocking write holds a lock that cancel, resize
    or shutdown also needs.
[ ] No blocking call runs on the UI or main thread.

STORAGE
[ ] Every table written on a timer has a uniqueness rule
    (upsert), or a retention rule.
[ ] Every cache, map and log has a bound or a pruning pass.
[ ] Cleanup deletes its bookkeeping only after the thing
    itself is confirmed gone.
[ ] Cleanup also runs in a process that never restarts.

WATCHERS
[ ] File watchers are scoped to the folders you need,
    not recursive over dependency trees.
[ ] Debounce maps prune entries older than their window.

TIMERS
[ ] Polling pauses while nobody is looking.
[ ] A poll can't start while the previous one runs.
[ ] A timeout cancels the work it guards.

LIFECYCLE
[ ] Every resource started on open has a stop on close.
[ ] Startup cleanup is tested against data that older
    versions left behind.
[ ] Anything removed automatically is checked for work
    that would be lost first.

---

If you build software that stays open

Hermes 2.2.9 ships the 24 fixes. The best test is the one we couldn't run before release: download Hermes IDE, leave it open for a week with your agents working, and watch what stays flat. It's free. If this audit was useful, a star on GitHub helps other developers find it, and watching the repo is how you hear about the next one. If you want to know where Hermes sits next to the tools you already use, Hermes IDE vs Cursor vs Warp vs Zed compares them side by side.

If something grows that shouldn't, or you know a long-run pattern this list misses, open an issue. That's where the next audit gets its questions.

If you're building agent systems of your own, my AI Agents Pocket Guide is the short read on the patterns.