DEV Community

Space App
Space App

Posted on Edited on Fully Autonomous

Our agent reviews its own mistakes every day. Here is the loop, and the tests it produced

I build SpaceApp, a self-hosted workspace where AI coding CLIs run side by side in persistent rooms. Nous is the part of it that sees every room and pane (earlier write-up: Nous: one mind over every AI coding CLI), does the routine work itself and brings me only the decisions that need me.

An agent that does routine work makes routine mistakes, and the same ones come back. This post is about the loop that catches the repeats: a daily self-review that turns them into code changes, each with a test. Every part of it is in the public repository, so you can read the code instead of taking my word for it.

Why the review is no longer a cron job

The first version was a scheduled task typed into a pane once a day. It kept getting lost: the pane it was bound to closed, and the review was skipped on 21 and 23 October. Since 24 October, Space itself starts the review as a turn of its own, once per local day at or after the hour you choose, and skips it when a review already ran that day.

Source: daily-review.ts and standing-orders.ts.

What Nous reads

The review prompt is built from figures Space counts, not from the agent's own account of its day:

  • its turns in the last 24 hours, and how many failed, timed out or were stopped;
  • its tool calls, split into mistakes and calls a gate held back on purpose;
  • every tool that failed, as "name, failures of calls";
  • the decisions the owner answered, how many were marked "you could have decided this", and the cards answered against Nous's recommendation;
  • the owner's corrections.

The split between mistakes and gate refusals matters. Without it, a safety rule doing its job would show up in the review as a bug to fix.

What it has to produce

Five orders, in this order:

  1. Read its scorecard and the approved actions that failed.
  2. For each repeated cause, update a "lessons" note: what went wrong, why it mattered, what to do instead. Merge duplicates; delete a lesson that turned out wrong.
  3. Save one memory entry for the day with the figures and the lessons it changed.
  4. Append one line per repeated cause to a queue file: OPEN <day> REVIEW: <cause>; evidence: <tools and counts>; fix: <the code change you want> Nothing when no cause repeats.
  5. End with five lines: turns and failures, lessons changed, what it will do differently, and what needs the owner.

A lesson in a note helps only while the agent reads it. A line in the queue asks for a change in the code, so the next turn, or the next model, does not have to remember.

What the fixes look like

The queue is worked in separate upgrade rounds. The fixes that came out of it carry a test named after the review or queue item that found it, in nous-self-upgrade.test.ts. A few of them:

  • A database read failed on a wrong column, and the error was cut off before the list of real columns (queue item 26). Now the whole answer comes back, real column names included, even past 1,000 characters, so the next query can be right.
  • A step sent "now" to a busy pane failed. Now it goes after that pane's current turn and says so. A pane that is not at a prompt gets the text held until it is ready, instead of a raw refusal.
  • A pane was looked up in the wrong room (daily reviews of 22 and 24 October). The answer now names the rooms where that pane is open, and a pane id that had been tried in four rooms is read in the room it is actually in.
  • Text held by the safety guard (daily review of 20 October, queue item 28). Sending it again holds it again, so the answer now tells Nous to ask the owner, and names the words that read as destructive so it can rephrase. The guard's threshold stays where it was.

One group of tests is titled "errors that say the next step" (daily review of 21 October): a shell terminal with no AI CLI is refused before anything is sent, with "do not send again"; a copilot pane points to the copilot's own tools; an answer that left the screen unchanged says "read the pane before you answer again; never repeat the same answer". (For background on how tools and terminal sessions are isolated across 11 CLIs, see our earlier article: One tool layer for eleven AI coding CLIs).

Reading the file back, most of these fixes are not clever. They are error messages rewritten so the next step is inside the message. A bare "failed" gives a model nothing to do except try the same thing again.

Limits

  • Nous is off on a new install; you turn it on with Work on my behalf in its dock. On a public install the daily review is a standing order that is also off by default.
  • The review only sees repeats in its own turn figures. A mistake that happened once, or one nobody corrected, does not show up.
  • The review writes a lesson, a memory entry and queue lines. It changes no code itself; the code changes come in the upgrade rounds.

Try it

You need Linux, macOS or Windows 11, Node.js 20.11 or newer, and at least 4 CPUs, 8 GB RAM and 15 GiB of free disk:

npx --yes run-spaceapp@latest install
Enter fullscreen mode Exit fullscreen mode

SpaceApp is MIT-licensed. Code and docs: https://github.com/oll4com/spaceapp ยท site: https://spaceapp.dev

If you run agents for long stretches: what do you count as a mistake, and what do you count as a guard doing its job?

Top comments (0)