It Looked Finished on Day One

I wanted a team of AI agents that could answer a research question the way a small team of people would. One plans the work, one searches the web and reads the sources, one writes it up, and one checks the write-up before I see it. I also wanted to watch them do it: which agent is working, what it's reading, what it handed to whom, and what it cost.

The app is called FORGE, and it runs on a small server in my house. This post covers how it got from an idea to something that works end to end, which took three days. Most of that time wasn't building features. It was finding the places where the app looked like it worked and didn't.

Day one: the part that looks finished

The first version was a dashboard for a system that didn't exist yet.

It had a workflow canvas where each agent is a card and each arrow is a handoff. It had a live event stream, a timeline, a replay scrubber, per-agent cost and token counters, an agent builder and a workflow designer. Behind it sat a simulator: fake agents on a fake clock, producing realistic-looking events so every screen had something to show.

One design decision from that first day held up for everything that came after. A run is a log of events, not a record that gets updated. "Researcher started", "tool call finished", "handed the notes to the writer": every screen is computed from that log. Live view and replay are the same code; replay just stops reading the log partway through. When real agents replaced the simulator later, the screens didn't change at all, because real agents emit the same events.

It looked gorgeous, and none of it was real.

Making the agents real

Real agents meant real model calls, and I wanted them cheap. I started with three agents: a researcher, a writer and a reviewer.

• Models: most agents run GLM-5.3 Flash through OpenRouter, at about $0.15 per million input tokens. The reviewer runs on Claude.
• Search: DuckDuckGo through the ddgs Python package. It's free and needs no key.
• Reading pages: a small script fetches a URL and pulls out the main text (trafilatura for web pages, pypdf for PDFs).
• A server: a small Node.js server runs the agents, stores every event, and streams them to the browser as they happen.

The first live runs failed in ways the simulator never could.

Every call timed out, instantly. The server has no IPv6. When a hostname has several addresses, Node.js tries them one after another and gives each attempt only 250 milliseconds by default, which isn't enough on this network. The fix was two lines: prefer IPv4, and give each attempt three seconds.

Some replies came back empty. The model had spent its whole output budget reasoning and had nothing left for the answer. It now gets one retry with double the room.

The researcher never stopped researching. It kept searching and reading until it hit the step limit, then had no notes. Now every tool result tells it how many rounds it has left, and the last round forces it to write up what it has.

Some websites never answered. One slow host could eat a minute. Page fetches now give up after 20 seconds, and a host that times out once is skipped for the rest of the run.

The honesty pass

With real agents working, I went through the app page by page and asked one question: is this real or simulated?

A lot of it was simulated and looked real. The model-provider page showed made-up usage. The tools page listed tools with success rates and response times for tools that had never run. The run history was full of sample runs.

Then I asked the team a real question and got a confident, well-formatted answer to a completely different question. The run had used the simulator, which produces plausible text and ignores the question entirely. It was the default for most workflows, and nothing on screen said so.

The rule I took from it: never ship a placeholder that looks real. Label it or remove it. Simulated data is now clearly marked everywhere. The simulator survives only as a "Dry run" button in the workflow designer, for checking the wiring for free. You can't start a simulated run from the New Run screen at all.

One store, not one per browser

I use FORGE from a desktop and a laptop, and they showed different data. Everything lived in each browser's local storage, so each device had its own private copy.

The server now owns a single SQLite database. Every change goes to the server, and every open tab gets pushed updates over a live connection. Each browser uploaded its old local data once, and the server merged it.

This surfaced a subtler bug. Run IDs had been picked by each browser, and the laptop picked an ID the desktop had already used, overwriting a run. Now only the server issues run IDs.

Spreading the cost

I added a planner in front of the researcher. It writes a short plan with sub-questions and search queries, costs about a hundredth of a cent, and makes the research noticeably more focused.

Then I checked the bill. One OpenRouter key was paying for everything, and it had $1.74 left. So Claude models now go straight to Anthropic's API, and everything else stays on OpenRouter. If Anthropic's key fails on billing, the call falls back to OpenRouter instead of failing the run.

The reviewer was the expensive part. I tested three reviewers on the same flawed draft, one with a real error: it credited a performance claim to the wrong source.

• Claude Sonnet, high effort: caught the error. About $0.03 per review.
• Sonnet, lower effort: approved the flawed draft. About $0.014.
• Claude Haiku: sent the draft back for other reasons but missed that error. $0.009.

I chose Haiku. The next section is what closed that gap.

Making it trustworthy

One run hit seven failed web searches in a row. An outside review of the logs blamed DuckDuckGo throttling and suggested switching to a paid search API.

The logs said otherwise. The search library rotates between several engines, and all four timed out in the same three minutes. The same queries worked in two to four seconds when I re-ran them. It was a short network outage on my side, and a paid API would have timed out too. So instead of switching providers:

• Search fails fast and retries: a 12-second limit, one retry, and after three failures in a row a 30-second wait for the network to come back.
• Not enough sources is flagged in code, not just the prompt: if the researchers read fewer than two pages, the answer carries a warning banner saying it's unverified. The writer is told too, but the flag doesn't depend on it listening.
• The reviewer checks the sources itself: it opens two or three of the pages the draft cites and checks they say what the draft claims. That's what the cheap reviewer was missing: a source it could check against. Pages the researcher already fetched are reused, so the check costs no extra time.

Reaching it from my phone

I already run a personal assistant agent called Hermes that talks to me on Telegram. FORGE and Hermes now connect both ways.

• FORGE → Telegram: every finished run sends the answer, cost, time and a link to the run page. Failed runs send an ❌ with the error.
• Telegram → FORGE: a /forge command in Hermes hands a question to the team. Hermes can only wait three minutes for a command, and a run takes longer. So the command starts the run and returns straight away, and the answer arrives on Telegram when the team finishes.

There are now three levels: quick (no reviewer, about half a cent, answers marked "not fact-checked"), verified (the default, two to four cents) and a larger six-agent team for deep questions. I pick the level myself. I considered a classifier that decides whether a question needs checking. It would have been cheap, but it would sometimes skip checking on exactly the question that mattered.

Letting the agents talk

Handoffs only go one way: the writer gets the notes, and that's it. If the notes leave a gap, the writer can only guess or leave it out.

So agents can now ask a teammate a question mid-task. The writer can ask the researcher, and the researcher answers from its notes or does one quick lookup. In one run the writer spotted two researchers giving different numbers for the same fact and asked one to settle it. The researcher went back to the source and answered with the bill that set the figure.

I also added a parallel team: a lead splits the question into three angles, three researchers work at once and post findings to a shared board, then the writer and reviewer finish.

Its first run failed at the 20-minute limit while every agent was busy. One researcher's final write-up had taken five minutes on its own, because the model spent it reasoning. My stall detector watched for a stream that goes quiet, and a reasoning model streaming its thoughts never goes quiet.

I measured the same three-sentence prompt. With no reasoning effort set, the model took 93 seconds and 3,200 reasoning tokens. With effort: low, it took 9 seconds. Leaving the setting out didn't mean a sensible default; it meant no limit. After setting it for every call, the same parallel question finished in 5 minutes 10 seconds for four cents.

Watching it from the outside

The last piece was monitoring. I have a dashboard, Operator Pulse, that shows every scheduled job on the server and alerts me when one fails. FORGE now appears on it as a service: whether the server is up, recent runs, success rate, and how much OpenRouter credit is left.

The first time the new FORGE node rendered, it was amber: $0.47 of credit left. That key also pays for Hermes, so both would have stopped working without warning.

FORGE can also run scheduled questions now. Each one is a small script that the dashboard tracks like any other job.

Where it ended up

A finished quick run: three agents, 1 minute 40 seconds, about a tenth of a cent. Every event on the right can be replayed.

Mode	What runs	Typical cost	Typical time
Quick	planner → researcher → writer	$0.005	1–2 min
Verified	+ a reviewer that checks cited pages	$0.02–0.04	1–4 min
Parallel	lead → 3 researchers at once → writer → reviewer	$0.04	~5 min

Every run is fully replayable. Every answer lists the pages it came from, and says so plainly when it couldn't verify enough of them.

What building it taught me

The dashboard was done on day one. It had every screen, every counter and every animation, and it was all simulated. Everything after that was the same job repeated: find a place where the app looks like it works and make it actually work.

Some of those places were loud: timeouts, empty replies, a run that died at twenty minutes. The ones that mattered most were quiet: a confident answer to the wrong question, two devices each sure their data was the real one, a timeout that never fired because the model never stopped talking, a credit balance nobody was watching.

A demo shows that something can work. Making it trustworthy meant finding every place it only looked like it worked.