We rebuilt our autonomous agent from scratch: from a lexical knowledge base to deterministic authority, and the results are not close

Kapil Soam — Tue, 14 Jul 2026 08:00:00 +0000

The first version of our autonomous agent at Eko worked, and I was the one who eventually decided to throw most of it away. This is the story of that call, what pushed us to it, and what we run now instead.

Some context. We build autonomous agents that resolve real operational work against a live financial system. Not a chatbot that drafts a reply for a human to approve, but a system that reads a request, decides what to do, and makes the actual change to production records on its own. When the thing you are automating can move money or change who controls an account, the bar is not "usually right." It is correct, provably, every time.

The first architecture: lexical matching and a knowledge base

Our first cut was a reasonable design. We built it as an OpenClaw agent. To work out what a request was asking for, we leaned on a lexical architecture: keyword and lexical matching to route an incoming ticket to the right kind of action. To give the agent the context it needed, we put our operational knowledge into a knowledge base and retrieved the relevant pieces at request time.

It worked. For a while it worked well enough that we were proud of it. And it taught us exactly where that shape of system breaks, which is the most valuable thing an early version can do.

Where it quietly broke

Two problems, and the second one is the one that changed my mind.

The first was authority. A lexical router plus a retrieval step is fine for reading. The moment the agent needs to write to a financial record, you have handed a probabilistic system the ability to take an irreversible action, and you are relying on prompt discipline and retrieved context to keep it in bounds. That is not a foundation you want under a regulated write path. There was no single place that held the authority and could be reasoned about, tested, and proven correct.

The second was subtler and worse. The system got less reliable the more we taught it. Every time we added a new kind of request, the lexical matching and the retrieval had more to sort through, and the right answer started getting crowded out. A keyword that used to point cleanly at one action now pointed at three. A retrieval step that used to surface the right knowledge now surfaced something adjacent and confident. We were investing effort into the system and watching it get worse, which is the opposite of what a learning system is supposed to do.

That is the moment you stop iterating and start rebuilding. Not because the first version was bad, but because its shape had a ceiling, and we had hit it.

The principle we rebuilt around

The rebuild is organized around one sentence: intelligence in the agent, authority in the API.

The model is allowed to be smart about understanding a request. It is not allowed to hold any power. It reads a ticket and produces a single typed proposal, nothing more. A separate deterministic core holds all the authority, and it does not trust the model. It validates the proposal, works out who the request is really about from the authenticated identity rather than the text, checks permission against the org hierarchy, writes an audit record before it does anything, and only then executes, with an idempotency key and a way to roll back.

That split is the whole thing. The smart part can be wrong without being dangerous, because the part that can actually act is boring, deterministic, and provable.

Compile the knowledge, do not retrieve it

The other big change is how the agent knows what it can do, and it is a direct answer to the "gets worse as you teach it" problem.

We stopped retrieving. Instead of a knowledge base we query at request time, we compile our operational knowledge offline into a single catalog, and at decision time we hand the model the whole thing at once, as a compact menu. It reasons over the entire set of possibilities every time and picks one, or picks nothing.

This sounds more expensive, and it is, by a couple of thousand tokens. It is worth every one of them. A retrieval step can hide the correct answer the day it ranks sixth in a top five, and you will never see it happen. A full compiled menu cannot hide anything, because nothing is filtered before the model reasons. Keyword signals still exist, but they only inform a confidence score now. They never decide what the model is allowed to consider. Teaching the system something new makes it better, not worse, which is how it should have worked all along.

The rest of the safety structure

Around that core we built the things a regulated write path needs and a lexical prototype never had. An append only, hash chained audit log that is written before any side effect, so if we cannot account for an action it does not happen. Exactly once execution with a compensating rollback. And a precision wall, a frozen set of cases the system must never act on, that every release has to score perfectly against or it does not ship. That last one is not a document we promise to uphold. It is a gate that runs the real code and turns red on its own if someone weakens a check.

The results

The honest summary is that the two systems are not comparable. Matching is more accurate and gets better as we teach it rather than worse. The write path is defensible in a way the first one never could have been, because the authority lives in code we can test and prove rather than in prompts we hope hold. And we run fully autonomously, with no human clicking go, precisely because the safety comes from the structure rather than from a person watching.

What I took from it as a lead

The lesson I keep coming back to is about timing. You do not rewrite because the first version is embarrassing. You rewrite when you can see the ceiling, and you can articulate why the shape of the thing, not the quality of the work, is what is capping it. Our first version was good work on an architecture that could not get where we needed to go. Naming that clearly, and having a team that could rebuild fast and carefully, is most of the job.

We built all of this together, and we are doing it on open source foundations with the intent to give the hard-won patterns back. If you are making these calls on your own systems, I would genuinely like to compare notes.

DEV Community: Kapil Soam