A few months ago, I helped ship an agent that matched supplier invoices against purchase orders and approved the clean ones automatically. It was honestly one of the more satisfying things I’ve built. The accounts team went from drowning in a queue to reviewing only the odd ones out, and for a while everyone seemed very happy.
Then someone from finance pinged me with a screenshot of an approved invoice and a single question: “Why did it let this one through?”
I opened our logs, fully expecting an easy answer. What I found was a timestamp, an invoice ID, and the word "APPROVED." That was it.
We logged the outcome, not the decision
Looking back, the mistake is almost embarrassing. We had treated the agent like any other service, so we logged what it did. But finance doesn’t care much about what the system did. Instead, they care about why it did it, which data it looked at, and whether it was even allowed to make that call in the first place.
We couldn’t answer any of that; we couldn’t tell which version of the matching rules ran, what the model saw, or whether a person had been involved at any stage. Technically, nothing had failed but practically, though, we’d built something that made financial decisions and then forgot its own reasoning the moment it finished.
Why is this harder in finance than anywhere else?
In most products, a wrong output is a bug you fix and move on from. In finance, a wrong output becomes a posted entry, and posted entries are records people may need to explain long after the code that created them has been rewritten multiple times. Auditors don’t accept that “the model thought it looked fine.” They want to trace an action back to a rule, the data behind it, and someone accountable for it.
That’s really the core tension in autonomous finance. The automation itself is the easy part to demo, but the moment a system can act on the books without a person, the engineering problem quietly shifts from “can it do the work” to “can we prove what it did and why.”
What we changed
The first thing we did was give every decision its own record. Now, whenever the agent approves or rejects something, it writes down the inputs it used, the rule version, the model version, its confidence, and a plain-language reason. Storage is cheap, and that one table answered more finance questions in the following month than our dashboards had in a year.
Next, we split the agent from the action. The agent no longer posts anything itself. Instead, it proposes, and a separate, boring, well-tested service checks that proposal against hard limits before anything touches the ledger. If the amount crosses a threshold, or the supplier is new, or the match isn’t exact, the proposal goes to a human. The model can be as clever as it likes, but it can’t talk its way past a rule that simply isn’t negotiable.
We also narrowed what the agent could reach. Early on, it had broad read access because that made development easier. Eventually, we locked it down to a short list of approved queries and actions, which meant that when someone asked what it could possibly have done, we would have had a definite answer.
Finally, we started versioning the matching rules and prompts like real code, with reviews and change history. A tweak to a tolerance setting is a financial policy change, after all, even if it only looks like a one-line diff.
It’s not about slowing things down
I was worried all of this would kill the speed that made the project worth doing in the first place. But it didn’t. Most invoices still flow straight through, and the only difference is that each one now leaves a trail behind it. The ones that do stop are usually the ones that should have stopped anyway.
The bigger change was in how I think about these systems. When software only records decisions, logging is a debugging aid. But when software makes the decisions itself, the log is the product’s memory, and if it can’t explain itself months later, it probably shouldn’t have been trusted with the decision in the first place.
So build the agent, by all means. Just make sure it can answer the question your finance team is eventually going to ask.
Top comments (0)