It is easy to end up in one of two absolutes. Either: traditional coding is dead and AI agents are here to dominate, or, AI is too non-deterministic and it has no place in modern software development. But there may be a third way, one that recognizes the problems, and addresses the risk in a methodical way.
Enter "Cost of Cognitive Debt", or CCD. Not understanding the inner workings of a system that a company strategy relies on has real world costs. You would, for instance, be unable to create a plan for overtaking your competitors without understanding your own technical limitations and advantages. Given that we have a system with clear boundaries and clear separation of abstraction layers, we can begin to estimate what it would mean that no one understands a specific part of the system.
The cost of recovering insight outweighs any potential gains
We can classify parts of systems that are not-humanly-understood as subject to Cognitive Debt, different from technical debt. Where technical debt describes understood technical problems within a system - Cognitive Debt describes the fact that you don't even understand if you have a problem, and consequently have no solution. It is reasonable to assume that Cognitive Debt eventually can lead to Cognitive Meltdown - a state where no human is able to build a mechanical intuition for how a system works given any reasonable amount of time, and hence would render the system useless from a business point of view. The cost of recovering insight outweighs any potential gains.
So it seems natural that the core business rules of a system needs to be well understood by not just developers, but by leadership and stakeholders as well.
What this means for agentic workflows is a conservative view of what parts of the system we can expose to Cognitive Debt. For instance, we could rule out extensive unattended agentic workflows in layers of a system that is concerned with critical business rules. This part of the system would score a 10 on a 1-10 score of Cost Of Cognitive Debt.
It also seems obvious that any infrastructure that could bring the system down for any period of time would have to bee overseen and understood by humans. These parts of the system would score a high CCD score as well.
Now, below these levels of system architecture the impact of Cognitive Debt may become a little more contained. Inside of a non-critical domain, within a contained blast radius, we could allow ourselves to accumulate some Cognitive Debt, as long a the cost thereof would be outweighed by potential gains.
This could still result in Cognitive Meltdown within this scoped context, but not all domains are equally essential for the system fulfilling its design goal, so we could make a risk assessment and accept this CCD.
Now, in finance, debt is something that you are expected to eventually pay back. If we are to introduce the idea of calculated acceptance of Cognitive Debt, it make sense to plan for how to get out of the debt. No one in their right mind would borrow more than they are theoretically able to pay back (except for some countries playing by different rules, that is, ehum).
Agentic workflows are here to stay and we need a framework for reasoning about non deterministic workflows in products that entire businesses rely heavily on. Leadership boards and stakeholders used to be oblivious to the risks and impact architectural decisions would have on their products, and relied heavily on responsible technical experts, who often saved them by understanding the business needs. Now that many technical experts have resigned to the same state of delightful ignorance, we need to address this in a structured way.
Is this a reasonable way of addressing and reasoning about the risk introduced by agentic workflows? Please leave a comment and let me know your thoughts.
Top comments (0)