Brian Jin

Posted on Aug 5

What I Learned Trying to Make AI-Agent Decisions Testable

#opensource #ai #showdev

What I Learned Trying to Make AI-Agent Decisions Testable

AI agents can retrieve the right policy, call the right tools, and still reach the wrong conclusion.

That gap became increasingly hard for me to ignore.

Most agent systems are getting better at access and execution. They can search documents, query databases, call APIs, use MCP servers, and complete multi-step workflows.

But many important business decisions depend on something different:

Which evidence is actually required?
Which rule applies in this situation?
Which exception changes the outcome?
What should happen when evidence is missing or conflicting?
When must the agent stop and escalate?

Those questions are often buried inside prompts, application code, policy documents, or the judgment of a few experienced people.

That makes them difficult to test, review, reuse, and improve.

The Problem Is Not Always Missing Knowledge

Imagine an agent reviewing a new vendor.

It retrieves:

the vendor-onboarding policy
the sanctions-screening result
the tax form
the proposed annual spend
the business justification

The agent has access to all the relevant information.

But it still has to determine:

whether the request is in scope
whether all mandatory evidence is present
whether a sanctions match creates a hard stop
whether high annual spend requires committee review
whether incomplete evidence means rejection or escalation
whether it has enough information to decide at all

Retrieval helps the agent find the policy.

It does not guarantee that the agent will apply the policy correctly.

A model may produce a plausible explanation while overlooking an exception, treating missing evidence as negative evidence, or claiming that approval occurred when no authorized person was consulted.

The answer can sound reasonable and still be indefensible.

Why Coding Agents Often Feel More Reliable

Coding agents currently provide one of the clearest examples of useful multi-step agent behavior.

I do not think that is only because models are unusually good at code.

Coding agents work inside an environment that already knows how to challenge their output:

compilers
unit tests
type systems
linters
runtime feedback
version control
code review
CI/CD

The model proposes work.

The surrounding environment evaluates it.

Most business agents do not have an equivalent judgment harness.

They may have access to tools and knowledge, but the conditions behind a defensible decision remain implicit.

What I Tried First

The obvious place to start was prompts.

A prompt can say:

always check sanctions
require a tax form
escalate high-value vendors
never approve when evidence is incomplete

This can work for a narrow workflow.

But as the decision grows, the prompt becomes difficult to review and test. Rules, exceptions, evidence requirements, explanations, and workflow instructions become mixed together.

A model can also interpret the same prose differently across runs or implementations.

The next option is application code.

Hard-coded logic is more deterministic, but it creates another problem. The judgment becomes coupled to one application, one programming language, and one deployment.

A domain expert cannot easily review it. Another agent cannot reuse it without reproducing the logic. Changes become software releases rather than reviewed updates to an organizational decision contract.

Skills and tool definitions help package behavior.

MCP helps standardize access to context and capabilities.

Policy engines can evaluate explicit rules.

All of these are useful. None of them, by themselves, fully represent the evidence, applicability, exceptions, uncertainty, and escalation conditions behind a business decision.

That led me to experiment with separating the judgment from both the model and the workflow.

The Idea: A Testable Judgment Contract

I started working on what is now the Judgment Pack Specification.

A Judgment Pack is a declarative representation of the conditions behind a decision.

At a high level, a pack can describe:

what question is being decided
when the pack applies
which evidence is required
which facts are relevant
which rules and exceptions affect the outcome
which conditions make the decision unresolved
when escalation is required
which outputs are valid
how expected behavior can be tested

A simplified vendor-onboarding pack might express:

Decision:
May this vendor be onboarded?

Required evidence:
- sanctions-screening result
- tax-form status

Exceptions:
- sanctions match → hard stop
- annual spend above threshold → committee review

Unresolved conditions:
- required evidence missing
- screening could not be completed
- approval authority is unknown

The important part is not the file format.

The important part is that the decision conditions become a reviewable and testable artifact rather than hidden instructions inside a prompt.

What Testability Changes

Once the judgment is explicit, we can write scenarios such as:

Scenario 1:
Sanctions clear
Tax form received
Spend below threshold
Expected result: eligible for approval

Scenario 2:
Sanctions match
Tax form received
Expected result: hard stop

Scenario 3:
Sanctions screening missing
Tax form received
Expected result: unresolved

Scenario 4:
Sanctions clear
Tax form received
Spend above threshold
Expected result: committee review

Now a change to the pack can be evaluated against known expectations.

If someone modifies the annual-spend exception and an existing scenario starts producing direct approval instead of committee review, the test should fail.

That is closer to what coding agents already have: an environment that can challenge the result.

Correctly Refusing to Decide Is Part of Correctness

One lesson has become especially important.

A business agent should not always produce an answer.

Sometimes the correct outcome is:

evidence required
unresolved
not applicable
escalate
human approval required

Many current evaluation approaches reward producing the expected answer. But in enterprise decisions, a system can also fail by deciding when it should have stopped.

That means abstention and escalation cannot be treated as fallback error states.

They are first-class outcomes.

A useful judgment system must be able to distinguish:

“The answer is no”

from:

“The system does not yet have authority or evidence to answer.”

Those are operationally very different.

What Judgment Pack Does Not Solve

I do not see this as a replacement for agent frameworks, MCP, policy engines, or knowledge graphs.

They solve different parts of the system.

MCP can connect an agent to a sanctions service.

A gateway can control whether the agent is allowed to call it.

A policy engine can evaluate a deterministic rule.

A knowledge graph can represent relationships and organizational context.

A Judgment Pack is intended to describe how those inputs contribute to a defensible decision.

It also does not prove that source data is true.

If a system receives a false sanctions result, a correct evaluator can still produce the wrong real-world outcome. Provenance, source trust, authorization, and evidence acquisition remain separate concerns.

The current project is an attempt to make those boundaries explicit rather than claiming one specification solves all of agent governance.

Questions That Are Still Open

Opening the project as open source is important because several of the hardest questions should not be answered by one person.

Some of the questions I am currently exploring are:

How should an agent select the correct pack?
How should multiple packs compose?
Where should Judgment Pack end and policy engines begin?
How should conflicting evidence be represented?
Which evaluator semantics must be portable across runtimes?
How should evidence freshness be expressed?
How should organizations version and approve judgment changes?
How should authorization be separated from decision evaluation?
What makes an explanation defensible rather than merely plausible?

These are specification questions, runtime questions, and organizational questions at the same time.

Why I Opened It

I spent many years building enterprise data platforms, operational systems, and AI infrastructure.

Across those environments, organizations often had plenty of data and documentation. What they struggled to preserve was the judgment behind important decisions:

which evidence mattered
why an exception was allowed
when a rule stopped applying
who had authority
what uncertainty required escalation

Frontier models can reason over more context and use more tools.

But every organization still has its own conditions for what counts as an acceptable decision.

That judgment should not remain trapped inside prompts, undocumented code, or individual experience.

It should be possible to review, test, version, and improve it.

The Project Is Now Open Source

The Judgment Pack Specification and runtime are now open source.

I am looking for contributors in several areas:

Specification review

Challenge how the project represents evidence, exceptions, uncertainty, escalation, and outcomes.
Real-world decision packs

Contribute a difficult decision from procurement, data governance, security, support, finance, operations, or another domain.
Adversarial scenarios

Add cases where a reasonable-looking agent should refuse, escalate, or produce a different outcome.
Runtime and tooling

Improve the Go runtime, evaluator, CLI, documentation, and agent integrations.

The most useful first contribution may not be code.

A real decision that exposes where the current model breaks can be more valuable than another feature.

Website: judgmentpack.org
Specification: Judgment-Pack/judgment-pack-spec
Good first issues: Browse beginner-friendly issues

I would especially value critical feedback from people working on agent evaluation, MCP, policy engines, rules-as-code, human approval, and enterprise AI systems.

The project will improve faster through strong disagreement and real use cases than through agreement alone.

Disclosure: AI tools assisted with editing and structure. The project design, technical content, examples, and final review are my own.