What I Learned Trying to Make AI-Agent Decisions Testable

AI agents can retrieve the right policy, call the right tools, and still reach the wrong conclusion.

That gap became increasingly hard for me to ignore.

Most agent systems are getting better at access and execution. They can search documents, query databases, call APIs, use MCP servers, and complete multi-step workflows.

But many important business decisions depend on something different:

• Which evidence is actually required?
• Which rule applies in this situation?
• Which exception changes the outcome?
• What should happen when evidence is missing or conflicting?
• When must the agent stop and escalate?

Those questions are often buried inside prompts, application code, policy documents, or the judgment of a few experienced people.

That makes them difficult to test, review, reuse, and improve.

The Problem Is Not Always Missing Knowledge

Imagine an agent reviewing a new vendor.

It retrieves:

• the vendor-onboarding policy
• the sanctions-screening result
• the tax form
• the proposed annual spend
• the business justification

The agent has access to all the relevant information.

But it still has to determine:

• whether the request is in scope
• whether all mandatory evidence is present
• whether a sanctions match creates a hard stop
• whether high annual spend requires committee review
• whether incomplete evidence means rejection or escalation
• whether it has enough information to decide at all

Retrieval helps the agent find the policy.

It does not guarantee that the agent will apply the policy correctly.

A model may produce a plausible explanation while overlooking an exception, treating missing evidence as negative evidence, or claiming that approval occurred when no authorized person was consulted.

The answer can sound reasonable and still be indefensible.

Why Coding Agents Often Feel More Reliable

Coding agents currently provide one of the clearest examples of useful multi-step agent behavior.

I do not think that is only because models are unusually good at code.

Coding agents work inside an environment that already knows how to challenge their output:

• compilers
• unit tests
• type systems
• linters
• runtime feedback
• version control
• code review
• CI/CD

The model proposes work.

The surrounding environment evaluates it.

Most business agents do not have an equivalent judgment harness.

They may have access to tools and knowledge, but the conditions behind a defensible decision remain implicit.

What I Tried First

The obvious place to start was prompts.

A prompt can say:

• always check sanctions
• require a tax form
• escalate high-value vendors
• never approve when evidence is incomplete

This can work for a narrow workflow.

But as the decision grows, the prompt becomes difficult to review and test. Rules, exceptions, evidence requirements, explanations, and workflow instructions become mixed together.

A model can also interpret the same prose differently across runs or implementations.

The next option is application code.

Hard-coded logic is more deterministic, but it creates another problem. The judgment becomes coupled to one application, one programming language, and one deployment.

A domain expert cannot easily review it. Another agent cannot reuse it without reproducing the logic. Changes become software releases rather than reviewed updates to an organizational decision contract.

Skills and tool definitions help package behavior.

MCP helps standardize access to context and capabilities.

Policy engines can evaluate explicit rules.

All of these are useful. None of them, by themselves, fully represent the evidence, applicability, exceptions, uncertainty, and escalation conditions behind a business decision.

That led me to experiment with separating the judgment from both the model and the workflow.

The Idea: A Testable Judgment Contract

I started working on what is now the Judgment Pack Specification.

A Judgment Pack is a declarative representation of the conditions behind a decision.

At a high level, a pack can describe:

• what question is being decided
• when the pack applies
• which evidence is required
• which facts are relevant
• which rules and exceptions affect the outcome
• which conditions make the decision unresolved
• when escalation is required
• which outputs are valid
• how expected behavior can be tested

A simplified vendor-onboarding pack might express:

Decision:
May this vendor be onboarded?

Required evidence:
- sanctions-screening result
- tax-form status

Exceptions:
- sanctions match → hard stop
- annual spend above threshold → committee review

Unresolved conditions:
- required evidence missing
- screening could not be completed
- approval authority is unknown

The important part is not the file format.

The important part is that the decision conditions become a reviewable and testable artifact rather than hidden instructions inside a prompt.

What Testability Changes

Once the judgment is explicit, we can write scenarios such as:

Scenario 1:
Sanctions clear
Tax form received
Spend below threshold
Expected result: eligible for approval

Scenario 2:
Sanctions match
Tax form received
Expected result: hard stop

Scenario 3:
Sanctions screening missing
Tax form received
Expected result: unresolved

Scenario 4:
Sanctions clear
Tax form received
Spend above threshold
Expected result: committee review

Now a change to the pack can be evaluated against known expectations.

If someone modifies the annual-spend exception and an existing scenario starts producing direct approval instead of committee review, the test should fail.

That is closer to what coding agents already have: an environment that can challenge the result.

Correctly Refusing to Decide Is Part of Correctness

One lesson has become especially important.

A business agent should not always produce an answer.

Sometimes the correct outcome is:

• evidence required
• unresolved
• not applicable
• escalate
• human approval required

Many current evaluation approaches reward producing the expected answer. But in enterprise decisions, a system can also fail by deciding when it should have stopped.

That means abstention and escalation cannot be treated as fallback error states.

They are first-class outcomes.

A useful judgment system must be able to distinguish:

“The answer is no”

from:

“The system does not yet have authority or evidence to answer.”

Those are operationally very different.

What Judgment Pack Does Not Solve

I do not see this as a replacement for agent frameworks, MCP, policy engines, or knowledge graphs.

They solve different parts of the system.

MCP can connect an agent to a sanctions service.

A gateway can control whether the agent is allowed to call it.

A policy engine can evaluate a deterministic rule.

A knowledge graph can represent relationships and organizational context.

A Judgment Pack is intended to describe how those inputs contribute to a defensible decision.

It also does not prove that source data is true.

If a system receives a false sanctions result, a correct evaluator can still produce the wrong real-world outcome. Provenance, source trust, authorization, and evidence acquisition remain separate concerns.

The current project is an attempt to make those boundaries explicit rather than claiming one specification solves all of agent governance.

Questions That Are Still Open

Opening the project as open source is important because several of the hardest questions should not be answered by one person.

Some of the questions I am currently exploring are:

• How should an agent select the correct pack?
• How should multiple packs compose?
• Where should Judgment Pack end and policy engines begin?
• How should conflicting evidence be represented?
• Which evaluator semantics must be portable across runtimes?
• How should evidence freshness be expressed?
• How should organizations version and approve judgment changes?
• How should authorization be separated from decision evaluation?
• What makes an explanation defensible rather than merely plausible?

These are specification questions, runtime questions, and organizational questions at the same time.

Why I Opened It

I spent many years building enterprise data platforms, operational systems, and AI infrastructure.

Across those environments, organizations often had plenty of data and documentation. What they struggled to preserve was the judgment behind important decisions:

• which evidence mattered
• why an exception was allowed
• when a rule stopped applying
• who had authority
• what uncertainty required escalation

Frontier models can reason over more context and use more tools.

But every organization still has its own conditions for what counts as an acceptable decision.

That judgment should not remain trapped inside prompts, undocumented code, or individual experience.

It should be possible to review, test, version, and improve it.

The Project Is Now Open Source

The Judgment Pack Specification and runtime are now open source.

I am looking for contributors in several areas:

1. Specification review
   
   Challenge how the project represents evidence, exceptions, uncertainty, escalation, and outcomes.

2. Real-world decision packs
   
   Contribute a difficult decision from procurement, data governance, security, support, finance, operations, or another domain.

3. Adversarial scenarios
   
   Add cases where a reasonable-looking agent should refuse, escalate, or produce a different outcome.

4. Runtime and tooling
   
   Improve the Go runtime, evaluator, CLI, documentation, and agent integrations.

The most useful first contribution may not be code.

A real decision that exposes where the current model breaks can be more valuable than another feature.

• Website: judgmentpack.org
• Specification: Judgment-Pack/judgment-pack-spec
• Good first issues: Browse beginner-friendly issues

I would especially value critical feedback from people working on agent evaluation, MCP, policy engines, rules-as-code, human approval, and enterprise AI systems.

The project will improve faster through strong disagreement and real use cases than through agreement alone.

---

Disclosure: AI tools assisted with editing and structure. The project design, technical content, examples, and final review are my own.

Comments

[Linhao Liu]

Like your article.

Taking your third question I have a suggestion - in my practice, when the retrieved evidence exists but doesn't support a conclusion, it returns a suggestion with an insufficient_evidence flag set, so agent can avoid false positive.

Besides, I still have two questions:

1. Is there a plan to measure whether pack-mediated judgment produces better decisions than the prompt-only baseline it replaces? I believe it's not mentioned in article and Github README.
2. Are we assuming determinisitic evidence to prove the scenario? E.g, in scenario 2 - "Sanctions match", how the agent decides sanctions match or not?

[Brian Jin]

Thanks, Linhao. The insufficient_evidence flag is a useful pattern and closely matches a distinction JPS is trying to make explicit: “the conclusion is negative” and “the available evidence does not support a conclusion” should produce different operational results.

JPS does not currently represent this as one Boolean flag. It uses a tri-state evidence input - present, absent, or unknown - and returns an unresolved disposition:

• Required evidence known to be absent - missing-required-evidence
• Evidence availability cannot be determined - unknown

This behavior is defined in JPS Core §7.5 and §8. The original design and rationale are recorded in RFC 0006, which also includes a concrete absent-evidence example.

So your insufficient_evidence flag maps closely to:

kind: unresolved

reasons: [missing-required-evidence]

You are also right to ask about a prompt-only baseline. I should clarify that the study is already designed, but it has not yet been fully run.

Study 001 is a preregistered, three-arm experiment using RuleArena:

• Arm A: the Collective Bargaining Agreement text verbatim in the prompt
• Arm A′: the pack’s semantic content rendered as prose in the prompt
• Arm B: the Judgment Pack evaluated through the experimental evaluator

Arm A′ is the most important control. It separates the benefit of the pack machinery from the benefit of a human first spending time disambiguating and structuring the policy. If B beats A but not A′, the finding would be that careful policy analysis helped, not that the Judgment Pack itself helped.

The planned comparison uses identical facts across all arms, two model families, five trials per instance, and judge-free scoring against RuleArena’s gold decisions and rule citations. It measures repeated-run reliability, accuracy, correct escalation, false escalation, citation quality, parse failures, and evaluator refusals.

The pipeline already runs end to end, and Arm B has been executed across the corpus. However, the prompt arms have not been run at scale against real models. Arm B currently shows 57.9% accuracy across answerable instances and 80.1% accuracy on the subset where it produced a decision, but those numbers are not evidence that JPS outperforms a prompt because Arms A and A′ are still missing.

So the honest answer today is that the comparison is preregistered and the machinery exists, but there is no efficacy result yet. Your question reinforces that completing those two prompt arms should be a priority.

Your second question is also a key boundary. JPS should not independently determine whether a sanctions match is true. An evidence-acquisition layer or trusted tool should provide a normalized fact such as clear, match, or unknown, ideally with provenance. The pack then defines what that fact means for the decision.

For the same normalized facts and the same pack version, conforming JPS evaluators should produce the same disposition rather than leaving the conclusion to a model’s interpretation on each run.

JPS also creates a feedback loop around the judgment itself. Evaluation results, adversarial cases, and real-world outcomes can reveal where a pack is incomplete or incorrect. Authorized authors can then revise the pack, run its test scenarios, approve the change, and publish a new version with a reviewable history of how the organization’s judgment evolved.

I would be very interested in your experience designing insufficient_evidence. Would you be open to contributing a few adversarial scenarios or helping shape the prompt-only versus pack-mediated benchmark?

[Linhao Liu]

Thanks for the detailed answer.

On the experiment. Before the prompt arms run, my prediction: B ties A′ on accuracy and beats it clearly on consistency. If that's the shape of the result, the conclusion isn't "packs are more accurate" — it's "packs make a given level of accuracy reproducible and reviewable," which I think is a stronger position anyway.

For adversarial scenario,
1). Reviewed description and evaluated condition disagree:
{ description: "5000 or more spend requires review",
"when": {
"op": "fact",
"path": "/expense/amount",
"operator": "greater-than",
"value": "5000"
}
}
I believe this can is a common issue in rules - reviewer believes the description is correct but the actual rule mismatches.

2). Like I said, required evidence is present but doesn't actually bear on the question. E.g, A pack requires a signed legal agreement before approving. The document exists (it's not exactly missing-required-evidence), so evidence availability is present — but it's a partial copy, and the section stating the decision is missing. The evaluator proceeds and returns a disposition it shouldn't have reached.

Anyway I expect to see your experiment outcome 😀

[Brian Jin]

Thanks, Linhao - these are exactly the kind of adversarial cases I was hoping to get.

I also agree with your prediction about the experiment. If B ties A′ on accuracy but materially improves consistency, reproducibility, and reviewability, that would be a meaningful result. The value proposition does not need to be "JPS makes the model smarter." It may be that JPS makes organizational judgment deterministic enough to test, inspect, and govern.

Your first scenario exposes an important authoring problem:

• description says 5000 or more
• executable condition says greater-than 5000

Both are individually valid, but they disagree at exactly 5000. A reviewer may approve the pack based on the description while the evaluator executes something different. This should become an adversarial/conformance case, and possibly an authoring validation concern.

The second case is even more interesting. JPS currently distinguishes evidence availability - present, absent, unknown - but "present" does not necessarily mean "sufficient to support the fact being asserted."

A partial agreement can exist while the relevant clause is missing. Treating that as sufficient evidence would allow the evaluator to reach a disposition it should not reach.

That suggests a distinction between:

evidence exists

and

evidence sufficiently supports this fact/claim

This may need to live partly at the evidence-acquisition/grounding boundary rather than inside the deterministic evaluator itself, but the pack should be able to require it rather than silently treating presence as sufficiency.

I’m going to capture both as adversarial cases. They are good examples of where real-world use can expose gaps that a clean benchmark may not.

[Brian Jin]

Thanks for reading. I am especially interested in criticism of three areas:

• Where should Judgment Pack end and policy engines begin?
• How should an agent select the correct pack?
• Is unresolved or escalation expressive enough as a first-class outcome?

Real decision examples and adversarial cases are equally valuable, I am not only looking for code contributions.