DEV Community

Brian Jin
Brian Jin

Posted on

I Broke My AI Decision System 20 Ways to See Which Layer Would Catch What

#ai #agents #testing #opensource

When an AI agent makes a business decision - approve this request, charge this fee, escalate this case - and something goes wrong, the failure can live in very different places.

Sometimes the decision logic itself is wrong.

Someone encoded:

$1000 or more
Enter fullscreen mode Exit fullscreen mode

as:

more than $1000
Enter fullscreen mode Exit fullscreen mode

Or removed an exception.

Or treated missing evidence as false.

Other times, the decision was correct but the surrounding application failed.

The agent received the wrong fact.

It was told to escalate and continued anyway.

It made the right decision and invoked the wrong tool.

Those are different failure classes.

They need different owners, different tests, and different fixes.

I wanted to know whether we could actually separate them experimentally.

So I took JPS, the deterministic judgment system I am working on, integrated it with an independently developed agent regression harness called EvalForge, and broke the combined system in 20 different ways on purpose.

Then I asked a simple question:

Which layer catches what?

Why use someone else's test harness?

I could have built another JPS-specific test framework.

That would have been easier.

It also would have been weaker evidence.

EvalForge was built independently. It knew nothing about JPS.

It evaluates agent behavior from the outside by inspecting things such as:

  • which tools were called
  • which arguments were passed
  • whether a disallowed action occurred
  • whether expected actions happened
  • whether the execution stayed within the scenario constraints

That made it useful as an external observer.

The point of the experiment was not to make JPS "pass EvalForge."

It was to see whether the boundary between judgment correctness and integration correctness actually survives contact with another system.

The setup 

                 one business case
          same facts, same expected outcome
                          |
          +---------------+---------------+
          |                               |
          v                               v
 ARM A: prompt-mediated            ARM B: deterministic

 policy as plain text              structured facts
          |                               |
          v                               v
   model decides                  JPS evaluator
          |                               |
          +---------------+---------------+
                          |
                          v
                 shared execution layer
          decision -> tool call / escalate
                          |
                          v
               EvalForge scores trajectory
Enter fullscreen mode Exit fullscreen mode

Three independent detection mechanisms watched each injected failure:

[J] Judgment-layer tests
    Does the decision logic itself still behave correctly?

[F] EvalForge
    Does the surrounding agent execute the correct tools and arguments?

[G] Study gate
    Does the observed outcome still match the pinned expected result?
Enter fullscreen mode Exit fullscreen mode

The two primary paths used the same business cases and the same expected operational outcomes.

The only major difference was where the judgment came from.

The test surface

The study contained 21 business cases covering situations such as:

  • straightforward approval
  • hard-stop conditions
  • exact numeric thresholds
  • values just below thresholds
  • missing required evidence
  • unknown evidence
  • conflicting rules
  • escalation
  • routing to a specific destination
  • protected actions that must never execute under certain dispositions

Then we introduced 20 deliberate failures, one at a time.

Six were inside the judgment semantics.

Fourteen were in the surrounding integration.

Before running the scored experiment, we wrote down which detector we expected to catch each mutation.

The protocol was then frozen after three rounds of adversarial review by a different vendor's model.

Only after that did the scored run begin.

Finding 1 - the boundary was measurable

Every rule-level mutation was detected exactly where we predicted.

Examples included:

  • changing an inclusive threshold into an exclusive one
  • moving a threshold
  • removing an exception
  • collapsing unknown into false
  • changing conflict behavior

Those belong to the judgment layer.

Its own tests caught them.

The surrounding integration did not need to know why the rule was wrong.

The opposite was also true.

When the decision logic remained correct but the surrounding system misused the result, the judgment evaluator could not see the failure.

Examples included:

  • mapping the wrong fact into the evaluator
  • ignoring an escalation
  • taking a protected action after an unresolved result
  • calling the wrong tool
  • passing the wrong argument

Those failures appeared in the execution trajectory.

That is where EvalForge caught them.

Its simple trace-level scorers ended up doing most of the work:

zero_disallowed_actions
tool_called
argument_correctness
Enter fullscreen mode Exit fullscreen mode

The more semantic grounding and contradiction scorers were mostly irrelevant.

In hindsight, that makes sense.

The injected failures were often not changing what the agent said.

They were changing what it did.

So the trace was the evidence.

Across the full mutation matrix, there were zero divergences from the preregistered detection ownership.

That includes three mutations we predicted none of the three detectors would catch.

They escaped exactly as expected.

That matters.

A good experiment should expose where detection stops, not quietly redefine every escaped case as something the system was never supposed to catch.

Finding 2 - the independent adversarial cases were more useful than my own

After the protocol was frozen, an independent reviewer contributed four hidden adversarial scenarios.

I did not know their exact contents before execution.

One of them targeted a field the judgment layer could not see:

the configured escalation destination.

The judgment itself could still be correct:

escalate
Enter fullscreen mode Exit fullscreen mode

But the application could route it to:

Fee review
Enter fullscreen mode Exit fullscreen mode

instead of:

Fee review queue
Enter fullscreen mode Exit fullscreen mode

From the judgment layer's perspective, nothing was wrong.

It had correctly decided to escalate.

EvalForge's argument_correctness scorer caught the bad destination downstream.

That was exactly the kind of separation I wanted this study to expose.

More importantly, it revealed a real blind spot.

The runtime's test rows could assert that escalation should occur, but they could not assert the exact configured escalation target.

That capability has since been added and released in the runtime.

So the feedback loop became:

external adversarial case
        |
        v
blind spot exposed
        |
        v
external harness catches it
        |
        v
runtime capability improved
        |
        v
new regression coverage
        |
        v
versioned release
Enter fullscreen mode Exit fullscreen mode

That may be the most useful result of the study.

The experiment did not merely produce a score.

It changed the system.

Finding 3 - prompt-mediated judgment got 62 of 63 decision outcomes right

We also ran the comparison originally suggested by the EvalForge author.

Same business cases.

Same expected outcomes.

But instead of evaluating a structured Judgment Pack, a strong model received the policy as plain prose and made the decision itself.

The model performed very well.

It got:

62 / 63
Enter fullscreen mode Exit fullscreen mode

decision outcomes correct.

It never executed a forbidden protected action.

It correctly abstained on the blocked cases.

So this study does not support a claim that prose-based judgment is generally unreliable.

The interesting part was where it failed.

Exact conflict boundary

One case sits at an exact threshold where two rules genuinely conflict.

The deterministic evaluator is required to surface that conflict.

It is not permitted to invent a tie-break.

Across identical prompt runs, the model sometimes did something else.

In one run, it quietly chose a winner.

That produced a plausible answer.

It was also precisely the behavior the deterministic semantics prohibit.

More importantly, the behavior was inconsistent across repeated identical runs.

That is a different failure from simply "getting the answer wrong."

It means the system did not have a stable rule for what happens when the policy itself contains unresolved conflict.

Structured routing identity

A second failure appeared in a different place.

The model repeatedly altered a configured routing destination.

The structured value was:

Fee review queue
Enter fullscreen mode Exit fullscreen mode

The prose-mediated path produced plausible variations such as:

Fee review
Enter fullscreen mode Exit fullscreen mode

The semantic intention is close.

Operationally, the identifier is not the same.

That distinction matters when downstream systems expect exact configuration rather than approximate language.

So these are two separate failure modes:

1. judgment instability at a real rule boundary

2. loss of exact operational configuration
   when structured data is carried through prose
Enter fullscreen mode Exit fullscreen mode

The model's overall accuracy number hides both.

The important result was not "deterministic beats model"

That would be the wrong conclusion.

The model performed strongly.

And this is a small study.

The result I care about is that the failure classes had owners.

A deterministic judgment evaluator can tell you whether:

same facts
+
same judgment artifact
=
same disposition
Enter fullscreen mode Exit fullscreen mode

But it cannot necessarily tell you that the application fed it the wrong facts.

It cannot tell you that another layer ignored escalate.

It cannot tell you that the right decision was followed by the wrong tool call unless that behavior is brought back into its test surface.

Likewise, an execution regression harness can tell you:

the wrong tool fired
Enter fullscreen mode Exit fullscreen mode

or:

the tool received the wrong argument
Enter fullscreen mode Exit fullscreen mode

But that does not automatically tell you whether the underlying organizational rule was encoded incorrectly.

Those are different questions.

Testing the layers separately made that boundary visible.

The detection model that emerged

The study ended up looking less like one testing system and more like a chain of responsibility:

business policy
      |
      v
judgment artifact
      |
      v
[J] judgment tests
      |
      v
deterministic disposition
      |
      v
integration / agent
      |
      v
[F] trajectory tests
      |
      v
tool execution
      |
      v
[G] end-to-end expected outcome
Enter fullscreen mode Exit fullscreen mode

Each layer can be correct while another layer fails.

That is not duplication.

It is defense through separation.

The blind spots matter too

Three registered mutations were expected to escape all three detectors.

They did.

I think that is useful evidence.

There is a temptation in evaluation work to make every test failure somebody's responsibility after seeing the result.

We tried not to do that.

If neither the judgment artifact nor the execution trajectory contains enough information to observe a particular failure, then a detector cannot reconstruct that missing evidence afterward.

The correct outcome is:

We are blind here.

That can become the input to the next design change or experiment.

But it should remain a blind spot until something actually closes it.

What EvalForge taught me

The integration also surfaced several implementation observations in EvalForge itself, which I filed upstream as issues #269-#274.

But the larger takeaway was positive.

Its:

  • scenario format
  • runner
  • payload isolation
  • artifact layer
  • deterministic trace scorers

all held up under adversarial use.

The blocking zero_disallowed_actions scorer was particularly useful.

Every scenario where a protected action executed when it should not have was stopped there.

That is a very clean operational invariant.

What changed in JPS

The study also produced a concrete runtime improvement.

One hidden adversarial case showed that JPS could assert:

escalation must happen
Enter fullscreen mode Exit fullscreen mode

but could not test:

escalation must go to this exact configured destination
Enter fullscreen mode Exit fullscreen mode

That gap is now addressed in:

judgment-pack-runtime v0.17.0
Enter fullscreen mode Exit fullscreen mode

This is the feedback loop I want around judgment artifacts:

author
  |
  v
test
  |
  v
evaluate
  |
  v
observe failures
  |
  v
improve semantics / tooling
  |
  v
review
  |
  v
version
Enter fullscreen mode Exit fullscreen mode

The artifact should evolve because evidence exposed a gap, not because someone casually changed a prompt.

Honest limits

This study is deliberately narrow.

It involved:

  • 20 planted mutations
  • 21 business cases
  • one external regression harness
  • one model in the prompt-mediated arm
  • one JPS decision family
  • deterministic scoring wherever possible

It does not prove that JPS improves model accuracy generally.

In fact, an earlier experiment on a harder third-party benchmark produced a less favorable efficacy result, and that result still stands.

Study 013 answers a different question.

It asks whether failures inside the decision semantics can be distinguished from failures in the surrounding execution path.

Within this study, they could.

It also does not prove that the mutation set is complete.

There will be failures outside all three detectors.

The study already contains examples.

And the prompt-mediated comparison is too small to support broad claims about model reliability.

The interesting value is where the failures occurred, not the headline percentage.

Receipts

Everything needed to inspect or reproduce the study is public.

Study 013

https://github.com/Judgment-Pack/judgment-pack-evaluator-experiments/tree/main/studies/013-agent-eval-forge-integration

The repository contains:

  • preregistration
  • adversarial review
  • mutation definitions
  • hidden-case handling
  • raw run artifacts
  • deterministic scoring
  • detection matrix
  • analysis

Agent Eval Forge

https://github.com/deghosal-2026/agent-eval-forge

The harness was pinned and used as an independently developed external observer.

Runtime change

The blind spot identified by the hidden escalation-target case was addressed in:

judgment-pack-runtime v0.17.0
Enter fullscreen mode Exit fullscreen mode

The takeaway

I started with a fairly simple claim:

A deterministic judgment layer should make business decision logic testable independently of the agent executing it.

The experiment made that claim more precise.

The useful separation is:

Judgment layer:
Was the decision implied by these facts and rules?

Integration layer:
Did the surrounding system respect that decision?

Execution layer:
Did the correct action actually happen?
Enter fullscreen mode Exit fullscreen mode

A correct answer at one layer does not guarantee correctness at the next.

That is why the most interesting result was not:

JPS caught X and EvalForge caught Y.

It was:

We could identify which layer owned each registered failure - and we could also identify failures owned by neither.

That gives us somewhere concrete to improve.

In your agent stack, who owns the failure when the decision was right but the action was wrong - and would anything in your current tests tell you which one happened?

Top comments (0)