Should Probabilistic AI Judges Be Treated as Sensors Rather Than Verifiers?

藍凰 — Fri, 02 Oct 2026 10:57:09 +0000

I’ve been thinking about a class of AI systems that do not primarily generate text.

Instead, they take some state or evidence and return bounded judgments such as:

is_risky: 0.91

category:
  safe: 0.03
  suspicious: 0.79
  unknown: 0.18

review_priority: 2.6 / 3

Recent systems such as Jev make this pattern particularly visible: instead of asking a language model to generate an explanation or JSON object and then parsing it back into a decision, the model directly produces typed probabilistic judgments.

That seems extremely useful for agent runtimes.

But it raises a question I’m not sure how these systems should be placed inside the control plane.

A tempting architecture

Suppose an agent wants to use a tool.

A probabilistic model inspects the request and returns:

dangerous_operation = 0.94

The obvious implementation is:

if dangerous_operation > 0.8:
    reject()

This is simple, fast, and probably useful in many applications.

But I’m increasingly uncomfortable with calling the probabilistic model a verifier.

The model did not prove that the operation was unsafe.

It produced an observation about the operation.

That seems closer to a sensor.

Sensor versus verifier

The architecture I am experimenting with separates four roles:

Artifact / Runtime State
          ↓
   Finding Producer
          ↓
 Finding + confidence
          ↓
      Policy
          ↓
      Decision
          ↓
      Verifier
          ↓
      Enforcer

A probabilistic AI model would live at the first stage:

AI judge
   ↓
FindingProducer

For example:

Finding:
  type: SEMANTIC_RISK
  probability: 0.94
  model_version: X
  evidence_ref: Y

A separate policy layer could then decide what that finding means:

if SEMANTIC_RISK >= 0.90:
    require_human_review

The verifier would check whether the policy was applied consistently, whether the correct policy version was used, and whether required preconditions were satisfied.

The enforcer would finally allow, block, quarantine, or escalate the action.

So:

probabilistic judgment
        ≠
policy decision
        ≠
verification
        ≠
enforcement

Why bother separating them?

Imagine the model changes.

Version A says:

risk = 0.92

Version B evaluates the exact same evidence and says:

risk = 0.41

Was the original decision wrong?

Maybe.

But another interpretation is that the historical decision was valid under:

model A
+
policy version 3
+
evidence snapshot E

while a new evaluation would produce a different result under a new model.

That suggests model outputs might need provenance similar to other evidence:

model identity
model version
question / rubric version
input evidence version
timestamp / epoch
confidence

Otherwise, replaying or auditing an agent decision later becomes ambiguous.

Another problem: calibrated does not mean correct

Even if a probabilistic model is well calibrated, a result such as:

0.93

does not mean:

this statement is objectively 93% true

It is still a model judgment.

And even a perfectly type-safe system can confidently choose the wrong member of the allowed output space.

So I currently prefer this interpretation:

A probabilistic AI judge is a semantic observation mechanism, not an authority source.

Its result can influence a decision, but it should not silently become the decision itself.

But perhaps this separation is too strict

There are obvious counterexamples.

Spam filters routinely make automated decisions.

Fraud systems block transactions.

Content moderation models directly suppress material.

Anomaly detectors can trigger circuit breakers.

At some point, a probabilistic judgment clearly does become operational authority.

Maybe the right distinction is not:

AI model may never decide

but instead:

model produces judgment

policy explicitly grants that judgment
a bounded amount of authority

For example:

confidence >= 0.99
AND
effect is reversible
AND
blast radius is low
→ automatic action allowed

otherwise
→ review / escalation

In that architecture, the model still does not define its own authority.

The runtime does.

The question

So I’m curious how others model this boundary.

Should probabilistic AI judges be treated primarily as sensors / finding producers rather than verifiers?

More specifically:

Should model output ever directly authorize an irreversible action?
Should model version and rubric version be part of the evidence provenance?
If a newer model disagrees with an older model, should historical decisions be re-evaluated?
Where should confidence thresholds live: inside the model interface, inside policy, or inside the application?
Is there established terminology for separating probabilistic semantic judgment from deterministic policy enforcement?

I’m especially interested in related patterns from:

runtime assurance
policy engines
autonomous systems
fraud / risk systems
capability security
agent runtimes
formal methods

The architecture that led me to this question is an experimental agent-runtime project called HANDS, but I’m deliberately trying to phrase the problem independently of that implementation.

I would be particularly interested in counterexamples where treating the model as “just a sensor” is the wrong abstraction.

Disclosure: I used ChatGPT to help structure and edit this post. The underlying architecture question and design are from my own ongoing project, and I reviewed the content before publishing.

Can a Self-Improving Agent Runtime Safely Modify Its Own Verifier?

藍凰 — Thu, 01 Oct 2026 14:00:50 +0000

I’m experimenting with an agent-runtime architecture where execution results can become evidence for changing the runtime itself.

The simplified loop looks like this:

execution
    ↓
evidence
    ↓
modification candidate
    ↓
verification
    ↓
activation
    ↓
new runtime state

This creates a problem I’m trying to reason about.

Suppose runtime epoch e uses verifier V_e.

An agent observes its execution history and proposes a modification C. That modification includes an update to the verifier itself, producing V_(e+1).

My current rule is:

A modification must not change the verifier, authorization policy, or evidence-acceptance rules used to authorize that same modification.

So:

Accept(C) = V_e(C, Evidence_e)

may authorize:

V_e → V_(e+1)

but this should not be allowed:

Accept(C) = V_(e+1)(C, Evidence_e)

because the candidate would effectively participate in defining the rules by which it is accepted.

I’ve been calling this a no same-epoch self-authorization rule.

But I’m not convinced that this is sufficient.

For example:

What if the candidate leaves the verifier unchanged but changes the evidence-selection mechanism?
Is validation of V_(e+1) by V_e enough, or does that merely create a chain of inherited trust?
If the verifier itself contains a bug, what mechanism should be allowed to replace it?
Does safe self-modification ultimately require a small non-self-modifying root of trust?
Or can verifier evolution itself be safely modeled as a layered adaptation process?

I’m especially interested in existing work or implementation experience from:

self-adaptive systems
runtime assurance
capability security
formal methods
reflective systems
agent runtimes

I’m not claiming this is a new problem. Quite the opposite: I suspect there are already better terms and established models for parts of it.

If you know of a relevant concept, paper, system, or counterexample to the rule above, I’d like to know what I should be comparing this design against.

Disclosure: I used ChatGPT to help structure and edit this post. The underlying architecture question and design are from my own ongoing project, and I reviewed the content before publishing.

DEV Community: 藍凰

Should Probabilistic AI Judges Be Treated as Sensors Rather Than Verifiers?

A tempting architecture

Sensor versus verifier

Why bother separating them?

Another problem: calibrated does not mean correct

But perhaps this separation is too strict

The question

Can a Self-Improving Agent Runtime Safely Modify Its Own Verifier?