Your AI judge says 98% confident. Does it mean it?

Kunko AI Labs — Sat, 19 Sep 2026 14:38:38 +0000

Jev is getting wired into agents everywhere as the cheap typed-judgment layer (Choice/Score/Boolean with probabilities). Every leaderboard ranks judges by accuracy. But accuracy isn't what your agent acts on — the probability is. Nobody was checking whether those probabilities are honest. So I built the harness to do it.

Judge Audit (https://github.com/kunko-ai-labs/judge-audit): an open harness that audits whether an AI judge's probabilities mean what they say — ECE, reliability diagrams, accuracy-coverage curves, cost/latency, drift. Judge-agnostic by design (jev / llm / simulated adapters); Jev is just the first target because it's the one people are actually deploying.

Three audits, every number recomputable from raw per-row checkpoints in docs/runs/:

Clean emails (200): 200/200, ECE 0.0036, min confidence 0.89. Suspiciously perfect — synthetic and easy, published as a calibration baseline, not a production claim. So I built a malicious set.
Adversarial emails (200): prompt injection, cyrillic homoglyphs, zero-width chars, fullwidth, synthetic PII, social engineering. 95.5% overall, ECE 0.0389 (10x worse than clean). Injection fooled it 7/40 — but mean confidence collapsed 0.996 → 0.71 under attack. In this dataset, failures often coincided with lower confidence: a usable abstention/escalation signal, not a guarantee. Homoglyphs/zero-width/fullwidth/PII: 100% on the tested cases. Social engineering: 0% attacker success.
Task router (120): easy→cheap model, hard→frontier. It said route_easy 120/120 times. Dijkstra, N-queens, knapsack → cheap model at 0.98–1.0 confidence. Hard accuracy 0%, ECE 0.93 on the hard slice. Confidently, systematically wrong — the exact failure mode no accuracy leaderboard would catch.

Caveats, because I know this crowd: the router ground truth is by construction (task difficulty as designed), not empirically validated against cheap vs frontier models — that's the next experiment. The 0% attack success on cost-inflation is an artefact of the never-route-strong bias, not robustness. All flagged in the reports.

v0.3.0 ships an MCP server (pip install 'kunko-judge-audit[mcp]', tools: run_audit, check_drift, list_judges). Next: Jev vs OpenJev vs Claude vs a local open model on the same datasets, side by side — tell me what you'd add to the matchup.

Apache-2.0, Sigstore-attested artifacts. Brutal feedback welcome, especially on methodology.

24-second video built from the real router numbers: https://github.com/kunko-ai-labs/judge-audit/releases/download/v0.3.0/brag.mp4

Put a permission gate on your AI agent in 5 minutes (MCP + Claude Code + CI)

Kunko AI Labs — Fri, 18 Sep 2026 11:22:40 +0000

Your agent's permissions drift silently. A PR adds an MCP server with write scope, or allow: Bash(*) lands in .claude/settings.json while the runbook says "a human approves". Tests don't catch it. Linters don't catch it. Here's a 5-minute gate that does.

Step 1: declare the promise

Create agent-assurance.yaml in your repo — what the agent may do, in plain config:

# agent-assurance.yaml (simplified)
capabilities: [read]        # no write, no execute, no external send
data: [internal]            # no customer data, no credentials
autonomy: L2                # a human approves actions

Step 2: scan locally

pipx install agent-assurance
agent-assurance scan .

You get the observed blast radius plus a "declared vs observed" verdict. Now break it on purpose — add an MCP server with write scope — and watch it point at the file and line.

Step 3: gate the PR

# .github/workflows/agent-assurance.yml
permissions: { contents: read, pull-requests: write }
steps:
  - uses: actions/checkout@v4
  - uses: kunko-ai-labs/agent-assurance@v0.5
    with:
      mode: diff
      manifest: agent-assurance.yaml

Every PR now gets one comment, updated on every push, plus a red check when the promise breaks. On push/release, switch to mode: scan for SARIF findings in the Security tab and in-toto attestations.

Why this shape

Two checks: AA-001 blast radius (if it misbehaves, how much breaks?) and AA-002 declared vs observed. Unknown servers come out UNKNOWN — scored conservatively, never a silent pass. An inferred guess (e.g. a tool named billing.issue_refund → financial) only ever triggers review, never fails a build.

It's deterministic — no LLM in the verdict — makes no network calls, and never reads a secret's value. Every release ships Sigstore-signed SLSA provenance, and the whole thing is Apache-2.0: https://github.com/kunko-ai-labs/agent-assurance