TL;DR
While validating an LLM security dataset with a 3-judge LLM-as-judge pipeline, one threat category — impersonation — hit 98.2% inter-judge disagreement (3/166 examples with unanimous-enough agreement), far above every other category. Instead of treating this as annotation noise, I built a small framework to decompose "impersonation" into four independently-scoreable axes, defined a per-axis ambiguity metric, ran a 40-example pilot, and documented where the pilot's results disagreed with my own hypotheses. Preprint linked at the bottom, code included.
The setup: why one category broke the pipeline
I was building SemGuard, a multilingual LLM security gateway, and needed to validate a generated dataset of candidate threat examples before training anything on it. The validation method: run every candidate through three independent judge models — GPT-4o, Grok-4, Llama 3.3 70B — and treat unanimous-enough agreement as "this label is trustworthy."
Six threat categories behaved normally, with disagreement scaling roughly with how genuinely contested the category is:
| Category | Disagreement rate |
|---|---|
| Safe queries | 5.0% |
| Phishing / privacy leakage | 15.2–39.5% |
| Injection / jailbreak / violent incitement | 72.2–72.7% |
| Impersonation | 98.2% |
That last number isn't a rounding artifact. Out of 166 generated impersonation candidates, the three judges agreed closely enough on three. Tightening the annotation rubric to reduce ambiguity — the standard fix — made agreement worse, not better. That's the signature of a genuinely multi-dimensional construct being forced through a one-dimensional decision boundary, not measurement noise.
The core idea: stop asking one question
The hypothesis: "impersonation" isn't a single latent variable. It's at least four separable judgments getting silently collapsed into a binary label:
R — Target Realism: real, identifiable target vs. fictional character
D — Deceptive Intent: signals of concealing the fictional/synthetic frame from a third party
C — Consent/Context Boundedness: transparently scoped role-play vs. framed to escape the frame
A — Downstream Actionability: could the output be reused outside the conversation to cause harm
Instead of f(x) → {0, 1}, score x as a vector (R, D, C, A) ∈ [0,1]^4, independently, per judge.
Quantifying the disagreement: the Impersonation Ambiguity Index
For m independent judges scoring axis a, the per-axis ambiguity is the scaled population variance:
def iai_axis(scores: list) -> float:
"""scores: list of m judge scores in [0,1] for one axis of one example."""
m = len(scores)
mean = sum(scores) / m
pop_var = sum((s - mean) ** 2 for s in scores) / m
return 4 * pop_var # bounded in [0,1] for any m, by Popoviciu's inequality
def composite_iai(axis_scores: dict) -> float:
# max, not mean: agreement on 3 axes doesn't cancel out
# irreducible disagreement on the 4th
return max(iai_axis(v) for v in axis_scores.values())
Worth flagging since it bit me in an earlier draft: use population variance (/m), not sample variance (/(m-1)). With small judge counts (e.g. m=3), the naive sample-variance form can push the result above 1, breaking the [0,1] bound the whole downstream gating logic depends on. Population variance, scaled by 4, stays bounded for any m ≥ 2 by Popoviciu's inequality — with the minor wrinkle that for odd m the ceiling of 1 isn't actually reachable (an m=3 maximal split tops out at 8/9), which is a property of finite discrete disagreement, not a bug.
Running a real pilot
I ran this on 40 examples (random sample, seed 42) from a pool of 527 SemGuard-rejected impersonation candidates, using the same three judges via Azure AI Foundry (GPT-4o, Grok-4) and Groq (Llama 3.3 70B), each judge returning all four axis scores per example in one structured JSON call.
Axis-level Fleiss' κ:
| Axis | κ |
|---|---|
| D | 0.650 |
| C | 0.433 |
| R | 0.312 |
| A | 0.302 |
Pairwise axis correlation:
| Pair | r |
|---|---|
| R–D | 0.810 |
| D–A | 0.763 |
| D–C | 0.756 |
| R–A | 0.714 |
| C–A | 0.701 |
| R–C | 0.536 |
Composite IAI distribution: mean 0.394, median 0.222, max 0.889, with 10/40 (25%) examples above the 0.8 "maximally ambiguous" threshold.
Where the pilot broke my own predictions
I'd expected R and A to be the most language-agnostic, highest-agreement axes, and D/C to be the most culturally sensitive with lower agreement. The pilot showed the reverse ordering entirely — D had the highest agreement, A the lowest.
I'd also expected only one axis pair (D/C) to be meaningfully correlated, with R/A staying independent. Instead, 5 of 6 pairs cleared the 0.6 correlation threshold I'd set as a flag.
Rather than quietly dropping these predictions from the writeup, I documented both, along with a confound I can't yet rule out: scoring all four axes in one combined judge call is exactly the kind of setup recent 2026 LLM-as-a-judge literature has shown to be susceptible to position bias (score inflation tied to an option's position in a rubric list) and halo effects (a judge's read on one dimension biasing its score on an unrelated one in the same response). I've specified — but not yet run — a two-armed ablation to test this: isolated single-axis calls, and a balanced-permutation rotation of axis order across runs, reporting raw percent agreement alongside κ (since κ is known to deflate substantially relative to raw agreement in large-scale judge audits).
Try it / build on it
The pilot script (Azure AI Foundry + Groq judges, Fleiss' κ, pairwise correlation, IAI computation) is straightforward to adapt to other ambiguous security-labeling categories — the four-axis decomposition here is specific to impersonation, but the "decompose before you threshold, then measure the residual disagreement per axis" pattern isn't.
Preprint (Zenodo, DOI): https://doi.org/10.5281/zenodo.22302106
Happy to talk through the methodology, the ablation design, or the axis-correlation question in the comments — genuinely unresolved and I'd like more eyes on it before running the full-scale (500+ example) protocol.
Top comments (0)