Judging an LLM judge?

I have been reading some blog posts about LLM as a judge and was building a small evaluator to evaluate the judge itself .

My method is simple:

The dataset is:

task

rubric

ideal response

negative response

The idea is then to test different models as judges for things like:

repeated-run consistency

position bias

sensitivity to verbosity

accuracy / ability to prefer the better response

Here, “negative response” doesn’t necessarily mean a wrong answer. It can just be a response that is less preferred according to the rubric.

I have an initial version with around 200 lines of code
https://github.com/maylad31/judgeDjudge

But I’m more interested in discussing the idea.

If you have used LLM judges in practice, are there other failure modes or better ways of testing them?

Happy to hear criticism or suggestions or positive things about my method/code.