LLM as a judge

#ai #goldendatasets #codequality

Gone are the hours of careful thought and planning that go into coding a new feature. Vibe coding is too risky though, so another Driven Development was created. I'm referring to SDD (Spec Driven Development) of course. The vibe coding approach is great for prototypes and throwaway code, but this way of working falls apart when teams realise that the code needs to be maintained. So the thing that helps fix this is SDD.

Create a spec once from clear technical specs and then generate some high quality code. Sounds great, right. Reminds me of IaC, where you use a templating language to create infrastructure. Software as Code maybe. SaC anyone? Unfortunately, in practice it's not that straightforward. Thoughtworks have placed SDD into an "Assess" category and warned that it could be an anti-pattern for releasing software.

Deterministic vs Probabilistic

This article isn't about SDD. I'm more interested in discussing the output of SDD and how that is tested. Code can now be generated fast these days. So what better to test AI-written code than with AI itself. There are a lot of concepts and technical terms for the Quality Assurance part of AI generated code. One of these is the LLM-as-a-Judge idea. This idea is used to score the output of an LLM based on some explicit criteria. Traditionally, the way to evaluate an LLM was to judge its output on the helpfulness or faithfulness (using something called "exact-match" metrics). Sometimes it was usually down to a human to do this. It also changes the way that Quality is Assured when dealing with AI-written code.

Traditional QA is built on deterministic checks; either something does or does not fail. Something like expect(x).toContainText(y);. A failing test means that something is wrong. Then the bug can be fixed in the code and the test will pass. However, the outputs of an LLM are probabilistic, so it breaks the traditional pass/fail model. This is where a judge comes in. Instead of pass/fail, it can assign a score based on how good the output is. This changes the focus of the team too, as now the focus becomes reliability and consistency and a continuous evaluation of the code.

There's Gold in those Data Sets

Before I go into a few use cases and examples, I'd like to cover the term Golden Data Set and what it is. LLMs don't really have a good idea of what good looks like until you show it. There are different approaches to this which is where the Golden Data Set comes in. It is just a fancy way to say a versioned set of input/output test cases. The data set includes passes and fails, known issues, edge cases, and some of the tests have the expected outcomes and some do not. This data set can be taken from a working system whose outputs have already been tested and are known to be correct. These assertions can then be used in the first-pass testing of the AI-generated output. Then after each new version of the AI-generated code, we can compare the data sets and use those for testing the next version of the AI-generated code. Think of it like a growing versioned regression test suite. Importantly, the Golden Data Set needs to be curated by a human and it is best to use quality test cases rather than increase the quantity of test cases.

Use case: Regression testing in CI/CD

In this scenario there are 2 LLMs; one that generates the code and one that acts as The Judge. Every PR that touches a prompt, model or config is gated by an LLM Judge. The Judge can then be given rules to run on the output, as well as use the Golden Data Set as a reference to compare the results with. Instead of giving a score like 39/50 test cases passed, it will give a rating of confidence based on whatever was defined previously. Examples could be the quality, detail or even accuracy of the response. Usually, The Judge LLM would run at a temperature of 0 (0 meaning that there would be no creativity in the responses of the Judge). A threshold can be set that blocks the merge. The table below shows an example of what a CICD flow could look like, step-by-step.


1	Trigger	A PR changes a prompt, model version or config
2	Testing	Tests run against a golden dataset
3	Output	Generator produces outputs
4	Scoring	The Judge does this
5	Checking	Scores are compared to a previous run, not evaluated in isolation.
6	Deploy	If a metric we care about regresses, the pipeline blocks merge/deploy
7	Logging	Store logs (timestamp, dataset version, judge model snapshot and judge prompt)

Use case: Evaluating BDD tests

There is a paper called "LLM-as-a-Judge for Scalable Test Coverage Evaluation: Accuracy, Operational Reliability, and Cost" which as the title suggests was a study into LLM-judge framework scoring and a discussion of the idea of "execution-based vs. judgment-based testing". I'll leave a link at the bottom of the page.

100 Jira tickets were selected to simulate a realistic software development life cycle. Then using an LLM, Given-When-Then style BDD tests were generated for each of them. These tests were then reviewed by 3 senior QA Engineers experienced with the product that the test cases targeted.

The Judges, a selection of LLMs with different weightings and reasoning efforts (for example GPT-4, GPT-4o Mini and GPT-5), were tasked with "understanding" the tests. Each Judge received the text of the Jira ticket, the BDD test cases and a few guidelines on what different technical details meant (such as HTTP-methods). They could then assess whether the executed BDD tests address the specified requirements and output a coverage percentage and a written rationale about their scoring.

Use case: Chatbot/agent functional testing

There is a team somewhere building an AI powered chatbot that offers a voice feature. The team are running single-turn eval tests, but customers are complaining the bot is going in circles and forgetting what the user has just said. The problem is that the team aren't using a good testing strategy. The single-turn eval tests are almost like individual unit tests. Just relying on unit tests to deliver quality software isn't a good approach. It's like grading a movie by looking at random frames instead of watching the film.

Below is a list of the important evaluation steps. As you can see, there's a lot of things to evaluate, so this is where the LLM-as-a-Judge can be used instead of manually checking.

Task Completion
- If the task didn't get done, nothing else matters.
Tool call accuracy
- Was the right tool called with the right arguments?
Workflow completion
- Did it follow the intended multi-step sequence with proper data flow between tool calls?
Grounding
- Were the tools called actually used in the response or was the response invented?
Stay in character
- Does it maintain its role the whole way through the conversation? For example a support agent staying empathetic, a sales bot not discussing competitors.
Autonomy
- Was workflow completed without needing the user to guide it?
Knowledge retention
- Does the bot remember what the user already told it?
Follow-up quality
- Did it ask for missing information before going ahead, rather than guessing?
Conversation relevancy
- Does each reply actually track the live thread of the conversation rather than drifting?

Security

Something that I haven't included in the list above is the testing of security in the response. This is because it should actually be a separate activity. There are 3 ways of testing evals:

correctness which includes the golden dataset and The Judge
adversarial/security which includes red teaming, Attack Success Rate tracking (a Red Team activity) and guardrail testing
operational which includes canary deployments and observability

The Judge is itself an LLM, so it inherits LLM vulnerabilities. Testing the adversarial-robustness of The Judge itself is also important through things like fake reasoning, prompt injection, adversarial phrasing and jailbreak-style manipulation. This is more down the path of Red Teaming activity which is getting off topic for this article.

As a QA I hate the sound of probabilistic testing

At this point, you may be thinking "So you're saying that instead of minimising bugs, we're moving to the idea that we're probably going to find a few? That doesn't sound like a good way to do QA". And yeah, good point. With AI now in the loop of generating code it means that QA roles will change. Just as scripting made coming up with test scenarios and creating test data easier, and automated tests made running a suite of tests easier and deployments faster, LLMs become part of a test design approach.

Teams need to use different strategies to test AI-generated code. The fast, deterministic checks where "pass" is unambiguous are always going to be valuable, so they should be kept. These should be layered with a continuous cycle of evaluation. This means that teams should create Golden Data Sets and maintain them, as these collections will drift. The dataset evolves quickly as failures are discovered. Every time a new failure is discovered it should get added as a new case. This is the same philosophy as the pre-AI-coding way of managing suites of tests. The LLM-as-a-Judge model of testing introduces a different way of thinking about testing too. Every time The Judge prompt itself changes, old scores against the dataset aren't directly comparable anymore. In fact, whenever something changes, like the model's settings or preferences or even the model itself, then we need to set a new base-line for The Judge and the data sets.

Something I haven't covered explicitly is that this process has a sometimes mandatory requirement for a human to be in the loop, especially if the code is for an application (like in sensitive, highly regulated environments). Any failures in scores or drops in confidence need a human to investigate further. And that's a general theme in QA that always comes up; if software is being built for humans then a human needs to be in the loop at some point.