DEV Community: Jan-Felix Schmakeit

How to Write Reliable Rubrics for LLM-as-a-Judge Evaluations

Jan-Felix Schmakeit — Wed, 02 Sep 2026 16:35:00 +0000

Follow up to Part 1: How to Design AI Evaluations You Can Actually Trust

At Google, we are publishing a suite of Agent Skills for Google products and technologies on GitHub. My team is interested in measuring their performance to understand how they perform. Deterministic tests, like checking if generated code compiles, are ideal. Unfortunately they can't be easily created at scale for nuanced, generative responses, such as answers to open-ended questions or information retrieval tasks.

In my previous post we looked at what you test, which means the evaluations that are the actions you ask the agent to perform. The next step is to look at how you assert whether the agent succeeded. This means creating reliable and accurate assessments of the agent's response.

To assess complex outputs at scale, especially when topics cover broad areas with nuanced parts, we use an "LLM-as-a-judge" approach. Responses are assessed against a structured rubric using a model based grader. The judge evaluates each response using a set of true/false questions. Aggregated, these answers provide an accuracy score for a response.

Giving an LLM a vague prompt or subjective questions leads to ambiguity in its response. This ambiguity introduces noisy data and causes inconsistent evaluations. Ultimately, it wastes your token budget on metrics that aren't useful.

To make these assessments more reliable, you must treat your rubrics like formal specifications. By constraining the judge to evaluate strict, objective boolean truths, you reduce the chance of hallucinations. Because evaluating strict boolean truths is a less complex task, you can even use smaller, faster models for grading.

Here are four lessons we learned to help you write robust rubric questions for your LLM-as-a-judge scorer.

Keep Questions Atomic and Distinct

Evaluating multiple requirements in a single question, such as "Does the response contain a metadata property and format the output as JSON?", forces the LLM judge to guess which clause is more important. This ambiguity leads to inconsistent grading and wasted tokens.

Split compound questions: Instead of writing one large check, divide your requirements into discrete, atomic TRUE/FALSE questions. (e.g., Check 1: Does it contain a metadata property? AND Check 2: Is the output JSON?)
Avoid overlapping questions: Never test the same underlying concept multiple times across your rubric. Overlap risks double-penalizing the evaluated model for a single mistake, which corrupts your accuracy score.
Reduce the reasoning load: Eliminate the need for the judge to weigh competing clauses. When every question assesses exactly one distinct fact, your grading becomes more consistent.

Constrain the Judge: Objective Facts Over Subjective Reasoning

The rubric-based approach exists because giving an LLM judge a full prose prompt to evaluate a complex response leads to inconsistent numbers. If you ask the judge subjective questions like "Was this a comprehensive answer?" or ask it to explain "Why did the agent do this?", you introduce ambiguity that creates noisy, unrepeatable data.

Focus on observable facts: Do not ask the judge to evaluate concepts that require interpretation, such as intent, quality, or reasoning. Only assess concrete facts that you expect to find in the response.
Write formal specifications: Use strict, objective language, such as RFC 2119 terminology (MUST, MUST NOT, REQUIRED), to test for observable outcomes. The judge should never have to guess what you mean.
Test for negative constraints: Explicitly verify what the agent should not do. Instead of asking if the agent "used best practices", check that it did not suggest a specific deprecated feature.
Require strict true/false answers: By forcing a strict TRUE/FALSE classification on objective facts, you reduce the reasoning load and reduce the variance in ratings.
Avoid opportunities for cheating: Agents will tailor answers to game your tests if given the chance. Keep scoring rubrics isolated in a separate system. Design rubrics that focus on strict functional outcomes or specific topics, rather than broad keywords matches.

Grade Only What You Asked For

When building rubrics, it is easy to accidentally evaluate an agent on requirements that were never stated in the prompt. Doing this creates false negatives and reduces the accuracy of your measurement.

Align the rubric to the prompt: Assess only what was explicitly requested. For example, do not penalize a model for failing to provide citations if the prompt never asked for them.
Grade the destination, not the journey: Avoid writing rubrics that check if the agent used a specific tool or followed a rigid sequence of steps. Pre-trained models might bypass custom tools entirely if they already know the answer.
Evaluate the final response: Grade the objective output. If you need to evaluate a step-by-step process, prompt the agent to output an execution plan and evaluate the plan itself.

Calibrate Your Judge

Even if you follow these rules and write perfectly atomic, objective questions, your LLM judge might still misinterpret your scoring instructions and rubrics. To guarantee your pipeline generates consistent ratings and reliable signals, you must prove that the judge's scoring aligns with how a human subject matter expert would evaluate the exact same response through calibration.

Establish a human baseline: Ask subject matter experts to manually grade a "golden set" of test responses.
Run the comparison: Run your LLM-as-a-judge against this golden set and compare the automated scores against the human scores.
Identify the delta: If the LLM judge disagrees with your human experts, it is usually a sign that your rubric or the grading instructions are too ambiguous.
Iterate until aligned: Tweak and calibrate your rubric questions or the scoring instructions until the LLM judge consistently aligns with human experts. Only then is your judge ready.

Summary

Once you have this reliable data, the next step is making it visible. In AI Evals at a Glance: Heatmaps for Stakeholders Joe Spiro explains how to take these raw measurements and visualize evaluations.

When building our agent skills, we learned that ambiguous evaluation rubrics do not provide useful signals and feedback. Forcing your LLM judge to evaluate strict, boolean facts eliminates this noise. It makes your tests repeatable, optimizes your token spend, and lets you confidently measure whether your AI tool is actually improving.

Photo by William Warby on Unsplash

How to Design AI Evaluations You Can Actually Trust

Jan-Felix Schmakeit — Tue, 01 Sep 2026 16:35:00 +0000

As part of my work at Google, we are publishing a suite of Agent Skills for Google products and technologies on GitHub. These agent skills are designed to help AI agents interact with our technologies. But how do you test that these skills are useful and work as expected? My team in Developer Relations has been focused on this question, because having reliable signals on their performance is critical to help us improve them over time.

Just as you wouldn't deploy a production API without writing unit tests, you should apply the same standard to your AI agents. As Joe Spiro showed in the Designing AI Evals post series, scaling AI tools means moving beyond "vibe testing" in a terminal. Instead, you should set up a structured, automated evaluation pipeline to benchmark your integration. The evaluations (evals) are the actions you asked the agent to perform, which are graded using scorers (for example rubrics) that assert whether the agent succeeded. We'll focus on evaluations in this post and tackle tips for scoring rubrics in the next post.

However, AI evaluations cost real tokens. You need to make sure that you use these tokens as efficiently as possible. They need to provide real value that helps you build better tools. Writing good evaluations is critical. Poor evaluations provide false signals, waste your token budget, and create noise in your metrics.

Here are five rules we learned to design better evaluations you can trust. Follow them to ensure that every token you spend produces a useful metric.

Know Your Evaluation Environment

Before writing evaluations, you need to understand the setup and limitations of your chosen framework. This includes systems like Harbor, Inspect AI, or integrations in development tools like in the Agent Development Kit. Does it use an ephemeral sandbox? What tools are available? How is the output captured

Tailor your graders to the environment: For example, if you can deterministically access the sandbox to evaluate code, that's an option. Alternatively, ask the agent to print its response to the console. Your framework can capture this output and pass it directly to your scorers. Adjust your grader to handle these environments.
Be aware of dependency limitations and access to "real" resources: If an evaluation task requires access to "real" resources (for example an authenticated gcloud session with access to a Google Cloud project), create ephemeral resources or credentials that isolate and limit access so they don't impact other evaluations. Alternatively, you could provide mock tools instead of real test credentials.
Evaluate plans instead of tasks that are difficult to isolate: An easier approach might be to evaluate the plan to accomplish the task, rather than the actual execution.
Avoid interactive prompts: Multi-turn agent sessions are complex to evaluate. When getting started, design your evaluations using one-shot prompts.

Avoid the Ceiling Effect

If your evaluations show a high baseline accuracy (i.e., without your agent tool), it might not prove its value, or the evaluation prompts are too easy.

Write harder prompts: You cannot measure the impact of a new agent tool or skill if the baseline model already knows the answer.
Require multi-step reasoning: Design prompts that reflect complex, real-world use cases where your tool can actually differentiate itself from the model's pre-training.
Revisit the scope of your tool: If tests are repeatedly reporting a high accuracy without using your tool, it might be time to revisit it. The underlying model and agent may have improved and are able to accomplish the task without additional help. It might be time to refocus or deprecate your tool.

The Prompt-Grader Mismatch

You cannot grade an agent on something you did not explicitly ask it to do. Your evaluation prompts and graders should be complementary. This means that they should only test for things included in the prompt.

Avoid scope creep: If you asked a broad question, you can expect a similarly broad response. For example, if your prompt is "How do I secure Google Cloud Run?", your grader cannot penalize the agent for missing a specific, unprompted IAM role.
Be explicit: If you want to evaluate specific knowledge or exact implementation details, you must state those requirements clearly in the prompt.

Grade the Destination, Not the Journey

Agents possess inherent model knowledge and might skip your custom tools entirely to arrive at the correct answer. (That on its own is some useful feedback!)

Do not evaluate the trajectory: Avoid writing graders that check if the agent used a specific help command or followed a rigid sequence of steps.
Evaluate the final answer: Grade the objective output. If you absolutely must evaluate the agent's planning phase, explicitly ask it to output a detailed execution plan and evaluate the plan instead.

Curate Your Evaluation Dataset

A strong evaluation suite tests real, diverse use cases. But testing the same capability repeatedly causes overfitting and creates noisy metrics.

Use real-world examples: Evaluations should include real user journeys and focus on goals users want to achieve. Consider including additional context, such as sanitized sample data, to ground the evaluations.
Maximize your signal: Ensure every prompt in your evaluation suite tests a distinct concept or unique capability. Think of this like code coverage for traditional tests.
Remove overlap: Consolidate redundant prompts. A smaller, more curated data set provides clearer metrics, prevents overfitting and saves tokens.

Summary

You cannot improve AI tools if you can't measure them accurately. If you treat AI evaluations with the same focus as traditional unit tests, you improve the quality of your metrics and get more robust signals.

By applying these five rules, you eliminate false signals that waste your token budget. Instead of generating noise, your test suite gives you actionable feedback you can use to guide your engineering decisions and improve your tools.

Figuring out what to test is only the first step. A well-designed evaluation is only useful if the scorer grading answers is reliable and returns meaningful results. In my next post, we will look at how to test. You will learn how to write lean, atomic rubrics that minimize ambiguity for an LLM grader and make every token count.

Photo by William Warby on Unsplash