When we build an AI agent, we usually expect it to complete a task and return the result.
The workflow looks simple:
User
↓
Agent
↓
Output
↓
Done
But there is one important question:
How does the agent know whether its output is actually good enough?
This is where RubricMiddleware in Deep Agents becomes useful.
What is a Rubric?
A rubric is simply a set of criteria that defines what a good result should contain.
For example, if we ask an agent to write a research report, our rubric could be:
- Include an introduction
- Discuss existing research
- Include citations
- Identify research gaps
- Provide a clear conclusion
The agent generates the report, and another model checks whether these requirements were satisfied.
How does RubricMiddleware work?
The basic workflow looks like this:
┌─────────────────┐
│ Agent │
└────────┬────────┘
↓
Output
↓
┌─────────────────┐
│ Grader Agent │
│ (LLM Judge) │
└────────┬────────┘
↓
Check rubric
↓
┌───────────┴───────────┐
↓ ↓
Everything OK Something wrong
↓ ↓
SATISFIED NEEDS_REVISION
↓
Feedback injected
↓
Agent
↓
New output
↓
Grader
So instead of simply:
Agent → Output → Done
we now have:
Agent → Output → Check → Fix → Check → Done
This is the main idea behind RubricMiddleware.
Worker Agent vs Grader Agent
One important thing to understand is that there are effectively two roles.
1. Agent
The main agent performs the actual task.
For example:
"Research sensor-based tree disease detection and write a report."
The agent searches for information, reads papers, analyzes them, and creates the report.
2. Grader
The grader doesn't primarily perform the original task.
Instead, it asks:
"Does the result satisfy the requirements?"
For example:
Rubric:
✓ Did the report discuss existing research?
✓ Did it include enough papers?
✓ Are claims supported by citations?
✓ Did it identify research gaps?
✓ Is the conclusion clear?
If everything passes:
SATISFIED
If something is missing:
NEEDS_REVISION
The feedback is then sent back to the agent so it can improve the result.
A Simple Example
Imagine asking an agent:
"Write a Python function to check whether a number is prime."
Our rubric could be:
1. Handle numbers smaller than 2.
2. Correctly identify prime numbers.
3. Correctly identify non-prime numbers.
4. Avoid unnecessary iterations.
The agent might initially create a working function, but use an inefficient loop.
The grader checks the rubric:
1. PASS
2. PASS
3. PASS
4. FAIL
So instead of accepting the result, the system sends feedback:
The function works correctly, but the implementation
should avoid checking every number up to n.
The agent revises the code and the grader checks it again.
If everything passes:
PASS
↓
SATISFIED
↓
Final result
Why is this useful?
Without a rubric, an agent can confidently produce an incomplete or incorrect result.
RubricMiddleware adds a quality-control loop around the agent.
It is especially useful when the task has clear requirements, such as:
- Research reports
- Coding tasks
- Document generation
- Data analysis
- Structured outputs
- Complex multi-step tasks
You can also configure a maximum number of iterations so the agent doesn't keep revising forever.
Rubric vs System Prompt
These two concepts are easy to confuse.
A system prompt tells the agent how it should behave.
You are a senior Python developer.
Write clean and maintainable code.
A rubric defines how the final result should be evaluated.
- Tests must pass
- No hardcoded secrets
- Code must use type hints
- No unnecessary dependencies
In simple terms:
System Prompt
↓
How should the agent behave?
Rubric
↓
Is the final result good enough?
The Main Idea
The easiest way to remember RubricMiddleware is:
DO
↓
CHECK
↓
┌─────┴─────┐
↓ ↓
PASS FAIL
↓ ↓
DONE FIX
↓
CHECK
It essentially turns an AI agent into a generate → evaluate → revise system.
That is the main idea of RubricMiddleware in Deep Agents.
A runnable code example:
"""A small, runnable introduction to Deep Agents grading rubrics.
This example demonstrates the complete rubric loop from the documentation:
1. The deep agent receives a normal user request and writes an answer.
2. A separate grader model checks that answer against ``RUBRIC``.
3. The grader can call ``run_test_suite`` to collect objective evidence.
4. If a criterion fails, the feedback is sent back to the deep agent.
5. The loop ends when the answer is satisfied or ``max_iterations`` is reached.
The agent is asked to write a ``find_duplicates`` function. The rubric is
passed at invocation time because it describes what must be true for this
particular request. The middleware is configured once and can be reused for
other invocations with different rubrics.
Run this example with:
uv run rubric_example.py
It requires a provider API key in the environment or in a ``.env`` file. This
project already uses NVIDIA models, so NVIDIA is the default provider. The
NVIDIA endpoint used here supports one tool call per response, so the example
explicitly disables parallel tool calls. You can override the model names with
``DEEP_AGENT_MODEL`` and ``RUBRIC_MODEL``.
Important: ``run_test_suite`` executes generated Python in the current
process. That is useful for learning, but unsafe for untrusted code. In a
real application, run generated code in an isolated sandbox.
Documentation:
https://docs.langchain.com/oss/python/deepagents/rubric
"""
import os
from typing import Any
from dotenv import load_dotenv
from langchain.chat_models import init_chat_model
from langchain.tools import tool
from langgraph.checkpoint.memory import InMemorySaver
from deepagents import RubricMiddleware, create_deep_agent
from deepagents.middleware.rubric import RubricEvaluation
load_dotenv()
AGENT_MODEL = os.getenv(
"DEEP_AGENT_MODEL",
"nvidia:meta/llama-3.1-8b-instruct",
)
GRADER_MODEL = os.getenv("RUBRIC_MODEL", AGENT_MODEL)
def initialize_model(model_name: str):
"""Create a model, disabling parallel tools for NVIDIA endpoints."""
model_kwargs = (
{"parallel_tool_calls": False}
if model_name.startswith("nvidia:")
else None
)
return init_chat_model(model_name, model_kwargs=model_kwargs)
@tool
def run_test_suite(code: str) -> dict[str, Any]:
"""Run focused tests against the generated find_duplicates source code."""
namespace: dict[str, Any] = {"__builtins__": __builtins__}
try:
exec(code, namespace)
except Exception as exc:
return {
"ok": False,
"failures": [f"The generated source could not be executed: {exc}"],
}
find_duplicates = namespace.get("find_duplicates")
if find_duplicates is None or not callable(find_duplicates):
return {
"ok": False,
"failures": ["A callable function named find_duplicates is required"],
}
tests = [
("basic duplicates", [1, 2, 2, 3, 1], [2, 1]),
("empty input", [], []),
("no duplicates", [1, 2, 3], []),
("unhashable values", [[1], [1], 2], [[1]]),
]
failures: list[str] = []
for test_name, values, expected in tests:
try:
actual = find_duplicates(values)
except Exception as exc:
failures.append(f"{test_name}: raised {exc!r}")
continue
if actual != expected:
failures.append(
f"{test_name}: expected {expected!r}, got {actual!r}"
)
return {"ok": not failures, "failures": failures}
def print_evaluation(evaluation: RubricEvaluation) -> None:
"""Display the grader's verdict after each pass through the loop."""
iteration = evaluation["iteration"] + 1
print(f"\nGrader pass {iteration}: {evaluation['result']}")
print(f"Explanation: {evaluation['explanation']}")
for criterion in evaluation["criteria"]:
status = "PASS" if criterion["passed"] else "FAIL"
detail = criterion.get("gap", "")
suffix = f" - {detail}" if detail else ""
print(f" [{status}] {criterion['name']}{suffix}")
def build_agent():
"""Build an agent whose output is checked by an LLM-as-a-judge loop."""
agent_model = initialize_model(AGENT_MODEL)
grader_model = initialize_model(GRADER_MODEL)
rubric_middleware = RubricMiddleware(
model=grader_model,
system_prompt=(
"You are a careful code reviewer. You MUST call run_test_suite on "
"the final generated source before deciding the verdict. Treat "
"its ok and failures fields as authoritative evidence for the "
"test criterion; do not infer that tests pass without running it. "
"Give specific feedback for every failed criterion."
),
tools=[run_test_suite],
max_iterations=5,
on_evaluation=print_evaluation,
)
return create_deep_agent(
model=agent_model,
system_prompt=(
"You are a careful Python engineer. Return only readable Python "
"source code, with a function named find_duplicates. Follow the "
"user's requirements exactly."
),
middleware=[rubric_middleware],
checkpointer=InMemorySaver(),
)
def main() -> None:
agent = build_agent()
user_request = (
"Write a Python function named find_duplicates(values) that returns "
"every value appearing more than once. Preserve the order in which "
"duplicates first appear, include each duplicate once, and support "
"unhashable values such as lists. Return only the Python source code."
)
rubric = (
"- run_test_suite reports ok=True for every test\n"
"- The function is named find_duplicates and accepts one list argument\n"
"- Duplicate values appear once and in first-duplicate order\n"
"- The implementation supports unhashable values such as lists"
)
result = agent.invoke(
{
"messages": [{"role": "user", "content": user_request}],
"rubric": rubric,
},
config={"configurable": {"thread_id": "rubric-learning-example"}},
)
print("\nFinal agent answer")
print("=" * 60)
print(result["messages"][-1].content)
if __name__ == "__main__":
main()
output
Grader pass 1: satisfied
Explanation: The provided code meets all the criteria in the rubric.
[PASS] run_test_suite reports ok=True for every test
[PASS] The function is named find_duplicates and accepts one list argument
[PASS] Duplicate values appear once and in first-duplicate order
[PASS] The implementation supports unhashable values such as lists
Final agent answer
============================================================
This code defines a function `find_duplicates` that takes an iterable of values as input and returns a list of values that appear more than once in the input. The function uses an `OrderedDict` to keep track of the order in which each value first appears, and a `set` to keep track of the values that have appeared more than once. It iterates over the input values, and for each value, it checks if it has been seen before. If it has, it adds the value to the set of duplicates. Finally, it returns the list of duplicates. The function is written in Python and is saved to a file named `duplicates.py` in the current directory.
For the official implementation details and configuration options, see the Deep Agents Rubric documentation.
Top comments (0)