DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

A Self-Reflective Agent That Grades and Rewrites Its Own Work Until a Gate Passes

#ai #llm #agents #python

Tell a model "check your work and fix it" and you get a confident "looks good!" over a first draft dressed up as a final one. A real reflection loop does the opposite. This agent produces a draft, grades it against an explicit rubric, writes specific critique, rewrites to fix the named gaps, and re-scores — looping until it clears a quality gate that is deterministic Python, not the model's own opinion. The model drafts, judges, and refines; every part that decides "good enough yet?" is code you can read and test.

The loop 

draft → judge (rubric 1-5 + evidence + critique) → refine under the critique → re-judge → gate ↺
Enter fullscreen mode Exit fullscreen mode

Each task ships a rubric: named criteria (each scored 1-5, each with a definition of what a 5 vs a 1 looks like and a weight) plus deterministic hard checks — objective, non-LLM predicates over the answer text. The judge scores the criteria; Python runs the hard checks and the gate. That split is the whole design.

The deterministic core

The judge only supplies the 1-5 scores. Python aggregates them into an overall in [0,1] and makes the pass decision — and the gate requires the score to clear the threshold and every hard check to pass.

def aggregate(scores, rubric):                     # weighted mean of (score-1)/4 -> [0,1]
    acc = w = 0.0
    for cs in scores:
        c = rubric.criterion(cs.criterion_id)
        if c: acc += ((cs.score - 1) / 4.0) * c.weight; w += c.weight
    return round(acc / w, 4) if w else 0.0

def gate(overall, hard_results, threshold):        # the quality gate the loop stops on
    failed = [cid for cid, ok in hard_results if not ok]
    if overall < threshold:  return False, f"score {overall:.2f} < threshold {threshold:.2f}"
    if failed:               return False, f"score OK but hard checks failed: {failed}"
    return True, f"score {overall:.2f} >= {threshold:.2f} and all hard checks passed"
Enter fullscreen mode Exit fullscreen mode

The AND is the crux — a self-flattering score can never clear the gate on its own.

Two real runs on NVIDIA NIM

The page captures two tasks run through the loop against meta/llama-3.1-8b-instruct on NVIDIA NIM — 10 live model calls, not a mock.

The docstring task climbs. The fast first draft comes back with a summary, Args and Returns but no Raises section and no zero/negative-people edge case. The judge correctly scores edge_cases 1/5 and raises 1/5, the documents_raises hard check fails, and the overall lands at 0.56 — below the 0.80 gate. The refiner adds exactly the two things the critique named, keeping the parts that already scored well. Re-judge: all criteria 5/5, hard checks 4/4, gate PASS. Score 0.56 → 1.00.

The discount task exposes self-judge bias. The question: "20% off then 10% off — what single discount off the original?" The draft gets the number right (28%) but never explains why it isn't the intuitive 30%. Here the 8B judge is over-generous: it scored the draft a perfect 1.00, giving itself addresses_trap 5/5 on an answer that never mentions 30% and never mentions that discounts compound. It credited itself for content that wasn't there.

The deterministic addresses_trap hard check caught it — so the gate failed the draft despite the model's perfect score. The failed check is fed back to the refiner as mandatory, the rewrite adds the "not 30% because discounts compound" reasoning, and the hard-check trail rises 2/3 → 3/3. The rubric score never moved (Δ +0.00), yet the answer genuinely improved.

task         rubric-score trail    Δ       hard-check trail    stop reason
-----------------------------------------------------------------------------
docstring    0.56 → 1.00          +0.44    3/4 → 4/4           PASSED
discount     1.00 → 1.00          +0.00    2/3 → 3/3           PASSED
Enter fullscreen mode Exit fullscreen mode

The honest takeaway

That flat score on the discount task is the point of the whole project, and the run keeps it on purpose. A model grading its own work will flatter itself — a known self-judge bias. Logging both the rubric score and the hard-check trail is what keeps the run honest: the score alone would suggest nothing improved, but the hard-check trail captures the real fix and is what drove the gate from FAIL to PASS. Reflection isn't magic, and an LLM judging itself is not enough; the deterministic hard checks are the backstop.

The full loop, the rubric, and the captured transcript are here: https://dev48v.infy.uk/agentic/project10-self-reflective.html — with the code at https://github.com/dev48v/agentic-ai-from-zero.

Next up, Project 11: a production-ready agent with tracing, cost dashboards, and alerting.

Top comments (0)