Blog 10: Minecraft Mod Agent- I Made My AI Agent Judge Itself — And Used the Scores to Make It a Better Combat Advisor

Cloud Swords Mod — Blog 10

---

In Blog 9, I gave my Minecraft mod an AI brain. A Strands Agent that decides how many allies to summon based on your combat situation. It works. But "works" isn't a metric.

How do I know the AI is making good decisions? When a player has 3 hearts and 8 zombies closing in, does the agent summon 5 allies (correct) or 1 (death sentence)? When the player is full health with no threats, does it correctly do nothing — or waste resources summoning minions for no reason?

I needed an evaluation framework. So I built one. And the judge? Another AI.

---

The Problem: Non-Deterministic Decisions

The agent uses gpt-oss:20b. Same input can produce slightly different outputs between calls. You can't write assert response == expected because there's no single correct answer.

What you CAN evaluate:

• Is the action valid JSON? (deterministic check)
• Is the action within bounds? (1-5 minions, valid buff types)
• Is the response proportional to the threat? (requires judgment)
• Is the reasoning coherent? (requires judgment)

The first two are code. The last two need an LLM.

---

The Eval Dataset: Real Interactions

Remember interactions.json from the mock server? Every time the agent makes a decision, it's logged:

{
  "timestamp": "2026-05-19T19:17:34",
  "payload": {
    "player": "Carlos",
    "weapon": "sword_of_lambda",
    "health": 5,
    "nearby_mobs": 10,
    "biome": "nether",
    "dimension": "the_nether"
  },
  "decision": {
    "action": "summon",
    "count": 5,
    "duration": 10,
    "reason": "low health, many mobs"
  },
  "source": "agent"
}

I took 20+ real interactions and added expected behavior ranges:

eval_dataset = [
    {
        "context": {"health": 5, "nearby_mobs": 10, "biome": "nether", "dimension": "the_nether"},
        "expected_action": "summon",
        "expected_count_range": [4, 5],
        "expected_reasoning": "Should recognize critical threat (low HP + many mobs + dangerous biome)",
        "tags": ["critical", "nether"]
    },
    {
        "context": {"health": 20, "nearby_mobs": 0, "biome": "plains", "dimension": "overworld"},
        "expected_action": "none",
        "expected_count_range": [0, 0],
        "expected_reasoning": "Full health, no threats — should do nothing",
        "tags": ["safe", "overworld"]
    },
    {
        "context": {"health": 12, "nearby_mobs": 3, "biome": "dark_forest", "dimension": "overworld"},
        "expected_action": "summon",
        "expected_count_range": [1, 3],
        "expected_reasoning": "Moderate threat — proportional response",
        "tags": ["moderate", "overworld"]
    },
    # ... 17 more scenarios
]

---

Level 1: Code Evaluators (Free, Instant)

def eval_valid_json(response: str) -> bool:
    """Can we parse the response as JSON?"""
    try:
        json.loads(response)
        return True
    except:
        return False

def eval_action_in_bounds(decision: dict) -> bool:
    """Is the action within allowed bounds?"""
    action = decision.get("action")
    if action == "summon":
        count = decision.get("count", 0)
        duration = decision.get("duration", 0)
        return 1 <= count <= 5 and 5 <= duration <= 15
    elif action == "buff":
        valid_effects = {"regeneration", "resistance", "strength", "speed"}
        return decision.get("effect") in valid_effects and 3 <= decision.get("duration", 0) <= 10
    elif action == "none":
        return True
    return False

def eval_has_reason(decision: dict) -> bool:
    """Does the decision include a reason?"""
    reason = decision.get("reason", "")
    return len(reason) > 3 and len(reason) < 50

def eval_proportional(decision: dict, context: dict, expected: dict) -> bool:
    """Is the response roughly proportional to the threat?"""
    if expected["expected_action"] == "none" and decision.get("action") == "none":
        return True
    if expected["expected_action"] == "summon":
        count = decision.get("count", 0)
        low, high = expected["expected_count_range"]
        return low <= count <= high
    return decision.get("action") == expected["expected_action"]

These catch 60% of problems instantly: malformed JSON, out-of-bounds values, missing reasons, wildly disproportionate responses.

---

Level 2: LLM-as-Judge

For the subjective stuff — "is this a good decision?" — I use the same model as a judge:

JUDGE_PROMPT = """You are evaluating an AI combat advisor for a Minecraft mod.

Given the player context and the AI's decision, score it on:
1. **Threat Assessment** (1-10): Did the AI correctly gauge the danger level?
2. **Proportionality** (1-10): Is the response scaled to the threat?
3. **Reason Quality** (1-10): Is the reasoning clear and accurate?

Context: {context}
AI Decision: {decision}
Expected behavior: {expected}

Respond ONLY with JSON:
{{"threat_assessment": X, "proportionality": X, "reason_quality": X, "feedback": "one sentence"}}"""

def judge_decision(context: dict, decision: dict, expected: dict) -> dict:
    prompt = JUDGE_PROMPT.format(
        context=json.dumps(context),
        decision=json.dumps(decision),
        expected=expected["expected_reasoning"]
    )
    result = agent(prompt)
    return json.loads(str(result))

The meta part: the same AI model that makes the decisions is also judging them. This works because the judge has the expected behavior as reference — it's not evaluating in a vacuum.

---

The Eval Runner

class CombatAdvisorEval:
    def __init__(self, agent):
        self.agent = agent
        self.results = []

    def run(self, dataset: list[dict]) -> dict:
        for case in dataset:
            # Get agent decision
            decision = decide(self.agent, case["context"])

            # Level 1: Code checks
            checks = {
                "valid_json": eval_valid_json(json.dumps(decision)),
                "in_bounds": eval_action_in_bounds(decision),
                "has_reason": eval_has_reason(decision),
                "proportional": eval_proportional(decision, case["context"], case),
            }

            # Level 2: LLM Judge
            judge_scores = judge_decision(case["context"], decision, case)

            self.results.append({
                "context": case["context"],
                "decision": decision,
                "checks": checks,
                "judge_scores": judge_scores,
                "tags": case["tags"],
            })

        return self._summary()

    def _summary(self) -> dict:
        total = len(self.results)
        return {
            "total_cases": total,
            "code_pass_rate": sum(
                all(r["checks"].values()) for r in self.results
            ) / total,
            "avg_threat_assessment": sum(
                r["judge_scores"]["threat_assessment"] for r in self.results
            ) / total,
            "avg_proportionality": sum(
                r["judge_scores"]["proportionality"] for r in self.results
            ) / total,
            "avg_reason_quality": sum(
                r["judge_scores"]["reason_quality"] for r in self.results
            ) / total,
        }

---

A/B Testing System Prompts

The real power: I can test different system prompts and measure which one produces better decisions.

PROMPT_A = """You are a combat advisor. Given context, decide one action..."""  # Original

PROMPT_B = """You are a combat advisor. IMPORTANT: Always consider the biome danger level.
Nether and End are inherently more dangerous than Overworld.
Scale your response accordingly..."""  # Biome-aware variant

# Run eval with both
agent_a = Agent(model=model, system_prompt=PROMPT_A)
agent_b = Agent(model=model, system_prompt=PROMPT_B)

results_a = CombatAdvisorEval(agent_a).run(eval_dataset)
results_b = CombatAdvisorEval(agent_b).run(eval_dataset)

print(f"Prompt A — proportionality: {results_a['avg_proportionality']:.1f}/10")
print(f"Prompt B — proportionality: {results_b['avg_proportionality']:.1f}/10")

Prompt B scored higher on Nether scenarios because it explicitly considers biome danger. That insight came from the eval — not from vibes.

---

The Feedback Loop

Observe (interactions.json)
    ↓
Evaluate (code checks + LLM judge)
    ↓
Adjust (modify system prompt)
    ↓
Measure (re-run eval, compare scores)
    ↓
Deploy (update prompt in mock server)
    ↓
Observe again...

This is the same loop from the LLMOps series (Part 3: Eval-Driven Development) — but applied to a Minecraft mod. The pattern is universal.

---

Results: What I Found

After running evals across 20 scenarios:

Metric	Prompt v1	Prompt v2 (biome-aware)
Code pass rate	95%	95%
Threat assessment	7.2/10	8.4/10
Proportionality	6.8/10	8.1/10
Reason quality	7.5/10	7.8/10

The biome-aware prompt improved threat assessment by 1.2 points — specifically in Nether and End scenarios where v1 was under-responding.

The 5% code failure rate? JSON parsing issues when the model occasionally adds explanation text before the JSON. Fixed by making the parser more robust (find first {, last }).

---

Cloud/AI Concepts Taught

Eval Concept	What It Teaches
Code evaluators	Input validation, schema enforcement
LLM-as-Judge	AI evaluating AI (meta-evaluation)
Eval dataset	Test cases for non-deterministic systems
A/B testing prompts	Experimentation, data-driven decisions
Feedback loop	Continuous improvement, observability
Scoring thresholds	Quality gates, SLOs for AI
interactions.json	Observability, audit trails

---

What I'd Do Next

1. Automated eval in CI — every prompt change triggers eval, blocks if scores drop
2. Player feedback — thumbs up/down after each cloud invocation, feeds back into dataset
3. Seasonal prompts — different system prompts for different game phases (early/mid/endgame)
4. Multi-model comparison — test gpt-oss:20b vs Claude Haiku vs Nova Lite for cost/quality tradeoff

---

The Full Picture

Across 10 blog posts, this mod went from "7 swords with cloud names" to a full system with:

• Custom textures generated with Python + AI
• Energy systems with BFS algorithms
• Multiblock machines with custom GUIs
• A spell system with 28 spells
• Armor sets with dodge mechanics
• Wandering merchants with elite progression
• A serverless AI backend
• An evaluation framework for AI decisions

All teaching cloud computing through gameplay. All open source.

---

This concludes the Cloud Swords blog series. 10 posts, one mod, and more cloud concepts than most certification courses. Thanks for reading.

---

Code:

• Mod: cloud-swords-mod-1.20.1
• Backend: cloud-swords-backend-cdk
• Agent: cloud-swords-strands-agent

---

I'm Carlos Cortez — and this is Breaking the Cloud.

LinkedIn · GitHub · Dev.to

Comments

[Mads Hansen]

This is a great concrete eval loop. I would be careful with the same model generating and judging, though: shared blind spots can make both versions look better while external quality stays flat. Calibrate the judge against a small human-labeled gold set, randomize whether A or B appears first, hide the prompt/version identity, run multiple seeds, and report paired win rate plus confidence intervals rather than only average scores. The deterministic expected ranges already give you a strong anchor; use the LLM judge mainly for disagreement analysis. One implementation note: extracting from the first { to the last } can accept malformed mixed output. Schema-constrained generation plus strict validation/retry keeps “valid JSON” as a real invariant instead of weakening the parser until the metric passes.