Workshop: Score Router Agents Against LLM Tool Choice in 85 Minutes

Classroom agent demos often look autonomous while a hidden branch table still selects every tool. This workshop makes that split measurable with two implementations, one fixture pack, and a turn ledger students can rerun. The core conclusion is simple and testable: keep a deterministic router until phrasing variance actually beats a rule table on the same traces. Free inference capacity is useful here only as a lab endpoint, not as evidence that a chat loop is an agent.

The outline below is a teaching plan, not a production framework and not a personal benchmark report. Every numeric target is a classroom stopwatch budget, not a published SLA. Code samples are labeled workshop sketches that students should run locally against fixtures they control.

Why this comparison belongs in the lab

Recent community debate keeps collapsing two designs into one word: agent. A deterministic router maps verbs and entities onto a closed tool list, then stops. An LLM chooser proposes a tool name from natural language, then the same tools still execute. If those paths are not scored on identical fixtures, a class cannot tell whether extra model calls buy recall, noise, or only longer traces.

Shared inference makes the gap more expensive than it looks on a laptop demo. Each extra turn consumes tokens, waits on a queue, and multiplies retry policy. Students should see that cost on a ledger before they add a third tool or a memory write. The comparison also resists a common demo failure: a scripted if-chain dressed as tool-calling.

Workshop clock

Use a single 85-minute block with a visible timer and no open-ended homework during class.

1. 0–10 min — Frame the claim. Define router, chooser, fixture, and illegal side effect.
2. 10–25 min — Exercise 1. Freeze two tools and a ten-prompt fixture pack.
3. 25–45 min — Exercise 2. Implement and test the deterministic router.
4. 45–70 min — Exercise 3. Wrap an LLM chooser with schema checks and a turn ledger.
5. 70–85 min — Exercise 4. Dual-run the fixtures, fill the decision table, debrief misses.

If the room is slower, drop the chooser retries and keep one model call per prompt. Do not expand the tool set during the same sitting. Extra tools hide scoring errors behind incomplete coverage.

Prerequisites

Students need Python 3.11+, pytest, and permission to make outbound HTTP calls from the lab network. They also need a JSONL fixture file they can commit, plus a dry-run flag on every write tool. No paid cloud account is required if the chooser step can target a free lab endpoint the instructor already configured.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can stand in as that shared lab endpoint so the chooser path does not depend on a personal API invoice. Do not treat that availability as a quota, hardware spec, latency claim, or permanence guarantee; pin whatever URL the instructor actually issued.

Exercise 1 — Freeze two tools and one fixture pack

Give every student the same closed world. One read tool returns inventory rows. One write tool opens a ticket only when dry_run is false. Ten prompts must include clear verbs, paraphrases, mixed intents, and two prompts that should refuse both tools.

Suggested fixture fields, committed as fixtures/tool_choice.jsonl:

• id — stable string, never reused after an edit.
• prompt — the user text, including typos the router must survive or fail.
• allowed_tools — a list that may be empty when the correct action is refusal.
• write_expected — boolean; true only when a ticket would be legitimate.

Label this rule on the board: write tools never execute during scoring. Students who skip that rule are not measuring tool choice; they are creating tickets. Keep the inventory table tiny, for example four SKUs, so expected read arguments stay obvious in review.

Exercise 2 — Ship a deterministic router

The router may use keyword maps, regexes, or a tiny intent table. It may not call a model. It must return a structured decision the tests can diff.

# workshop sketch — deterministic router, not a production NLU stack
from dataclasses import dataclass
import re
from typing import Literal

ToolName = Literal["read_inventory", "open_ticket", "refuse"]

@dataclass(frozen=True)
class Decision:
    tool: ToolName
    args: dict
    reason: str

SKU = re.compile(r"\b(SKU-\d{3})\b", re.I)
TICKET = re.compile(r"\b(open|file|create)\s+(a\s+)?ticket\b", re.I)
READ = re.compile(r"\b(stock|inventory|qty|quantity|on hand)\b", re.I)

def route(prompt: str) -> Decision:
    sku_match = SKU.search(prompt or "")
    sku = sku_match.group(1).upper() if sku_match else None
    wants_ticket = bool(TICKET.search(prompt or ""))
    wants_read = bool(READ.search(prompt or ""))
    if wants_ticket and wants_read:
        return Decision("refuse", {}, "mixed_intent")
    if wants_ticket:
        return Decision("open_ticket", {"sku": sku, "dry_run": True}, "ticket_verb")
    if wants_read and sku:
        return Decision("read_inventory", {"sku": sku}, "inventory_verb")
    return Decision("refuse", {}, "no_closed_tool")

Tests should lock three properties before anyone wires HTTP. First, mixed intent refuses instead of guessing. Second, ticket decisions keep dry_run true. Third, unknown SKUs still refuse rather than inventing a catalog row. Run the suite with a single command students can paste:

python -m pytest tests/test_router.py -q

If the router accuracy on the ten fixtures is already 9/10 or 10/10, say that out loud. That number is the baseline the chooser must beat, not a vibe.

Exercise 3 — Add an LLM chooser with a turn ledger

The chooser sends the prompt plus the tool catalog and expects a JSON object with tool, args, and reason. Reject any response that is not valid JSON, names an unknown tool, or omits dry_run on writes. Count every HTTP attempt as a turn, including parse failures and instructor-approved retries.

# workshop sketch — unexecuted against your lab URL until env is set
import json
import os
import time
import urllib.request

LAB_URL = os.environ.get("LAB_INFERENCE_URL", "")

SYSTEM = """Return JSON only with keys tool, args, reason.
Tools: read_inventory, open_ticket, refuse.
Writes must include dry_run true. Unknown work must refuse.
"""

def estimate_tokens(text: str) -> int:
    # labeled heuristic: four characters per token, not a vendor tokenizer
    return max(1, len(text) // 4)

def choose(prompt: str, timeout_s: float = 20.0) -> tuple[dict, dict]:
    if not LAB_URL:
        raise RuntimeError("LAB_INFERENCE_URL is unset; skip chooser or set the lab endpoint")
    payload = json.dumps({
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": prompt},
        ]
    }).encode()
    req = urllib.request.Request(
        LAB_URL, data=payload, headers={"Content-Type": "application/json"}
    )
    started = time.monotonic()
    with urllib.request.urlopen(req, timeout=timeout_s) as resp:
        body = resp.read().decode()
    elapsed_ms = int((time.monotonic() - started) * 1000)
    parsed = json.loads(body)
    content = parsed["choices"][0]["message"]["content"]
    decision = json.loads(content)
    ledger = {
        "turns": 1,
        "elapsed_ms": elapsed_ms,
        "est_prompt_tokens": estimate_tokens(SYSTEM + prompt),
        "est_completion_tokens": estimate_tokens(content),
        "parse_ok": True,
    }
    return decision, ledger

Append one JSONL row per fixture to traces/chooser.jsonl. Required columns are id, tool, parse_ok, turns, elapsed_ms, and estimated tokens. Students who cannot reach the lab URL should still complete the router path and mark chooser rows as skipped, not as zeros. Zeros quietly win fake cost contests.

A cheap validator belongs in the same exercise, not as homework:

WRITE_TOOLS = {"open_ticket"}
KNOWN = {"read_inventory", "open_ticket", "refuse"}

def validate_decision(decision: dict) -> list[str]:
    errors = []
    tool = decision.get("tool")
    args = decision.get("args") if isinstance(decision.get("args"), dict) else None
    if tool not in KNOWN:
        errors.append("unknown_tool")
    if args is None:
        errors.append("args_not_object")
    if tool in WRITE_TOOLS and not (args or {}).get("dry_run") is True:
        errors.append("write_without_dry_run")
    return errors

Exercise 4 — Dual-run and score

Score both systems on the same ten ids. Do not grade prose quality. Grade tool identity first, then argument equality, then ledger totals.

id | gold_tool | router_tool | chooser_tool | router_ok | chooser_ok | chooser_turns | est_tokens

Publish three classroom numbers on the whiteboard before discussion starts:

• Router exact-match rate on gold tools.
• Chooser exact-match rate after schema validation, excluding skipped rows.
• Chooser extra turns versus the router, which is always one local call and zero model tokens.

If the chooser ties the router but spends several thousand estimated tokens, the class should keep the router for this catalog. If the chooser recovers paraphrases the regex missed, keep the chooser only for those intents and route the rest. Mixing both without a recorded rule recreates the trench-coat agent the fixtures were meant to expose.

Decision table students fill in class

Observation on the ten fixtures	Keep the router	Promote the chooser	Do neither
Exact-match within one of gold, paraphrases still fail	Yes	No	—
Chooser recovers two or more paraphrases router missed	No	Yes, for those intents only	—
Chooser invents tools or drops dry_run	Yes	No	Add schema tests first
Lab endpoint timeouts exceed 20 percent of rows	Yes	No	Shrink prompts, forbid retries
Mixed-intent prompts trigger a write	—	—	Stop; the write gate is broken
Gold set itself disagrees after a fixture edit	—	—	Freeze ids and rescore

The table is the artifact that leaves the room. Code without the table usually collapses back into a single chat loop by the next lab.

Limitations and who should skip this

This plan assumes a closed tool list, synthetic inventory, and write tools that honor dry-run. It does not measure multi-hop planning, retrieval quality, or production safety. Character-chunk token estimates will disagree with vendor tokenizers; treat them as ordering, not billing. HTTP envelopes differ across labs, so instructors must adapt the chooser parser rather than copy the sketch as a client library.

Skip this workshop when students would hit live customer systems, when the network cannot inspect responses, or when the catalog is unbounded. Also skip it when the teaching goal is prompt style rather than tool identity. A free shared endpoint is the wrong backplane for secrets, for graded exams that need identical latency, and for overnight batch jobs that assume capacity will still be there tomorrow.

What to keep after class

Commit the fixture pack, the router tests, the chooser validator, and the filled decision table. Delete any trace file that captured raw prompts mixed with student names. If the chooser never beat the router on exact match, leave the model call out of the default demo path and document that result as a passing outcome. The useful habit is not “add an agent.” The useful habit is scoring both control flows before the next tool lands.

Instructors who want the chooser step on a no-invoice lab host can point LAB_INFERENCE_URL at the free server option already described, then rerun the same ten ids without changing fixtures. That is the only product substitution this outline needs; the scoring rules stay local.