Last Thursday I stood in the Killam Library lobby with a laptop that thought it had booked me a room. The confirmation looked confident. Room 404, sixty minutes, my name spelled right. I walked to the fourth floor anyway. There is no Room 404.
That is the whole lesson in one bad receipt. Agents do not only hallucinate prose. They hallucinate arguments. They fill in a room id, a duration, a building code, and they do it with the same calm tone they use when the catalog actually contains the row. Can an eighty-line contract layer catch those assumed fields before they touch the world?
I wanted a case study I could rerun on a bus, not a framework tour. So I froze a three-room campus catalog, wrote a naive planner that impersonates a model, and put a validator in front of book_room. The planner is deliberately dumb. That is the point. If the contract only works when the model is brilliant, it is not a contract.
Background
I am an AI student in Halifax, and I keep trying to glue tiny agents onto student chores: quiet rooms, office hours, printer queues. The first version of this booking bot had two tools, list_rooms and book_room, and I let the model pick arguments from a fuzzy sentence like "book something quiet this afternoon." It booked CS-130 for three hours. CS-130 is real. Three-hour slots are not. The tool still returned ok: true because I had treated the model as a coworker who would never invent a field.
A tool call is not a suggestion. It is a function invocation with side effects. If you would not let a classmate pass extra keyword arguments into your lab code, why let a model do it?
The learning question is narrow. Given a frozen catalog and a frozen set of plans, can a schema-plus-catalog check refuse invented ids and missing durations before book_room runs?
Goal
Build one file, room_contract.py, that prints four fixtures. One should succeed. Three should fail in different ways: an invented room, a missing duration, and a duration the desk would never allow. No pip packages. Python 3.11 or newer. I ran it on 3.11.9.
If you want to predict the result before you scroll, do it now. Which fixture dies first, and with which code? assumed_room_id, assumed_fields, or assumed_duration?
Implementation
Save this as room_contract.py. It is the whole lab.
#!/usr/bin/env python3
"""Tiny study-room agent with a tool-contract layer. stdlib only."""
from __future__ import annotations
import json
from dataclasses import dataclass
from typing import Any, Callable
ROOMS = {
"KILLAM-212": {"seats": 4, "quiet": True, "building": "Killam Library"},
"CS-130": {"seats": 8, "quiet": False, "building": "Goldberg CS"},
"MCD-17": {"seats": 2, "quiet": True, "building": "McDonald Building"},
}
ALLOWED_DURATIONS = {30, 60, 90, 120}
@dataclass
class ContractError(Exception):
code: str
detail: str
def as_dict(self) -> dict[str, Any]:
return {"ok": False, "code": self.code, "detail": self.detail}
def list_rooms(quiet_only: bool = False) -> list[str]:
return [rid for rid, meta in ROOMS.items() if (meta["quiet"] if quiet_only else True)]
def book_room(room_id: str, minutes: int, student: str) -> dict[str, Any]:
if room_id not in ROOMS:
raise ContractError("unknown_room", f"{room_id} is not in the catalog")
if minutes not in ALLOWED_DURATIONS:
raise ContractError("bad_duration", f"{minutes} is not a bookable slot")
if not student.strip():
raise ContractError("empty_student", "student name is required")
return {
"ok": True,
"room_id": room_id,
"minutes": minutes,
"student": student,
"building": ROOMS[room_id]["building"],
}
TOOLS: dict[str, Callable[..., Any]] = {
"list_rooms": list_rooms,
"book_room": book_room,
}
SCHEMAS = {
"list_rooms": {"quiet_only": bool},
"book_room": {"room_id": str, "minutes": int, "student": str},
}
def validate_args(tool: str, args: dict[str, Any]) -> dict[str, Any]:
if tool not in SCHEMAS:
raise ContractError("unknown_tool", tool)
schema = SCHEMAS[tool]
extra = set(args) - set(schema)
missing = set(schema) - set(args)
if extra:
raise ContractError("extra_fields", str(sorted(extra)))
if missing:
raise ContractError("assumed_fields", f"refusing to fill {sorted(missing)}")
cleaned: dict[str, Any] = {}
for key, typ in schema.items():
val = args[key]
if not isinstance(val, typ):
raise ContractError("type_mismatch", f"{key} should be {typ.__name__}")
cleaned[key] = val
if tool == "book_room":
if cleaned["room_id"] not in ROOMS:
raise ContractError("assumed_room_id", cleaned["room_id"])
if cleaned["minutes"] not in ALLOWED_DURATIONS:
raise ContractError("assumed_duration", str(cleaned["minutes"]))
return cleaned
def dispatch(tool: str, args: dict[str, Any]) -> dict[str, Any]:
try:
cleaned = validate_args(tool, args)
result = TOOLS[tool](**cleaned)
return {"ok": True, "tool": tool, "result": result}
except ContractError as err:
return err.as_dict()
NAIVE_PLANS = {
"happy": ("book_room", {"room_id": "KILLAM-212", "minutes": 60, "student": "alex"}),
"invented_room": ("book_room", {"room_id": "KILLAM-404", "minutes": 60, "student": "alex"}),
"assumed_duration": ("book_room", {"room_id": "CS-130", "student": "alex"}),
"overlong": ("book_room", {"room_id": "MCD-17", "minutes": 180, "student": "alex"}),
}
def naive_agent(user_text: str, fixture: str) -> dict[str, Any]:
tool, args = NAIVE_PLANS[fixture]
print(f"# user: {user_text}")
print(f"# planned tool: {tool} {json.dumps(args)}")
return dispatch(tool, args)
if __name__ == "__main__":
tests = [
("Book a quiet room for an hour", "happy"),
("Book Killam 404, I think that exists", "invented_room"),
("Book CS-130 this afternoon", "assumed_duration"),
("Book McDonald 17 for three hours", "overlong"),
]
for text, name in tests:
print("=" * 60)
print(f"FIXTURE {name}")
print(json.dumps(naive_agent(text, name), indent=2))
The catalog is the ground truth. The schemas are the visa. validate_args does not "fix" a missing minutes by guessing 60. Guessing is how I ended up on the fourth floor. It refuses, and it names the refusal.
Notice the double door on book_room. The schema says minutes must be an int. The catalog says it must also be in {30, 60, 90, 120}. Type checks are not policy checks. An agent can pass a perfectly typed 180 and still be wrong.
Results
Run it from the same directory.
python room_contract.py
The happy path should look like this.
============================================================
FIXTURE happy
# user: Book a quiet room for an hour
# planned tool: book_room {"room_id": "KILLAM-212", "minutes": 60, "student": "alex"}
{
"ok": true,
"tool": "book_room",
"result": {
"ok": true,
"room_id": "KILLAM-212",
"minutes": 60,
"student": "alex",
"building": "Killam Library"
}
}
The invented room is the error input I actually walked into. Predict the code before you read it?
FIXTURE invented_room
# user: Book Killam 404, I think that exists
# planned tool: book_room {"room_id": "KILLAM-404", "minutes": 60, "student": "alex"}
{
"ok": false,
"code": "assumed_room_id",
"detail": "KILLAM-404"
}
assumed_duration never reaches book_room. The missing key dies at the schema door with assumed_fields. overlong has every key present and well typed, then dies on the catalog door with assumed_duration and detail 180. That split matters. One failure is "the model forgot a field." The other is "the model filled a field the desk does not sell."
If your output disagrees, check two things. Did you let book_room default minutes=60? Then you rebuilt the bug. Did you stringify 180? Then type_mismatch fires first and you never see the policy error. I did both, in that order, before the script above.
What broke, and why it felt obvious after
The naive planner is a stand-in for a live model. I did that on purpose. A live model adds temperature, retries, and a fog of almost-correct ids like KILLAM-241 versus KILLAM-212. The contract does not care about the fog. It only cares whether the arguments are closed under the schema and the catalog.
Think of the catalog as a tiny customs desk. The model can write a beautiful passport. If the room number is not in the book, the stamp is still no. I used to treat "the tool raised an exception" as the contract. That is too late. book_room in a real system might already have written a row, sent an email, or held a lock. Validation has to sit in front of the side effect, not inside the apology.
A common mistake is to "be helpful" in the validator: missing duration becomes 60, unknown room becomes the first quiet room, empty student becomes user. That is not a contract. That is a second, quieter model. Helpful defaults are how Room 404 gets a confirmation number.
Another mistake is validating only types. minutes: int will happily take 7, 180, or 100000. Policy lives in sets and ranges, not in isinstance.
Where a free model loop actually helped
Once the four fixtures were stable, I wanted to swap NAIVE_PLANS for a real model that reads the user sentence and emits a tool name plus JSON arguments. That swap is where iteration cost shows up. You will reject a lot of calls. You should. I did not want that loop to depend on a paid endpoint while I was still teaching myself the difference between a missing key and a bad key.
MonkeyCode is an open-source project that currently offers free model access and a free server option, which is enough to point the same dispatch() function at live tool JSON without standing up my own box. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a quota, a model name, or a benchmark here, because those numbers move and I did not measure them for this lab. The script above does not need the product at all. If you do wire a live model, keep the contract on your side of the HTTP call.
If you run the failing fixtures against a live model, I want the counterexample more than the screenshot. Which extra assumption did it invent that my four codes do not name yet?
Limitations, and who should not use this
This is a campus toy. It does not talk to a real booking API. It does not handle concurrent holds, auth, or closing hours. The naive planner cannot surprise you, which is great for learning and useless as a load test. A live model will emit extra keys like building or date. My schema treats extras as errors. Some production stacks strip extras instead. Stripping is a policy. Name it.
Do not use this pattern if the tool can charge a card, send mail to a class list, or lock a room other people need. Do not use it as proof that "agents are safe now." A contract on arguments does not check whether the user is allowed to book, whether Killam is open, or whether CS-130 is already full. Those are different doors.
If you need multi-tool plans, this file is the wrong shape. A planner that calls list_rooms then book_room can still assume the first id in the list is "the quiet one." That is a planning lie, not a schema lie. Catching it needs a second fixture: list, then book an id that was not in the list result. I left that as the extension on purpose.
What you should be able to explain after this
You should be able to draw three layers on a napkin: user text, planned tool JSON, world. The contract lives between the last two. You should be able to say why a missing field and a well-typed illegal field need different error codes. You should be able to point at a default argument in a tool function and call it what it is: an assumption with a shorter name.
Extension, if you want homework. Add a fifth fixture where the planner calls list_rooms with quiet_only: true, then books CS-130 anyway. CS-130 is real and loud. Should that fail as assumed_room_id, or do you need a new code, not_in_last_list? Write the smallest failing test first. Then decide whether your agent is allowed to remember rooms from a previous turn. Memory is just another way to assume.
Top comments (0)