Swap the Model, Keep the Grade: A Bootcamp Lab on Agent Provider Portability

Your agent doesn't work. It works on one endpoint.

That's the thesis, and a single-provider demo can never disprove it. The last week of DEV discourse keeps circling whether agents are real engineering or a trench coat full of if statements. I'm not going to relitigate that here. I'd rather hand out the lab that settles it for one team at a time: freeze the task, swap the endpoint, keep the grade.

The 60-second version

• Students build two adapters for one frozen prompt.
• The harness compares canonical traces, not raw JSON, not vibes.
• A divergence is not a failure — it's the finding. Grading rewards the report, not the agreement.
• The whole thing runs offline with stubs. Swap in a real endpoint only at checkpoint C3.

The contract students sign before writing code

Three artifacts, one command. No exceptions, no "it worked in the notebook".

1. prompt.txt — the user prompt, byte-for-byte, plus its SHA-256. Change one character and the run is invalid.
2. adapters/ — one adapter per endpoint. Each adapter's only job is normalizing its provider's wire format into the same canonical event type.
3. trace.json — canonical trace plus fingerprint, per run, per endpoint.

Then a single command: python provider_probe.py. It prints PASS or the index of the first step where the two endpoints disagree. Everything else in this lab exists to make that line meaningful.

The artifact: a harness that can actually fail

Here is the runnable core. It's deterministic, has no network calls, and no API keys. Two stubs share a plan but speak different wire shapes; a third deliberately diverges so you can see the harness catch it.

#!/usr/bin/env python3
"""provider_probe.py - canonical trace diff between two agent endpoints.

Runs offline. Every adapter here is a local stub; swap in a real HTTP
adapter later without touching the harness.
"""
from __future__ import annotations
import hashlib, json
from dataclasses import dataclass


@dataclass(frozen=True)
class Event:
    kind: str          # "tool" | "final"
    tool: str = ""
    args: tuple = ()   # sorted (key, json.dumps(value)) pairs
    text: str = ""

    def digest(self) -> str:
        return hashlib.sha256(repr(self).encode()).hexdigest()[:12]


def canon_tool(name: str, args: dict) -> Event:
    packed = tuple(sorted((k, json.dumps(v, sort_keys=True)) for k, v in args.items()))
    return Event("tool", name, packed)


def canon_final(text: str) -> Event:
    return Event("final", text=" ".join(text.split()))  # collapse whitespace


class Endpoint:
    name = "abstract"
    def plan(self, prompt: str) -> list[Event]:
        raise NotImplementedError


class EchoNotesA(Endpoint):
    """Friendly endpoint: tool args arrive as a dict."""
    name = "echo-notes-a"
    def plan(self, prompt):
        return [canon_tool("read_file", {"path": "notes.md"}),
                canon_tool("write_file", {"path": "summary.md", "body": "3 bullets"}),
                canon_final("Summary written to summary.md")]


class EchoNotesB(Endpoint):
    """Same plan, different wire shape: args arrive as a JSON string."""
    name = "echo-notes-b"
    def plan(self, prompt):
        raw = [("read_file", '{"path": "notes.md"}'),
               ("write_file", '{"body": "3 bullets", "path": "summary.md"}')]
        events = [canon_tool(n, json.loads(blob)) for n, blob in raw]
        events.append(canon_final("Summary written to summary.md\n"))
        return events


class ChattyC(Endpoint):
    """Divergent endpoint: renames a tool, adds a confirmation turn."""
    name = "chatty-c"
    def plan(self, prompt):
        return [canon_tool("read", {"path": "notes.md"}),
                canon_tool("write_file", {"path": "summary.md", "body": "3 bullets"}),
                canon_final("Summary written to summary.md")]


def first_divergence(a, b):
    for i, (x, y) in enumerate(zip(a, b)):
        if x != y:
            return i, x, y
    if len(a) != len(b):
        return min(len(a), len(b)), None, None
    return None


def fingerprint(trace) -> str:
    return hashlib.sha256("|".join(e.digest() for e in trace).encode()).hexdigest()[:12]


def report(a: Endpoint, b: Endpoint, prompt: str) -> bool:
    ta, tb = a.plan(prompt), b.plan(prompt)
    print(f"{a.name:14s} {fingerprint(ta)}   {b.name:14s} {fingerprint(tb)}")
    d = first_divergence(ta, tb)
    if d is None:
        print("  PASS - identical canonical trace")
        return True
    i, x, y = d
    print(f"  FAIL - first divergence at step {i}")
    print(f"    {a.name}: {x}")
    print(f"    {b.name}: {y}")
    return False


if __name__ == "__main__":
    prompt = "summarize notes.md into summary.md"
    report(EchoNotesA(), EchoNotesB(), prompt)
    report(EchoNotesA(), ChattyC(), prompt)

Expected output:

echo-notes-a   a41f...   echo-notes-b   a41f...
  PASS - identical canonical trace
echo-notes-a   a41f...   chatty-c       7c02...
  FAIL - first divergence at step 0
    echo-notes-a: Event(kind='tool', tool='read_file', args=(('path', '"notes.md"'),), text='')
    chatty-c:      Event(kind='tool', tool='read', args=(('path', '"notes.md"'),), text='')

Notice what the stubs buy you: the two "b"-style differences are not divergences. Dict-vs-JSON args is an adapter bug, and the adapter fixes it. read vs read_file is a design question, and the adapter must not hide it.

Checkpoints

C0 - Contract written. Prompt hash, allowed tools, expected artifacts, on paper, before code. Half the cohort will skip this. Those are the runs you can't grade.

C1 - Single-endpoint baseline. Five runs on the first endpoint. All five fingerprints identical? If not, you have nondeterminism, and you need to say so in the write-up instead of hiding it behind one lucky run.

C2 - Canonical form. trace.json holds canonical events only. Raw provider payloads go in raw/ as evidence, never in the trace. A trace you can't diff is a log, not a contract.

C3 - The swap. Same frozen prompt, second adapter, second endpoint. Fingerprints compared. This is the checkpoint the lab exists for.

C4 - Divergence triage. Classify the first divergence: adapter bug, capability gap, plan difference, or prose difference. Write the classification down. Do not re-run until it passes.

Where a free second endpoint changes the lab

A portability lab needs two endpoints. That's the whole cost problem: students can afford exactly one.

This is the one place where the thing I'm writing about actually participates in the method rather than decorating it. MonkeyCode advertises free model access and a free server option, which is enough to provision the second adapter in C3 without a second invoice, and enough to run the swap from a machine the student doesn't own. I was told there is also a promotional token allowance attached; I'm deliberately not printing a number here, because allowances move and a stale figure in a syllabus is worse than no figure. Check the project's current terms before you commit a course to it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caveat that belongs in the syllabus, not the footnotes: a hosted endpoint is still someone else's endpoint. Free access does not mean the provider's tool-calling semantics are yours to assume. Your adapter is the place where that assumption gets tested, or it becomes the bug.

Capability probe: fill this in by hand

Before C4, run the same three prompts against both endpoints and record what you observe. No blanks. Each cell needs a command that reproduces it.

Probe	Endpoint A	Endpoint B	Reproduce with
Parallel tool calls accepted			one prompt, two tool calls
Args as object vs JSON string			dump the raw payload
Whitespace-stable final text			hash 5 runs
Refuses a determinism setting			set it, read the error
Extra chat turn before a write			count trace steps

The decision table students actually use

Observed on the second endpoint	Verdict	Action
Same semantics, different tool name	adapter bug	map names in the adapter, re-run
Args arrive as a JSON string	adapter bug	normalize in the adapter
Extra confirmation turn before a write	design question	if your contract allows it, encode it in the canonical form; if not, fail the run
Final prose differs, facts identical	pass	don't grade on wording
Tool order differs, both valid	design gap	you had no plan contract — fix that first
Endpoint won't accept your determinism setting	recorded limitation	document it; never silently drop it

Grading rubric (100 points)

• 20 - Contract. Prompt hash, tool allowlist, and expected artifacts exist and predate the first run.
• 25 - Baseline. Five fingerprints on endpoint A, plus a clear statement of whether they matched.
• 30 - Swap. Second adapter, canonical comparison, first-divergence report with a classification.
• 15 - Capability table. No blank cells, every cell reproducible by a pasted command.
• 10 - Retro. One paragraph on what single-endpoint testing hid.

Automatic deductions: raw provider logs pasted as the trace; prompt edited between runs; "works on my machine" with no fingerprints; re-running until green without documenting the earlier red.

Stretch goals

1. Mutation test the harness. On purpose, rename a tool inside an adapter and confirm the harness reports a divergence. A detector you've never seen fire is a detector you don't have.
2. Third endpoint, smaller model. Record where it fails first. Capability floors are more interesting than pass/fail.
3. Prompt-hash gate. Make the harness refuse to run if prompt.txt changed since the baseline. Freeze or it isn't a comparison.

A real HTTP adapter looks roughly like this — adapt it to whatever schema your endpoint documents, and treat it as untested until you run it:

class HttpEndpoint(Endpoint):
    def __init__(self, base_url: str, model: str, token: str):
        self.name, self.base_url = model, base_url
        self.headers = {"Authorization": f"Bearer {token}"}

    def plan(self, prompt: str) -> list[Event]:
        # NOTE: illustrative only. Your endpoint's response schema will differ.
        r = requests.post(f"{self.base_url}/chat", headers=self.headers,
                          json={"model": self.name, "messages": [{"role": "user", "content": prompt}]},
                          timeout=60)
        r.raise_for_status()
        return [canon_tool(c["name"], c["arguments"]) for c in r.json().get("tool_calls", [])] \
             + [canon_final(r.json().get("content", ""))]

Honest limitations

• Two endpoints agreeing is not correctness. It's two guesses matching, and correlated training data makes that cheap.
• This lab measures traces, not latency, cost, or throughput. Don't grade what you didn't measure.
• Stubs can't catch prompt sensitivity. That's what C3 is for.
• Free tiers and their allowances change. Pin the terms you relied on, with a date, in your own write-up.
• Don't run automated probes against an endpoint whose terms forbid it. Read them.

Who should skip this

If your contract locks you to one vendor and you can't add a second endpoint, the swap checkpoint is unrunnable and the rubric collapses to a writing exercise. If you want a speed benchmark, this is the wrong lab — the harness compares structure, not duration. And if you'd treat a green PASS as proof the agent is correct, don't ship it; you'll pass the lab and fail production.

The lab's payoff isn't the second model. It's finding out which parts of your agent were never yours to begin with. When you swap, the vendor-specific assumptions fall out on their own — loudly, at step 0.

If you want to run the swap without a second bill, the free model access is the door I'd use first; bring your own prompt hash and your own rubric.