On a midweek review call, a developer pastes a chat transcript in which an assistant invoked create_invoice and received HTTP 200. The room treats that paste as proof that billing integration is finished, even though nobody else can replay the exchange. Replay still needs the author's laptop, the author's billed key, and an uncommitted prompt file sitting on a desktop. The failure is not the model; the failure is the missing contract between the tool call and the runtime.
This confusion has grown louder as tool-calling explainers spread through developer feeds this week. A streamed function name looks like an API client, and a pretty JSON argument looks like a validated payload. Those appearances are cheap to produce and expensive to trust, because they hide retries, auth scopes, and durable side effects. The rest of this FAQ treats each repeated claim as a testable statement, then replaces it with a mental model you can actually run.
Myth: a green chat tool call means the integration shipped
A chat tool call is a demonstration that a model emitted a name and a blob of JSON under one prompt. It does not prove that the backend accepted the same blob twice, rejected a stale idempotency key, or refused a missing scope. It also does not prove that a teammate can obtain the same trace without sitting at the original keyboard. Treat the chat as a sketch of intent, not as a shipping document.
The corrected model is simple enough to say in one sentence and hard enough to operationalize in a repo. A tool call is a candidate transaction; a contract is a replayable exchange against a named runtime with a schema, an identity, and an expected status. Until those four pieces exist in files rather than in memory, the green bubble in the chat window is theater. Theater is useful for design reviews, and it is a poor substitute for a merge gate.
Myth: a personal API key is a shared evaluation environment
Personal keys encode one person's billing account, residual rate limit, and leftover context from yesterday's abandoned experiments. They are convenient, and they are not a laboratory. When the only passing run lives behind a secret that cannot leave a laptop, the team has a souvenir rather than a measurement. Souvenirs do not survive vacation, laptop replacement, or a rotated credential.
A shared runtime does not need to be expensive infrastructure or a theatrical staging cluster. It needs a hostname, a recorded clock, and credentials that a second person can use without copying a .env file out of chat. Disclosure: This article was prepared as part of MonkeyCode's product outreach. When a coding assistant such as MonkeyCode offers free model access and a free server option, the useful part is not the adjective free; it is that a second engineer can point the same contract file at a runtime that is not your MacBook. If the contract fails there, the original chat demo was never evidence.
Myth: free model access means you can skip pinning and recording
Free access removes a purchase order; it does not freeze the model, the tokenizer, or the tool-calling parser sitting in front of your mock. Without a recorded model identifier, a recorded schema hash, and a recorded request body, you cannot explain a later red run. People then argue about prompt tone when they should be comparing two traces line by line. That argument wastes the allowance they were trying to protect, and it teaches the team the wrong lesson about variance.
Record the minimum, not a novel and not a dashboard graveyard. Store the tool name, the canonical JSON, the HTTP status, and an idempotency key in a file the repository can diff. Store the runtime name as a plain string, even if that string is merely free-server-eval on the first pass. Tomorrow's incident review will thank you for those four lines more than for a screenshot of a chat bubble.
Myth: if the JSON looks right, the side effects were right
JSON can be well-formed and still charge the wrong customer, skip tax, or create a second invoice after a retry. Looking right is a syntax property that a linter can see. Side effects are a history property, and history lives on the server, not in the model's self-report. If your mock never writes a ledger row, you have tested a parser, not a billing path, and parsers do not refund money.
The useful analogy is a signed check versus a cleared check at the bank window. The signature can be beautiful while the account is empty, frozen, or already debited by a duplicate presentment. Your contract should assert both the signature shape and the ledger mutation, even when the ledger is a SQLite file created in CI. Skip the mutation assertion and you will ship a polite client that double-posts under packet loss.
Myth: localhost and a free shared server are interchangeable
Localhost has your hosts file, your Docker layer cache, your VPN, and your leftover process on port 8080 from last Thursday. A shared server has none of those gifts, and that absence is the point of using it. Failures that appear only after you leave the laptop are not flaky models; they are undeclared dependencies wearing a friendly stack trace. Declaring them in a contract file is cheaper than debugging them during a customer incident.
Run the same contract in both places and keep both traces beside the schema. If they diverge, the delta is your real environment document, written by failure instead of by optimism. If they match, you have earned the right to talk about the model instead of the laptop. Until then, model debate is a distraction from missing files.
A reproducible contract, labeled as an unexecuted example
The following files are a proposed harness, not a claim that any production system was measured with them. They turn create_invoice into a replayable exchange: a tiny ledger, a schema check, and a trace that a second machine can emit. Save them in a throwaway directory and run the commands only against a mock you control.
# contract_server.py — proposal: local mock, not a vendor runtime
import json, sqlite3, hashlib, os
from http.server import BaseHTTPRequestHandler, HTTPServer
DB = os.environ.get("LEDGER_DB", "ledger.sqlite")
SCHEMA_HASH = "invoice_v1"
def init():
con = sqlite3.connect(DB)
con.execute("""CREATE TABLE IF NOT EXISTS invoices(
idempotency_key TEXT PRIMARY KEY,
customer_id TEXT NOT NULL,
cents INTEGER NOT NULL,
schema_hash TEXT NOT NULL
)""")
con.commit()
return con
class Handler(BaseHTTPRequestHandler):
def do_POST(self):
if self.path != "/tools/create_invoice":
self.send_error(404); return
length = int(self.headers.get("Content-Length", "0"))
body = json.loads(self.rfile.read(length) or b"{}")
key = self.headers.get("Idempotency-Key") or body.get("idempotency_key")
customer = body.get("customer_id")
cents = body.get("cents")
if not key or not customer or not isinstance(cents, int) or cents <= 0:
self._json(400, {"error": "invalid_payload"}); return
con = init()
try:
con.execute(
"INSERT INTO invoices VALUES (?, ?, ?, ?)",
(key, customer, cents, SCHEMA_HASH),
)
con.commit()
created = True
except sqlite3.IntegrityError:
row = con.execute(
"SELECT customer_id, cents FROM invoices WHERE idempotency_key=?",
(key,),
).fetchone()
if row != (customer, cents):
self._json(409, {"error": "idempotency_conflict"}); return
created = False
digest = hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()[:12]
self._json(200 if created else 200, {
"ok": True,
"created": created,
"schema_hash": SCHEMA_HASH,
"body_digest": digest,
"runtime": os.environ.get("RUNTIME_NAME", "localhost"),
})
def _json(self, code, payload):
raw = json.dumps(payload).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(raw)))
self.end_headers()
self.wfile.write(raw)
if __name__ == "__main__":
init()
HTTPServer(("0.0.0.0", 8088), Handler).serve_forever()
# test_invoice_contract.py — proposal: asserts shape and ledger history
import json, os, sqlite3, urllib.request, uuid
BASE = os.environ.get("CONTRACT_BASE", "http://127.0.0.1:8088")
DB = os.environ.get("LEDGER_DB", "ledger.sqlite")
def post(payload, key):
req = urllib.request.Request(
BASE + "/tools/create_invoice",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json", "Idempotency-Key": key},
method="POST",
)
try:
with urllib.request.urlopen(req) as resp:
return resp.status, json.loads(resp.read().decode())
except urllib.error.HTTPError as exc:
return exc.code, json.loads(exc.read().decode())
def test_replay_does_not_double_write():
key = "idem-" + uuid.uuid4().hex
body = {"customer_id": "cus_demo", "cents": 1999, "idempotency_key": key}
status1, doc1 = post(body, key)
status2, doc2 = post(body, key)
assert status1 == 200 and status2 == 200
assert doc1["created"] is True and doc2["created"] is False
assert doc1["schema_hash"] == doc2["schema_hash"] == "invoice_v1"
con = sqlite3.connect(DB)
count = con.execute("SELECT COUNT(*) FROM invoices WHERE idempotency_key=?", (key,)).fetchone()[0]
assert count == 1
def test_conflict_on_key_reuse_with_new_amount():
key = "idem-" + uuid.uuid4().hex
post({"customer_id": "cus_demo", "cents": 500, "idempotency_key": key}, key)
status, doc = post({"customer_id": "cus_demo", "cents": 700, "idempotency_key": key}, key)
assert status == 409 and doc["error"] == "idempotency_conflict"
# unlabeled until you actually run it; this is the intended sequence
python3 contract_server.py &
export RUNTIME_NAME=localhost CONTRACT_BASE=http://127.0.0.1:8088 LEDGER_DB=ledger.sqlite
python3 -m pytest -q test_invoice_contract.py
# after copying the same two files onto a shared host:
# RUNTIME_NAME=shared-eval CONTRACT_BASE=http://SHARED_HOST:8088 LEDGER_DB=/tmp/ledger.sqlite
# python3 -m pytest -q test_invoice_contract.py | tee trace-shared.txt
The interesting output is not a speed number and not a model ranking. It is whether created flips from true to false on replay, whether a mutated amount returns 409, and whether RUNTIME_NAME in the JSON changes when the host changes. Those three facts tell you if you measured a contract or only a conversation. Keep the traces next to the schema hash so a later red run has a sibling to diff against.
A compact decision table belongs beside the tests, because teams otherwise invent a new story every time a chat window looks healthy. Read each row as a gate, not as a slogan.
| Observation | What it actually measured | Safe next step |
|---|---|---|
| Chat shows a tool name and 200-looking JSON | Prompt emission under one author's context | Write a schema and a mock; do not merge |
| Contract passes on localhost only | Laptop path, including hidden daemons | Replay on a named shared runtime |
| Contract passes on a free shared server with the same files | Replayable exchange, still not production traffic | Pin the schema hash and keep both traces |
| Replay returns 409 on amount change | Idempotency is real, not decorative | Add auth-scope cases before any live money |
| Only a personal billed key can produce the trace | Souvenir environment | Stop quoting the result in planning |
Free model access is relevant here as a way to draft the mock and the tests without turning the first hour into a procurement thread. The free server option is relevant as a second hostname that is not the author's laptop, so the middle row of that table can be filled in by someone else. Neither option replaces a production identity provider, a change window, or an audit log that your finance team would recognize.
Limitations, and who should not use this approach
This harness does not measure latency distributions, token prices, or model quality, and it should not be cited as if it did. It does not implement authentication, and the SQLite ledger is a teaching device rather than a store of record. If your tool can touch customer money, medical data, or production secrets, do not point this mock at those systems and do not paste live payloads into a shared eval host.
Skip the method if you need a vendor SLA, a dedicated region, or a formal penetration test. Skip it if the only goal is a demo gif. Skip it if your organization forbids putting even synthetic billing shapes on a shared machine you do not administer. In those cases the corrected mental model still applies, but the runtime must be one your security review already named.
The durable lesson is narrower than the week's tool-calling explainers usually admit. A model that can say a function name has not cleared a transaction, any more than a signed check has cleared a bank. Put the schema, the idempotency key, and the runtime name in files two people can run. If you want that second hostname to exist without requisitioning a laptop, the free server path is one way to fill the gap; the contract file remains the artifact that matters.
Top comments (0)