Fail Loudly: A Tool-Schema Decision Tree for Agent Steps That Return Wrong Answers Without Failing

The bug report is always a variant of the same sentence: the agent ran, nothing threw, and the ticket got filed against the wrong customer.

There is no stack trace because nothing crashed. A field defaulted to null, the tool accepted it, and the model wrote a confident summary on top. Silent failures like this burn more debugging hours than exceptions do, because an exception tells you where to look.

What follows is a decision tree for that exact class of bug. Six glossary terms, five branches, one worked example per leaf, and a linter you can run against your own tool schemas in about a minute. The failure below is a constructed illustration, not a logged incident — the shape is what matters.

Glossary: the six terms this tree depends on

1. Coercion surface — the number of places where a wrong type is silently converted instead of rejected. '3' becoming 3 is a coercion. 'three' becoming 3 is a guess, and guesses are where silent bugs live.
2. Sentinel ambiguity — how many separate meanings null, '', 0, and [] carry inside one schema. Two meanings is a bug. Three is a support ticket.
3. Closed vocabulary — a field constrained to an enum or a regex versus left as a free string. Free strings are where model-invented status values appear.
4. Echo field — an input value the model must copy into the call verbatim: an ID, a hash, a request token. It is the cheapest correctness probe you will ever add, because a wrong echo means every other field is suspect too.
5. Error channel — how a tool reports failure back to the model. A tool that returns {'ok': False, 'reason': 'customer_id not found in input'} gives the model something to repair against. A tool that raises into the void gives it nothing.
6. Required-field density — required fields divided by total fields, per tool. Low density means the model is allowed to leave things out, which it will.

Step 1: Lint the schema before you blame the model

The tree's branch inputs come from a static scan, not from intuition. This script is stdlib-only and runs on Python 3.9+.

# schema_lint.py — no dependencies
import json
import sys


def walk(node, path='$'):
    yield path, node
    for name, sub in (node.get('properties') or {}).items():
        yield from walk(sub, f'{path}.{name}')
    items = node.get('items')
    if isinstance(items, dict):
        yield from walk(items, f'{path}[]')


def lint(schema):
    findings = []
    props = schema.get('properties') or {}
    required = set(schema.get('required') or [])
    total = len(props) or 1
    density = len(required) / total

    if total >= 3 and density < 0.75:
        optional = sorted(set(props) - required)
        findings.append(['R1', f'density {density:.0%} < 75%; omittable: {optional}'])

    for path, node in walk(schema):
        kind = node.get('type')
        if kind == 'string' and 'enum' not in node and 'pattern' not in node:
            findings.append(['R2', f'{path}: open string, no enum/pattern'])
        if kind == 'boolean':
            findings.append(['R3', f'{path}: boolean flag, prefer explicit enum'])
        if node.get('default') is None or kind in ('null', ['string', 'null']):
            findings.append(['R4', f'{path}: null default, sentinel ambiguity'])

    return findings


if __name__ == '__main__':
    with open(sys.argv[1]) as fh:
        schema = json.load(fh)
    for code, msg in lint(schema):
        print(f'{code}  {msg}')

Run it as python schema_lint.py ticket_tools.json. A realistic first pass on a four-tool schema usually prints eight to fifteen lines. That is not a bad schema; that is a schema written for a human caller.

Step 2: Read findings as branch inputs

The codes are not a score. Each one routes you to a branch.

Finding	Question it triggers	Branch
R1 (low density)	Can this call succeed with fields missing?	3 or 4
R2 (open string)	Does the model invent values here?	3
R3 (boolean flag)	Does false mean "no" or "unset"?	2
R4 (null default)	How many meanings does null carry?	2
No findings, still wrong	Is the contract right but the run unstable?	5
Output does not parse at all	Is the grammar even enforced?	1

Step 3: Walk to a leaf

Branch 1 — output will not parse → freeze the grammar

Worked leaf: the model returns prose wrapped around JSON, your json.loads throws, you retry three times and pay three times.

Action: enforce a structured-output mode if your endpoint has one. If it does not, wrap the call in an extractor that takes the first balanced {...} and validates it. The leaf ends when parse failures drop below 1 in 50 calls — not to zero, because zero is usually a sign you stopped measuring.

Branch 2 — parses, but the echo is wrong → echo-first contract

Worked leaf: input carries request_id: r-8841, the call comes back with request_id: r-8814 or none. Everything downstream is now attached to a possibly wrong record.

Action: make the echo field required, add it to the required list, and reject the call server-side on mismatch. This converts a silent failure into a loud one for the cost of one string comparison. It is the highest-return change in the whole tree.

Branch 3 — echo right, enum drifted → closed vocabulary

Worked leaf: your schema says status: string, the model sends "resolved-ish" or "Pending Review". Your writer stores it and the dashboard silently buckets it as unknown.

Action: replace the free string with an enum of the values your code actually branches on, and add an error channel entry that returns the allowed set on rejection. Keep the enum short — six values, not sixteen.

Branch 4 — shape right, required values empty → two-phase fill

Worked leaf: customer_id is present and syntactically valid but the model filled it with a plausible-looking value it never saw in the input. R1 flagged this as low density.

Action: split the call. Phase one asks the model to select from candidates you supply; phase two fills the remaining fields with the selection fixed. Two cheap calls beat one confident hallucination.

Branch 5 — everything correct, unstable across runs → snapshot and pin

Worked leaf: the same input yields correct output four times out of five, and you cannot tell which run will be the fifth.

Action: record input, schema hash, and output for twenty runs. If variance is in field selection rather than values, you have a decoding or sampling problem, not a schema problem. Pin what you can, measure what you cannot.

Step 4: Measure the echo rate before and after

One number tells you whether the tree worked:

def echo_match_rate(samples, field='request_id'):
    hits = sum(
        1 for s in samples
        if s.get('args', {}).get(field) == s.get('input', {}).get(field)
    )
    return hits / len(samples)

Record it at 20 samples before the change and 20 after. If the rate does not move, you fixed the wrong branch. Report the number alongside the schema hash, because a schema edit without a version tag makes every later comparison meaningless.

Step 5: Know when to stop

Stop tightening when three conditions hold: echo rate is at or near 1.0, every field your code branches on has a closed vocabulary, and rejections are rare enough that a human reading the error channel would not call it noise. A schema with zero rejections is not safer — it just stopped defending.

Limitations, and who should not use this

This tree assumes you own the tool schema. If you are calling someone else's function-calling API and cannot change field definitions, branches 3 and 4 are unavailable and you are limited to prompt-side fixes plus an echo check.

It also assumes failures are cheap to replay. In flows where each call has an external side effect — sending mail, charging a card — you cannot just re-run 20 samples. Snapshot the calls, replay against a stub, and treat the tree as a pre-production gate only.

Skip this entirely if your agent has fewer than three tools and a human reads every output. The lint pass takes a minute, but the versioned measurement loop costs real time and earns nothing at that scale.

Where a free tier actually fits

The measurement loop in Step 4 is the expensive part, and it is boring work: twenty calls per schema revision, repeated until the echo rate stabilises. That is a batch job, not an interactive session. MonkeyCode's operator-supplied free model access and free server option cover both halves — the calls come from the free model endpoint, and the batch harness runs on the free server instead of your laptop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator also states a free allowance of roughly 10 million tokens. Treat any quota figure as perishable: open the product page, read the current limit, and write the number into your own notes with today's date before you plan a harness around it. I have not verified model names, throughput, hardware, or whether the allowance is permanent, and this post makes no claims about any of them.

One honest ordering note: run the linter first. A schema fix costs nothing and removes most of the failures you were about to spend tokens measuring. If you still want the batch pass after that, the free model access and free server option are the two pieces worth opening first.

What to check before you copy this

• Confirm your endpoint supports structured output before assuming branch 1 is solvable.
• Check whether your tool errors reach the model at all. Many frameworks swallow them.
• Version the schema hash in the same commit as the schema change.
• Re-measure after every model or endpoint swap, not just after schema edits.