Your Agent's Free-Text Output Is an API You Never Designed

Most agent demos fail in the same place, and it is not the model.

It is the boundary where the model's output leaves the model and enters your program. Up to that point it is a string. Strings do not have a schema.

So you end up with code like this:

const reply = await llm.complete(prompt);

// please work
if (reply.toLowerCase().includes("approve")) {
  await merge();
} else {
  await requestChanges();
}

This works until the model writes "I would not approve this yet" and your substring check matches approve. Now a change that should have been blocked got merged.

The hidden interface

Any time you parse meaning out of generated text, you have declared an API. You just did not write it down.

That interface has properties you probably did not intend:

• It is not versioned. Change the prompt and the contract silently changes.
• It is not validated. There is no error when the shape is wrong, only a wrong branch.
• It cannot be logged as a decision. You can log a paragraph, but you cannot log "the model chose X from a set of three."

The fix is not a better prompt. Prompt engineering narrows the failure rate; it does not remove the parser.

Ask for a decision, not a paragraph

An agent step usually needs two different outputs, and they should not be the same output:

1. A description for humans — what it noticed, what surprised it, what it is unsure about.
2. A decision for the program — a value the host can validate and branch on.

The decision should come from a closed set that your code already understands:

type ReviewDecision = {
  status: "pass" | "review" | "fail";
  confidence: "low" | "medium" | "high";
  findings: Array<{ file: string; note: string; severity: "info" | "warn" | "block" }>;
};

Now the failure modes separate cleanly:

What broke	Where to look
status came back as "probably fine"	Contract violation — reject before acting
status was valid but wrong	Judgment problem — improve evidence or prompt
status was right but the wrong branch ran	Implementation bug in your own code

Without the typed boundary, all three look like "the AI did something weird," and you have no way to tell them apart.

Validate before you act

The important part is not the schema. It is that the schema is checked before anything changes.

const ALLOWED = new Set(["pass", "review", "fail"]);

function assertDecision(value: unknown): ReviewDecision {
  if (typeof value !== "object" || value === null) {
    throw new Error("Decision is not an object");
  }
  const d = value as Record<string, unknown>;
  if (typeof d.status !== "string" || !ALLOWED.has(d.status)) {
    throw new Error(`Illegal status: ${String(d.status)}`);
  }
  if (!Array.isArray(d.findings)) {
    throw new Error("findings must be an array");
  }
  return {
    status: d.status as ReviewDecision["status"],
    confidence: d.confidence === "high" || d.confidence === "medium" ? d.confidence : "low",
    findings: d.findings as ReviewDecision["findings"],
  };
}

A thrown error is a feature here. It is a loud, recoverable failure that happens before a merge, a payment, or a deploy — instead of a silent wrong branch.

This is the pattern behind structured judgment tools, like the Choice and Score interfaces exposed over MCP in this case: the model supplies the judgment, the typing layer fixes its shape, and the calling program gets a value it can compare or gate on instead of prose it has to interpret.

Keep the evidence with the decision

A typed result is not automatically trustworthy. status: "pass" with no support is just a shorter guess.

So the decision record should carry:

• the inputs the model actually saw (or a hash and a reference)
• the allowed output space
• the chosen value
• the evidence behind it
• an identifier and a timestamp

That gives you a replayable record. When a bad decision ships, you can tell whether the right evidence was missing, the wrong rule was applied, or the validation layer was too loose.

What this does not fix

Typing the output does not:

• make the model correct
• make a subjective label objective
• replace tests, permissions, review, or rollback
• turn an experiment into a production control

It solves one narrow problem: the host no longer has to guess a control signal out of decorative prose.

That is worth a lot. It is much easier to reason about a system where you can see the allowed choices, the selected choice, the evidence, and the resulting action.

A short checklist

Before an agent step is allowed to change state, I want answers to these:

1. What is the model allowed to decide?
2. What is the program allowed to execute?
3. Are those represented separately?
4. Can an invalid decision be rejected before any action runs?
5. Is the evidence visible without reading the whole transcript?
6. Is there a pause or rollback path when confidence is low?

If those are all "no," the demo can still look impressive. It is just not yet an instrument.

---

Disclosure: I maintain JevCases, an independent index of Jev use cases and experiments. It is not affiliated with TypeSafe.

Comments

[arun rajkumar]

The substring example is the one I would put in front of anyone who thinks this is a style preference. "I would not approve this yet" matching approve is not a prompt problem, and no amount of prompt work removes the parser.

Two things I would add from the side where the branch spends money.

The enum gets validated. The mapping from enum to effect does not. A clean status: "fail" tells you the contract held. It says nothing about whether your code then called the right function, and that switch is usually written once and never tested against all three arms. So the closed set has to be closed over effects rather than over words. If two values lead to the same call they are one value, and if one value leads to a call that cannot be undone, it should not share a code path with the ones that can.

And rejection is a branch too. You throw on an illegal status, which is right, but something catches that throw. If the handler retries the same prompt, a model that produced prose once will often produce it again, and you have built an unbounded loop whose cost is real and whose failure mode is a bill rather than an error. Cap the attempts, and make the terminal state a refusal to act rather than a default action. The default is where the substring bug comes back wearing a schema.

Small one on confidence: "low" | "medium" | "high". That field reads like it will end up in a threshold, and the moment it does, it is a number the model chose about itself. Worth deciding early whether it is for humans or for branching, because it cannot safely be both.

[joseph taylor]

I familiar with React right now. But there are something I don't understand. Your explanation helped me understand why this approach works. I'm curious—is it useful you use the same pattern in other application?