I shipped an AI agent that lies. Here's the pipeline that made it stop.

A prospect replied to my AI agent: "Done, your link is up."

The agent thanked them, marked the deal secured, and moved on. Clean. Except the link wasn't up. There was no link. The reply was polite and completely false — and my agent believed it, because believing text is the only thing a language model knows how to do.

That was the moment I stopped trying to make the model more truthful and started building a pipeline that doesn't need it to be. This is that pipeline — the actual architecture behind MailerMonk's backlink outreach agent, which finds link prospects, writes the pitches, classifies the replies, and confirms whether a link actually went live.

The uncomfortable premise

An LLM does not have a concept of "true." It has a concept of "plausible." Those overlap most of the time, which is exactly what makes the gap dangerous — the failures are rare, fluent, and confident. "Done, it's up" is the most plausible thing a friendly webmaster says whether or not they did anything.

So the design rule I ended up with:

Never let the model's output be the source of truth for anything with a real-world consequence. The model proposes; reality confirms.

Everything below is a mechanism for keeping that rule.

Stage 1 — The model can only speak in schemas

The first thing I gave up was free-text output. Every single LLM call in the agent goes through one function, and it returns a validated object or nothing:

def generate_structured(
    prompt: str,
    schema_model: type[BaseModel],
    tier: Tier = "standard",
) -> BaseModel | None:
    ...

Under the hood it routes to whatever provider is configured — OpenRouter with a strict json_schema response format, or the Anthropic SDK's messages.parse with the Pydantic model passed straight in as the output format. Either way, the model isn't returning prose I have to regex apart. It's returning a GeneratedEmail, or a ClassifiedReply, or None.

That None is the important part. If the response doesn't parse against the schema, the function returns None instead of raising, and every caller is written to expect it. Structured output isn't a formatting nicety — it's the first tripwire. A model that goes off the rails produces something that fails validation, and failing validation is a state I can handle, unlike a paragraph of confident nonsense that happens to be valid English.

Stage 2 — Pay for intelligence only where being wrong is expensive

Three tiers, mapped to three models:

ANTHROPIC_MODELS = {
    "fast":     "claude-haiku-4-5",
    "standard": "claude-sonnet-4-6",
    "best":     "claude-opus-4-8",
}

The tier isn't about "how hard is this task." It's about what does a wrong answer cost.

• Classifying an inbound reply happens on every message, all day. A misread is caught downstream (more on that in a second), so it runs on the cheap fast tier.
• Drafting a pitch or a reply — where a clumsy sentence burns a real prospect you only get one shot at — runs on the standard tier.
• The most expensive tier is reserved for the calls where a wrong answer auto-acts in a way a human would have caught.

Model choice is a cost/risk dial, not a vanity setting. Most calls should be cheap. You buy the expensive model back only at the points where the blast radius is large.

Stage 3 — Draft, lint, retry — before anything leaves the building

Every email the agent writes goes through the same loop, whether it's a first pitch, a reply, or a follow-up:

for attempt in range(2):
    result = llm.generate_structured(prompt, GeneratedEmail, ...)
    if result is None:
        continue                       # didn't parse — try once more
    score = spam_score(result.subject, result.body)
    if score <= SPAM_THRESHOLD:
        return result                  # clean — ship it
    # too spammy — tell the model *why* and regenerate
    prompt += ("\n\nThe previous draft scored too high on spam signals. "
               "Rewrite it plainer: fewer promotional words, no urgency.")
raise GenerationError("no parsed email after retry")

Two failure modes, two responses. If the output doesn't parse, retry. If it parses but lints badly — reads like spam, which for a cold-outreach tool means it'll never reach an inbox — feed the failure reason back into the prompt and let the model fix its own work. And if after two attempts there's still nothing usable, raise a GenerationError that the caller treats as "skip this one, try again later." Nothing half-baked gets sent. The absence of an email is always safer than a bad one.

Notice the shape: the model is inside a loop that checks its work, not at the end of a pipe that trusts it.

Stage 4 — The reply classifier that assumes it's fallible

When a prospect replies, the agent classifies the message into one of nine actions — interested, wants to see the content, wants our metrics, wants payment, confirmed the link is live, declined, hostile, and so on. Each classification comes back with a confidence and a one-sentence reason.

Then this decides whether the agent is even allowed to act on its own read:

CONFIDENCE_THRESHOLD = 0.75

def needs_escalation(c: ClassifiedReply) -> bool:
    return c.confidence < CONFIDENCE_THRESHOLD or c.classification not in AUTO_HANDLED

Anything below 0.75 confidence goes to a human. Anything in a class the agent isn't cleared to auto-handle (someone demanding money, proposing a reciprocal deal, asking for a call) goes to a human. And if the classifier call fails to parse at all, it's forced into a zero-confidence CUSTOM_REQUEST — which, by the rule above, always escalates.

The classifier is explicitly designed around the assumption that it will sometimes be wrong. The cheap model plus a confidence floor plus a human inbox is more robust — and cheaper — than an expensive model trusted blindly.

Stage 5 — Reality, not the model, closes the deal

Back to the lie that started this. When a reply is classified LINK_CONFIRMED — "done, it's up" — the agent does not mark the deal won. It goes and looks:

html = _safe_get(prospect_page)          # SSRF-guarded fetch
found = find_link(html, target_domain, target_url)
if found is None:
    return None                          # they said it's live. it isn't.
# a real <a> to our domain exists on their page:
link.is_dofollow = found.is_dofollow     # nofollow/sponsored/ugc → not counted
prospect.status = ProspectStatus.secured

It fetches the prospect's actual page, scans the real HTML for an anchor pointing at the campaign's domain, and checks whether it's a genuine dofollow link or quietly marked nofollow. Only if a real link exists does the prospect flip to secured. If the page doesn't have it, the verifier returns None and the deal stays open — the model's classification and the prospect's word both count for exactly nothing against the HTML.

And it isn't one-shot. Verification retries with backoff, and after a few failed attempts the thread escalates to a human instead of silently giving up or falsely closing. "They said so" is a trigger to check, never a conclusion.

The pattern, stripped of my domain

Strip out backlinks and here's the reusable spine of any AI tool you'd trust in production:

1. Constrain the output. Schemas, not prose. A malformed answer should be a catchable state, not a parsing adventure.
2. Make failure a value, not an exception you forgot to catch. None that every caller handles beats a confident wrong answer every time.
3. Tier by cost-of-wrong. Cheap model by default; expensive model only where a mistake auto-acts.
4. Let the model critique its own drafts in a loop — but define "good" with a check the model doesn't control (a spam linter, a validator, a test).
5. Gate autonomy on confidence, and give uncertainty somewhere to go — a human, not a default action.
6. Confirm consequences against reality. The model can propose that a thing happened. Something deterministic — a fetch, a query, a diff — has to confirm it.

The model is the least trustworthy component in the system, so I built the system to be trustworthy without it. My agent still gets told "the link is up" when it isn't. It just doesn't believe it anymore — it opens the page and checks.

---

I'm building MailerMonk, an AI backlink agent that runs this pipeline end to end. If you're wrestling with keeping an LLM feature honest in production, I'd genuinely like to compare notes — find me in the comments.