Goal Completion Verification: The Step 90% of Agents Skip

• Book: AI Agents Pocket Guide: Patterns for Building Autonomous Systems with LLMs
• Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

---

A user asks the agent to "send a follow-up email to last week's leads." The agent thinks for 30 seconds and replies "Done." No email was sent. No error fired. The trace looks healthy. You only find out a week later, because a customer complains they never heard back.

This is the failure mode nobody talks about, and it's the one your monitoring can't see.

Validation isn't verification

The two words get used interchangeably. They aren't the same thing.

Validation asks: is the output well-formed? JSON parses. Schema matches. Required fields present. Email address looks like an email. Status code is 200. Pydantic accepts it.

Verification asks something harder: did the work actually happen in the world?

A send_email tool that returns {"status": "queued"} validates fine. The agent calls it, the JSON is clean, the agent's final message reads "Email sent to all leads." Everything passes. Except "queued" is not "sent," and three hours later the worker that drains the queue throws a DNS error and the message sits there forever.

Validation is a property of the output. Verification is a property of reality.

The shape of a silent-failure trace

You've seen it. The trace pane is a green ladder of tool calls.

[10:42:01] tool.send_email     args={to:"jane@..."}  → 200 OK
[10:42:01] tool.send_email     args={to:"raj@..."}   → 200 OK
[10:42:02] tool.update_crm     args={...}            → 200 OK
[10:42:02] agent.final_message "Done. Followed up
                                with 23 leads."

What happened underneath:

• send_email returned 200 because the SDK queued the message to a local buffer.
• The buffer flush task crashed at 09:00, two hours before the agent ran.
• update_crm 200'd because the upsert wrote last_contact_attempt (which is updated on every call, not every successful delivery).
• The agent's final message is a confabulation of "tool returned 200" plus its instructions.

Nothing here is unique to AI. It's the classic distributed-systems sin of treating "I sent the request" as proof of "the work got done." Agents make it worse because the LLM is fluent enough to write a confident summary on top of a half-broken stack.

The fix is to introduce one more step: a verifier that runs after the agent claims completion. Four patterns work, ranked by cost, latency, and confidence.

Pattern 1: Deterministic snapshot check

The cheapest verifier you can write. Snapshot the relevant state before the agent runs, snapshot after, diff the two. If the agent claimed "I updated 23 CRM records," the post-snapshot should show 23 rows touched since the pre-snapshot.

import hashlib
import json
from dataclasses import dataclass
from typing import Callable

@dataclass
class Snapshot:
    key_count: int
    fingerprint: str
    captured_at: float

def snapshot(query_fn: Callable[[], list[dict]]) -> Snapshot:
    rows = query_fn()
    # sort so order doesn't poison the hash
    rows_sorted = sorted(rows, key=lambda r: r["id"])
    payload = json.dumps(rows_sorted, sort_keys=True).encode()
    return Snapshot(
        key_count=len(rows),
        fingerprint=hashlib.sha256(payload).hexdigest(),
        captured_at=__import__("time").time(),
    )

def verify_changed(
    before: Snapshot,
    after: Snapshot,
    expected_delta: int,
) -> tuple[bool, str]:
    actual = after.key_count - before.key_count
    if before.fingerprint == after.fingerprint:
        return False, "no rows changed"
    if expected_delta and actual != expected_delta:
        return False, f"expected {expected_delta}, got {actual}"
    return True, "ok"

Wire it around the agent call:

def query_leads_touched_today():
    return db.execute(
        "SELECT id, last_contact_at FROM leads "
        "WHERE last_contact_at::date = CURRENT_DATE"
    ).fetchall()

before = snapshot(query_leads_touched_today)
result = agent.run("Follow up with last week's leads")
after = snapshot(query_leads_touched_today)

ok, reason = verify_changed(before, after, expected_delta=23)
if not ok:
    raise AgentVerificationError(f"silent failure: {reason}")

When to use it: any task whose effect lands in a database, file system, or other queryable store you control. CRM updates, ticket creation, scheduled job registration, config changes. If you can write a SQL query that asks "did this happen," deterministic check is the right pattern.

Gotcha: don't snapshot everything. Scope the query to the table and time window the agent could have touched. A whole-database hash is useless because background jobs will flip it on every run.

Pattern 2: Side-effect probe via webhook or audit log

When the work happens in a system you don't own (Salesforce, Stripe, an outbound email vendor, a third-party API), you can't snapshot it directly. What you can do is ask the system itself whether it saw the work.

Every serious external system exposes one of: a webhook for delivery events, an audit log endpoint, or an "events since" query. Use it as a probe.

import time
import requests

def probe_email_delivery(
    message_ids: list[str],
    since_ts: int,
    timeout_s: float = 30.0,
) -> tuple[bool, list[str]]:
    """Poll the vendor's audit log until every message_id
    appears as 'delivered' or we time out."""
    deadline = time.time() + timeout_s
    delivered: set[str] = set()
    missing = list(message_ids)

    while time.time() < deadline and missing:
        resp = requests.get(
            "https://api.email-vendor.example/v1/events",
            params={"since": since_ts, "type": "delivered"},
            headers={"Authorization": f"Bearer {API_KEY}"},
            timeout=5,
        )
        resp.raise_for_status()
        for event in resp.json().get("events", []):
            delivered.add(event["message_id"])
        missing = [m for m in message_ids if m not in delivered]
        if missing:
            time.sleep(2)

    return (len(missing) == 0), missing

The agent records the message_ids it claims to have sent. The probe verifies the vendor saw them and accepted them for delivery.

This works for Salesforce too. The audit log API returns every field change with a timestamp and an actor. If the agent claimed to update opportunity #5512, you ask Salesforce: did opportunity #5512 change in the last five minutes, and was the actor our agent's service account?

def probe_salesforce_change(
    object_id: str,
    actor_user_id: str,
    since_iso: str,
) -> bool:
    resp = sf.query(
        f"SELECT Id, CreatedDate FROM AuditTrail "
        f"WHERE ObjectId = '{object_id}' "
        f"AND ActorId = '{actor_user_id}' "
        f"AND CreatedDate > {since_iso}"
    )
    return len(resp["records"]) > 0

Gotcha: vendor audit logs lag. Stripe events arrive within seconds. Salesforce field history can take a minute. Pick your timeout based on the slowest realistic delivery, not the average. Tail latency is what burns you.

Pattern 3: Judge LLM, used carefully

Sometimes the goal isn't snapshot-able. "Draft a polite reschedule message and send it" has two parts: the send (Pattern 1 or 2) and the polite part. There's no SQL query for politeness.

This is where a judge LLM earns its keep. A small, cheap model (usually one tier below your agent's main model) scores the agent's output against the user's original goal.

import json
from openai import OpenAI

client = OpenAI()

JUDGE_PROMPT = """You evaluate whether an AI agent completed
a user's goal. Be strict. Score from 0 to 1.

User goal: {goal}

Agent's claimed result: {claimed}

Observable evidence (tool outputs, DB state): {evidence}

Respond as JSON with keys: score (float), reasoning (string),
unmet_criteria (list of strings).
"""

def judge_completion(
    goal: str,
    claimed: str,
    evidence: dict,
    model: str = "gpt-4o-mini",
) -> dict:
    resp = client.chat.completions.create(
        model=model,
        messages=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(
                goal=goal,
                claimed=claimed,
                evidence=json.dumps(evidence),
            ),
        }],
        response_format={"type": "json_object"},
        temperature=0,
    )
    return json.loads(resp.choices[0].message.content)

Two caveats nobody puts in the README.

First, cost. A judge call on every agent turn adds 30-60% to your inference bill. If your agent costs $0.12 per session and your judge costs $0.04, that's real money at scale. Mitigation: only judge the final turn, not every step. The intermediate steps don't claim completion.

Second, disagreement with humans. Published agreement rates between LLM judges and human raters sit around 80-95% on subjective criteria, which sounds great until you do the math. Five percent of your verifications are wrong. If your agent runs 10,000 times a day, that's 500 wrong calls. Some pass things that should fail. Some block things that succeeded. For low-stakes UX flavor (was the tone professional?), that's fine. For compliance, refunds, security boundaries, or anything irreversible, it isn't.

Pair the judge with one of the deterministic patterns. The judge says "yes the message is polite," the snapshot says "yes the message was sent." Either one alone is incomplete.

Pattern 4: User confirmation gate

The cheapest verifier in the world is the user. They wanted the thing. They know what "done" looks like. Ask them.

def gated_run(
    agent,
    user_id: str,
    goal: str,
    high_risk: bool,
):
    plan = agent.plan(goal)

    if high_risk:
        proceed = ask_user(
            user_id,
            f"I'm about to: {plan.summary()}.\n"
            f"This will: {plan.side_effects()}.\n"
            f"Confirm? (yes / edit / cancel)",
        )
        if proceed.action != "yes":
            return AgentResult.cancelled(proceed.reason)

    result = agent.execute(plan)

    if high_risk:
        post_check = ask_user(
            user_id,
            f"Here's what I did: {result.summary()}.\n"
            f"Did this match what you wanted?",
        )
        if not post_check.confirmed:
            audit.flag(result, reason=post_check.note)

    return result

Two confirmations. Pre-execution to catch wrong plans before they ship side effects. Post-execution to catch the cases where the plan looked right but the result didn't.

The instinct against this pattern is "but the whole point of an agent is autonomy." Autonomy is fine. Irreversibility isn't. The line you actually want to hold is no irreversible side effect ships without either a human in the loop or a deterministic verifier that proves the side effect succeeded.

User confirmation is also the only verifier that catches the case the other three miss: when the agent technically did the work, but the user meant something different. The follow-up email got sent, but to the wrong list. Pattern 1 says "23 emails sent, matches expected delta." Pattern 4 says "wait, I meant last week's qualified leads, not all of them."

Picking the right pattern per task type

Task shape	Best verifier	Latency cost	Confidence	Why
DB write (your DB)	Pattern 1 (snapshot)	<50ms	High	You own the source of truth
External API write (Stripe, SF, email vendor)	Pattern 2 (probe)	1-30s	High	Vendor's audit log is the source of truth
Subjective output (tone, summary quality)	Pattern 3 (judge) + Pattern 1 for delivery	500-2000ms	Medium	Snapshot can't score writing
Irreversible action (payment, deletion, sent message to external party)	Pattern 4 + Pattern 1	Seconds-minutes	Highest	Cost of being wrong dominates cost of asking
Multi-step: search + compose + send	Pattern 3 on compose, Pattern 2 on send	2-30s	High	Decompose the goal, verify per stage
Read-only research / answering	None needed	0	N/A	Nothing happened in the world

Pick verifiers like you pick database indexes: not by default for everything, but specifically for the tasks where being wrong costs the most.

Where to wire the verifier

There are two places. They aren't equally good.

The first is inside the agent loop, as the agent's last tool. The agent calls verify_completion(...) itself, sees the result, and decides whether to retry. This sounds clean. It isn't.

The agent that just confabulated "Done" is the same agent you're now asking to honestly evaluate its own work. If the LLM is willing to hallucinate that an email was sent, it's willing to hallucinate that the verifier said yes. You're trusting the same fox to count the hens.

The second place is out-of-loop. After the agent finishes, the runtime calls the verifier separately, with its own credentials, its own model context, its own audit log. The agent has no way to influence whether the verifier runs or what it returns. The verifier is a different process, not a different prompt.

class AgentRuntime:
    def __init__(self, agent, verifier_registry):
        self.agent = agent
        self.verifiers = verifier_registry

    def run(self, goal: str, task_type: str, **ctx):
        verifier = self.verifiers.get(task_type)
        if verifier is None:
            raise ConfigError(
                f"no verifier registered for {task_type}"
            )

        before = verifier.capture_pre_state(**ctx)
        result = self.agent.run(goal, **ctx)
        after = verifier.capture_post_state(**ctx)

        check = verifier.verify(
            goal=goal,
            claimed_result=result,
            pre=before,
            post=after,
        )
        if not check.ok:
            audit.log_silent_failure(goal, result, check)
            raise AgentVerificationError(check.reason)

        return result

The agent doesn't know the verifier exists. It can't lie its way past it. The verifier registry is configuration: task type maps to verifier instance. Adding verification to a new task type is a config change, not a code rewrite.

This is the pattern that catches the "send a follow-up email" failure from the opening. The agent runs, returns its confident summary, the runtime checks the email vendor's audit log, sees zero deliveries in the time window, raises. The user finds out in 30 seconds, not a week.

What to do this week

Pick the riskiest agent you ship. The one whose silent failure would cost the most. Add one verifier. Pattern 1 if the work lands in your DB. Pattern 2 if it lands in someone else's system. Wire it out-of-loop. Log every mismatch.

Which silent failure cost you the most? Drop the war story in the comments.

You'll find silent failures you didn't know you had. Everybody does on their first verifier. That's the point.

---

If this was useful

The four-pattern menu above maps directly onto Chapter 9 of the AI Agents Pocket Guide: Patterns for Building Autonomous Systems with LLMs, where I work through verifier design, out-of-loop validators, and the irreversibility threshold in more depth. If you're shipping agents that touch user-facing systems, the chapters on tool design and self-correction loops pair with this post.