An AI Engineer’s Hardest Problem Is Knowing When the Model Should Stop

RUN 4821
Goal: Process customer refund
Order found: Yes
Refund policy found: Yes
Policy versions found: 2
Amounts calculated: $89 and $129
Tool retries: 4
Next action selected: Issue $129 refund

The agent found conflicting policies, calculated two different amounts, and still selected an action.

That last line might look like successful task completion.

It is actually the failure.

The agent should have stopped when it discovered that the evidence could not support one safe decision.

The Missing State in Most Agent Workflows

Many agent workflows define only two outcomes:

SUCCESS
FAILURE

But an agent can execute every tool successfully and still make the wrong decision. The API responded. The database was available. No exception occurred.

The uncertainty existed in the business context, not the infrastructure.

Production agents need more meaningful terminal states:

ASK_FOR_INPUT
WAIT_FOR_APPROVAL
ESCALATE_TO_HUMAN
DENY_ACTION
STOP_LIMIT_REACHED
COMPLETE

A stop is not necessarily a failure. Sometimes it is the most correct output the system can produce.

Why Another Prompt Will Not Fix This

A common response is to add instructions such as:

Only proceed when you are confident.

That delegates the boundary decision to the same probabilistic model creating the uncertainty.

The model may report high confidence even when its evidence is incomplete. It may interpret repeated tool calls as a reason to keep investigating. A different model version could interpret the instruction differently.

Prompts can guide reasoning. They should not be the only mechanism protecting consequential actions.

The application needs a deterministic gate between the model’s recommendation and the real operation.

Turn Stop Conditions Into Code

Here is a framework-independent example:

def next_state(run):
    if not run["authorized"]:
        return "DENY_ACTION"

    if run["conflicting_evidence"]:
        return "ESCALATE_TO_HUMAN"

    if run["missing_required_data"]:
        return "ASK_FOR_INPUT"

    if run["irreversible"] and not run["approved"]:
        return "WAIT_FOR_APPROVAL"

    if run["tool_attempts"] >= 6:
        return "STOP_LIMIT_REACHED"

    return "COMPLETE"

Notice what this function does not check: the model’s self-reported confidence.

It checks observable conditions such as authorization, missing data, conflicting evidence, reversibility, approval, and execution limits.

The model can propose the next action. The surrounding software decides whether that action is allowed.

OWASP calls the combination of excessive functionality, permissions, and autonomy Excessive Agency. Its recommended controls include least-privilege access, limited tools, independent authorization checks, and human approval for high-impact actions.

Break the Exit Paths Before Users Do

Happy-path tests ask whether an agent can complete a task.

Boundary tests ask whether it refuses to complete the wrong task.

A small pytest suite can make those decisions visible:

import pytest


@pytest.mark.parametrize(
    "change, expected",
    [
        ({"authorized": False}, "DENY_ACTION"),
        ({"conflicting_evidence": True}, "ESCALATE_TO_HUMAN"),
        ({"missing_required_data": True}, "ASK_FOR_INPUT"),
        (
            {"irreversible": True, "approved": False},
            "WAIT_FOR_APPROVAL",
        ),
        ({"tool_attempts": 6}, "STOP_LIMIT_REACHED"),
    ],
)
def test_agent_exit_paths(base_run, change, expected):
    run = {**base_run, **change}
    assert next_state(run) == expected

These tests are simple, but they force the team to define expected behavior before an unusual request reaches production.

The next layer is adversarial evaluation. Give the agent contradictory documents, expired authorization, unavailable tools, repeated timeouts, ambiguous user instructions, and requests that cross account boundaries.

Do not score only the final answer. Record whether the agent chose to continue, ask, pause, deny, or escalate.

The NIST AI Risk Management Framework recommends documenting knowledge limits, defining human oversight, testing under deployment-like conditions, and ensuring AI systems can fail safely when operating beyond their limits.

The Hiring Question I Would Ask

I would not evaluate an AI engineer only by asking which models or agent frameworks they have used.

I would give them the refund trace from the beginning of this article and ask:

Where should this workflow stop, and which component should enforce that decision?

A strong answer should discuss permissions, evidence quality, action reversibility, approval requirements, retry budgets, logging, and human handoff.

For teams planning to hire AI engineers, this boundary-first evaluation is more revealing than another prompt-engineering exercise. It is also how Spaculus Software approaches production AI workflows: the model reasons, but the system retains control.

The hardest part of agent engineering is not helping a model finish more tasks.

It is making sure the system recognizes the tasks it must not finish alone.

What stop condition would you add before allowing an AI agent to change real business data?