orca_forge

Posted on Aug 25 Originally published at forge.workstyle.tech

Guard Implementation Patterns to Stop AI Agent Runaway Behavior — 7 Types Extracted from Real-World Logs

#ai #toolcalling #llm #computeruse

📝 Originally published (in Japanese) at forge.workstyle.tech.

Getting AI Agents to Safely Control PCs and Browsers with Tool-Calling: 7 Guardrails That Stop "Runaway" Behavior

When deploying AI agents that control PCs or browsers via tool-calling—closer to real-world use than demos—you’ll inevitably encounter "runaway" behavior that never appeared in testing. This article compiles real-world logs of runaway incidents from a voice-driven desktop agent and organizes the guardrails that stopped them into 7 patterns.

In one line: Most runaway behavior stems not from the LLM "rebelling," but from "not seeing the consequences of actions" and "not questioning input quality." Guardrails should be designed with this premise in mind.

Pattern 1: Limit Repeated Tool Calls — Total Call Limits Don’t Stop Spamming

Incident: When asked to "search for song X," the agent spammed open_url to the same search URL 5 times in 3 seconds (interleaved with read_screen). Browser tabs multiplied uncontrollably.

Why total call limits fail: The tool limit was already 8 calls per turn. The 5-spam occurred within that limit. The loop wasn’t about "number of calls," but repetition of the same action.

Implementation:

Same (tool name + normalized JSON arguments) limited to 2 calls per turn
Read operations (screen reading, etc.) allowed 3 calls per turn (e.g., "operate → read → operate → read" is valid)
Exceeding the limit returns an error as the tool result (see Pattern 2)
If two consecutive blocks occur, abort the entire loop (no progress = repetition)

key = f"{tc.name}:{json.dumps(tc.arguments, sort_keys=True)}"
if seen_calls.get(key, 0) >= repeat_limits.get(tc.name, 2):
    return tool_error("Repeating the same tool with the same arguments. "
                      "Try a different action or report the current state to the user.")
seen_calls[key] = seen_calls.get(key, 0) + 1

Pattern 2: Tool Results Are "Learning Materials" — Silence Implies Success

Incident: Initially, open_url only returned "(complete)" to the model. The agent couldn’t see what opened → tried to read the screen → failed → assumed "not opened yet" → kept retrying. This was the root cause of Pattern 1’s spam.

Principle:

On success: Return what happened and how (opened URL, page title, written path and size)
On block: Return reason and next step ("Already executed. Proceed with another action based on the result.")
Silent or one-word results create a blind agent

Guardrails (Pattern 1) are insurance; the real fix is richer perception. In this case, adding body text extraction to screen reading eliminated the motivation to retry.

Pattern 3: Explicit Request Gate — Don’t Let the Agent "Read the Air" for Destructive Actions

Incident: While a user was playing music, the mic picked up lyrics "!I'm begging, begging you!" (Måneskin – Beggin'), which STT passed to the tool-decision brain. The agent then created a meeting minutes file—with no user request.

Implementation: High-impact tools (file writes, minutes saving, sending) are only callable when the current utterance explicitly contains request words ("save," "minutes," "memo," "send") via prompt constraints.

- SAVE ONLY ON EXPLICIT REQUEST: call save_minutes / write_file ONLY when the
  CURRENT utterance explicitly asks to save/record/write.
  Never save because the conversation seems worth saving,
  never as a reaction to noise, and never re-save what was already saved.

LLMs’ good intentions ("this conversation is worth saving") become accidents when combined with noisy input.

Pattern 4: Precedent Prompts — Concrete Failure Examples Work Better Than Abstract Warnings

Abstract phrases like "act carefully" or "confirm if uncertain" rarely worked. What did work was writing actual failure patterns verbatim:

- INCOMPLETE UTTERANCE: If the utterance is cut off like "Uh, YouTube—", the user hasn’t finished speaking. Do not call tools.
- NOISE / SONG LYRICS: Meaningless repetitions ("bigbigbig"), alphanumeric strings ("B.T.G.I.K.E.T.E."), or lyric-like text are likely mic-captured media audio. Do not call tools.
- REPORTS ARE NOT COMMANDS: Phrases like "X was opened" are not new instructions.

Background: Voice input lacks chat-like guarantees—utterances aren’t guaranteed to be complete, and environmental noise pollutes them. VAD only silences based on volume; determining if an utterance is a complete command is the tool-decision layer’s responsibility. Note: Echo cancellation only removes sounds your app plays—music from other apps cannot be canceled (mitigation requires OS-level speaker separation).

Pattern 5: Don’t Trust Memory URLs or Paths — Measure Destinations with Tools

Incident: When asked to "research X," the model fabricated plausible URLs from training data. One existed; another was a nonexistent URL invented from a pattern.

Implementation: Standardize research flows:

Open search results page (query URL)
Use screen reading to get a list of real links (hrefs)
Only open those hrefs. Ignore URL strings in body text (e.g., ad display URLs redirect to homepages) or memory-based URLs

Apply the same rule to file paths ("don’t write to 'a familiar path'—verify existence with a system tool first").

Pattern 6: Confirmation Dialogs Are the Last Line of Defense — Don’t Sacrifice Them for Automation Thrills

The reason zero real damage occurred across all these runaways was the design: a human confirmation dialog always precedes OS operations. Even when type_text tried to type "bigbigbig" into an editor, the dialog stopped it.

Guards and prompts are probabilistic. Keep one deterministic layer. Practical criteria for mandatory confirmation: "irreversible?" or "visible externally?" (screen reading = no confirmation; writes/sends/keystrokes = confirmation required).

Pattern 7: Build Your Own Safety Nets for Billing and Resources

Incident: A cloud-GPU worker kept "alive" despite connection failure, not triggering the 25-minute auto-termination based on duration. While the framework’s safety net prevented infinite billing, we didn’t trust it until we verified it.