Earlier this year I wrote about CAS, a terminal shell where conversation and direct manipulation live in the same window. That post was about the interface. This one is about the part underneath that I care about more: how the thing decides what it's allowed to do.
The question that drove the whole design is simple to state and hard to answer well: when an LLM can call tools, browse the web, and act across multiple connected services, how do you keep it inside the lines?
The usual answer is to ask the model nicely in a system prompt and hope. I didn't want to hope.
The shape of the problem
CAS grew a set of capabilities that each let the model reach further out into the world:
- It connects to MCP servers and calls their tools.
- It fetches and reads web pages.
- It coordinates tasks that span two or more of those connected workspaces at once.
Each of those is a place where a confused or adversarial model could do something I didn't ask for. Call the wrong tool. Navigate somewhere it invented. Route a step to a workspace that wasn't part of the task. The more capable the system got, the more these seams mattered.
I decided early that the safety story could not live in prompts. Prompts are suggestions. I wanted guarantees that hold regardless of what the model outputs.
Every LLM call has a named owner
The first move was structural. In CAS, the shell — the thing that routes your messages — makes zero LLM calls. Not one. Every call to a model is owned by a named agent with a single responsibility:
GenerationAgent creates new workspace content
EditAgent applies a change to existing content
CombineAgent merges multiple workspaces into one
ChatAgent handles conversational turns
MCPAgent plans and executes MCP tool calls
WebAgent plans and executes web actions
OrchestratorAgent coordinates multi-workspace tasks
Seven agents. The shell detects intent (with regex, no model involved — that part's in the last post) and delegates to exactly one of them. The agent owns the call, the prompt, and crucially the contract around the call.
This matters because it gives every model interaction a boundary with a name on it. When something goes wrong, the error says mcp-agent: precondition violation: connection_has_tools. You always know which agent, which phase, which rule.
Contracts the model can't see
The boundary itself is a contract, in the Bertrand Meyer sense — preconditions checked before the work, postconditions checked after. The whole type is about thirty lines of Go:
type Rule struct {
Name string
Description string
Check func() bool
}
type Contract struct {
AgentName string
Preconditions []Rule
Postconditions []Rule
frozen bool
}
Three properties make this work as a safety mechanism rather than decoration.
It runs in Go, outside the model. The checks are ordinary functions. The model never sees them, can't reason about them, and has no path to modify them. There is no prompt injection that reaches a func() bool.
It's frozen before the call. An agent constructs its contract and calls Freeze() before it ever talks to the model. The agent cannot loosen its own constraints partway through. The contract for a given operation is fixed before the operation begins.
It fails closed. A violation returns an error and stops the operation. No fallback, no retry, no "let me ask the model to fix it." If a postcondition fails, the work is discarded.
Here's what that looks like in practice for the agent that creates content:
c.Preconditions = []contract.Rule{
{
Name: "workspace_type_allowed",
Check: func() bool {
return req.WSType == "document" ||
req.WSType == "code" ||
req.WSType == "list"
},
},
{
Name: "prompt_not_empty",
Check: func() bool { return strings.TrimSpace(req.Prompt) != "" },
},
}
c.Postconditions = []contract.Rule{
{
Name: "content_not_empty",
Check: func() bool { return contentSize > 0 },
},
{
Name: "content_size_within_limit",
Check: func() bool { return contentSize <= maxContentBytes },
},
}
Nothing surprising. That's the point. The constraints are boring, legible, and enforced by the compiler-checked language rather than the probabilistic one.
The interesting postconditions
The boring checks earn their keep. But a few postconditions are doing real work that a prompt could never guarantee.
The edit agent has a truncation guard. A common failure mode when you ask a model to "add a section" to a long document is that it returns just the new section — or a summarized version of the whole thing — instead of the full updated content. Silent data loss. The postcondition catches it:
{
Name: "result_not_drastically_shorter",
Check: func() bool { return contentSize >= minExpected },
// minExpected is 10% of the original length
}
If the model returns something less than a tenth the size of what it was editing, the operation fails closed and the original content is untouched. The model can't lose your document by misunderstanding the task.
The MCP agent has a tool-existence check. When the model decides which tool to call, the postcondition verifies the chosen tool name actually exists on the connected server:
{
Name: "tool_name_known",
Check: func() bool {
if toolCall == nil {
return true
}
for _, t := range req.Connection.Tools {
if t.Name == toolCall.ToolName {
return true
}
}
return false
},
}
A model that hallucinates a tool called delete_everything never gets to call it, because delete_everything isn't in the server's advertised tool list and the contract rejects the plan before execution.
The web agent does the same thing for URLs — a navigation target has to be a real, parseable, absolute URL, not something the model invented mid-sentence.
Coordinating agents without trusting them
The hardest case is the orchestrator. When you say "read the issue in the Linear workspace and open a GitHub PR for it," one instruction spans two connected services. The orchestrator's job is to decompose that into ordered steps and run them.
This is exactly where an agentic system can go off the rails — a multi-step plan is a lot of surface area. So the orchestrator never touches a workspace directly. It produces a plan, and a separate interface executes each step:
type StepExecutor interface {
ExecuteStep(ctx context.Context, wsID, instruction, prior string) (string, error)
}
The orchestrator knows nothing about MCP servers or web sessions. It emits steps; the shell routes each to the right agent. And the orchestrator's contract enforces the one rule that matters: every step in the plan must target a workspace that actually exists.
{
Name: "plan_steps_have_known_workspaces",
Check: func() bool {
for _, step := range plan.Steps {
if !idSet[step.WorkspaceID] {
return false
}
}
return true
},
}
The model is handed a list of real workspace IDs and asked to plan against them. If its plan references an ID that wasn't in the list, the plan is rejected before a single step runs. The model can't route work to a service it dreamed up.
Output from each step is fed as context into the next, so the GitHub step sees what the Linear step found — but the structure of the plan is validated by code, not trusted from the model.
A dial for how much rope
Validation handles "is this action well-formed." It doesn't handle "do I actually want this action to happen." For that there's an autonomy dial, and it's per-operation:
- suggest — the agent plans the action and shows it to you. Nothing executes.
- confirm — the agent executes, but pauses before each step for your approval.
- run — the agent executes freely within its workspace scope.
In the terminal, confirm mode is a real pause. The agent's goroutine blocks on a channel while the UI shows you what's about to happen:
CONFIRM │ [ws-linear] list open issues │ y: proceed │ n: skip │ esc: cancel
Press y and the step runs. Press n and it's skipped. Press esc and you bail. The goroutine is genuinely waiting — no polling, no timeout games, just a blocking read on a channel that the keypress writes to. Human-in-the-loop, implemented as backpressure.
And because every step — what it was, what it returned — is written to SQLite as it happens, there's a full audit trail afterward. You can reconstruct exactly what the system did and what flowed between steps.
What this buys
None of these pieces is clever on its own. A contract is an if-statement. A tool-existence check is a loop. The autonomy dial is a blocking channel read.
The value is in where they sit. The model proposes; Go disposes. Every reach into the outside world passes through a named boundary with constraints that were fixed before the model spoke, that the model can't see or alter, and that fail closed when violated. The security-relevant decisions live in the deterministic layer, not the probabilistic one.
It turns out a 40-year-old idea about software correctness — preconditions, postconditions, fail closed — is a remarkably good fit for the problem of keeping a language model inside the lines. Meyer was writing about catching programmer mistakes. The same machinery catches model mistakes just as well, and for the same reason: it doesn't trust the caller to be correct.
CAS is open source — github.com/goweft/cas. The contract layer is internal/contract, and each agent's contract lives next to it in internal/agent. It's all about as boring as the snippets above, which is exactly what I wanted from a safety layer.
Top comments (0)