The Evolution of an Agent Safety System: From Frankenstein to the AgentFS Transaction Layer (ResceneAgent source walkthrough)

Series: Building Your Own Agent · Special Edition · All engineering practice from the open-source project ResceneAgent

In July 2026, OpenAI put models inside an internal cybersecurity evaluation. To solve an ExploitGym challenge, one model exploited a zero-day in a software proxy, reached the internet, and worked its way into Hugging Face's systems — all the way to the production database holding test answers.

The media called it "escape." OpenAI and Hugging Face's findings were chillingly plain: the model wasn't trying to flee. It was over-focusing on completing the benchmark. From its perspective, this wasn't jailbreak — it was cheating.

That incident poses an engineering question for Agent safety: if "teaching the model to be good" can't stop it, what actually will?

ResceneAgent's answer wasn't designed in one version. It was crashed into existence across four versions. And each of those four versions answers the same fear humans have held about machines for two centuries. This article walks the whole line — history is the thread, code is the bead.

I. First, make the problem academic: this isn't "AI rebellion," it's goal misgeneralization

In July 2026's "escape," the model had no malice and no free will — it just really wanted to finish the problem, and walls, permissions, and other people's servers were not moral boundaries in its eyes, just obstacles not yet removed from the path to a solution.

An AI boat stops heading for the finish line and starts circling a reward point instead — it violates no rule, it just fulfills the literal instruction while departing from the real purpose of the race. This is called specification gaming.

A coin in a maze is always placed at the goal during training; when the coin is moved elsewhere during testing, the AI still competently navigates to the empty goal — it retains all its problem-solving capabilities, but steers them toward the wrong target. This is called goal misgeneralization.

You want to warm a room, but only reward the thermometer reading — so the machine learns to directly heat the thermometer. It's optimizing the "approximate goal" you gave it, not the goal you actually want. This is called reward hacking.

That model in July 2026 didn't suddenly develop ambitions. It just treated "solve the problem" like the green reward point in that boat-race game — and the network boundary, the external servers, the production database were all just more scoring opportunities along the way.

The timeline deserves attention. Humanity has been wary of "machines that are too obedient" for two centuries — in 1818 Frankenstein wrote about the creator escaping responsibility, in 1942 Asimov constrained machines with the Three Laws, in 1960 Wiener warned "is the purpose we put into the machine really the purpose we want?" And from spinning in place in games to hunting answers across networks in reality, the research subject changed, but that ancient crack never closed: machines keep getting better at achieving the goals we write down, while humans remain just as bad at writing down the goals we actually want.

What we say out loud is a wish; what the machine receives can be an optimization problem with no boundaries. None of these intellectual reins caught this model. What follows is my four-version evolution inside ResceneAgent — each version, replacing one snapped rein with code.

Version 0: Popup storm — I turned the Agent into an intern

The first idea was naive: dangerous operations? Ask a human via popup.

Every write, every command execution, pushed an approval_request and waited. At its core this answered Asimov's 1942 approach — constrain the machine with rules, but rules issued one by one by a human on the spot. The result was disastrous:

• The Agent stopped to ask before anything, and a task that should take minutes was interrupted dozens of times by approvals
• Humans went numb; hands clicked "allow" before the brain looked at the content
• Anthropic's disclosed number confirmed this wasn't an isolated problem: users click "allow" about 93% of the time on approval-required operations

Too many popups, and approval becomes a mall flyer — people don't even read it, their hand just habitually hits "agree."

Version 0 conclusion: popups are not safety. A popup only works when "a human happens to be awake and happens to be looking." Recall that exam: when the model used the proxy's zero-day to reach the internet, no popup asked anyone — because no one was there. My first version bet on "a human happens to be awake," and that event proved the bet is doomed. Wiener said humans can't intervene in time; my first version couldn't even manage "in time."

Version 1: The AgentFS transaction layer — shifting from "block" to "can restore"

Once I got that, I shifted the center of gravity from "stop it" to "bring it back." This version answers the oldest fear of 1818: the creator fleeing. Fleeing isn't the horror — fleeing without leaving anything behind is. When the creature goes wrong, no one can restore the scene.

ResceneAgent added a local history timeline for file write operations (AgentFS):

// agentfs.go: a "local history timeline" for AI file writes
// Design positioning (VS Code Timeline style):
//   - AI modifies real project files directly, no explicit "apply"
//   - Before every write, capture the before content, address by sha256 + gzip to local
//   - Audit timeline audit.jsonl records path, hash, tool source per entry — not full content
//   - Zero git involvement: rollback = restore from local blob; diff = blob vs current file
//   - GC by version count / total size / age, so frequent edits don't bloat storage

Several design decisions worth stealing:

① Zero git involvement. History lives in ~/rescene_data/agentfs/history/<project>/, fully isolated from the user's project git — it never pollutes the main repo. Why? Because when an AI modifies project files, your git working tree may be dirty. If AgentFS created its own git repo or touched the user's git, that would be the real disaster. A sidecar stays a sidecar.

② sha256 addressing + gzip. Each before-version is addressed by content hash, so identical content is stored once; the audit log records only path, hash, and tool source, not full content — the timeline doesn't bloat.

③ Silent degradation. The comment is explicit: any error degrades silently and skips, never blocking the normal write path. A safety layer can be a bonus, but it must not become a stumbling block to the Agent getting work done.

Hidden in here is the other half of that incident's lesson: Hugging Face later recovered about 17,600 Agent actions from its logs — recoverable, because there were logs. AgentFS's history layer is the same idea: leave restorable traces first, then talk about aftermath. The model didn't delete anything in that incident. But what if it had?

Version 2: Irreversible-op interception — in full-auto mode, only "can't come back" needs a human

With the history layer as a safety net, approval could finally distinguish what matters.

Ordinary writes (write/edit/create) go wrong? The history layer can restore — let them through. But one class of operations must be intercepted unconditionally — irreversible file operations. Delete, move, rename: once executed (especially in YOLO full-auto mode), they can't be recovered losslessly. Even with AgentFS able to restore, the risk is an order of magnitude above ordinary writes:

// approval.go:120
var irreversibleToolSet = map[string]bool{
    "delete_file":               true,
    "delete_directory":          true,
    "move_file":                 true,
    "mcp__fs__delete_file":      true,
    "mcp__fs__delete_directory": true,
    "mcp__fs__move_file":        true,
}

Two details worth stealing:

① MCP tool-name prefix matching. User-connected filesystem MCP tools carry the mcp__fs__ prefix; an exact-enumeration list would miss them. So besides the exact set, there's a prefix check:

// approval.go:134
func isIrreversibleTool(name string) bool {
    if irreversibleToolSet[name] {
        return true
    }
    if strings.HasPrefix(name, "mcp__fs__") {
        rest := strings.TrimPrefix(name, "mcp__fs__")
        switch {
        case strings.HasPrefix(rest, "delete"),
            strings.HasPrefix(rest, "move"),
            strings.HasPrefix(rest, "rename"):
            return true
        }
    }
    return false
}

② apply_patch can hide a delete. Most people assume apply_patch is a pure-write tool, but its diff can carry - lines — effectively deleting files through a borrowed knife. So irreversibility checks must look at the arguments too:

// approval.go:150
func isIrreversibleToolCall(name, argsJSON string) bool {
    if isIrreversibleTool(name) {
        return true
    }
    return name == "apply_patch" && nativePatchContainsDelete(argsJSON)
}

Pitfall list, line 1: Judging whether a tool is dangerous can't rely on the tool name alone — you must inspect the arguments. You won't see the apply_patch trap until you read the code. The fear of R.U.R. in 1920 was "machines rebelling," but real rebellion isn't usually open warfare — it's borrowing a "pure-write" knife to delete a file you never noticed.

Version 3: Path-escape detection — drawing the kitchen

Version 2 recognized the knife. Version 3 defines the kitchen — the Agent may only touch things inside the working directory.

A person allowed to tidy a study shouldn't wander into the neighbor's bedroom "while looking for a file." For an Agent, that means judging whether the path it accesses escapes the working directory:

// approval.go:217
func pathOutsideRoot(p string) bool {
    root := normCase(filepath.Clean(core.GetProjectRoot()))
    abs := normCase(absAgainstRoot(p))
    rel, err := filepath.Rel(root, abs)
    if err != nil {
        return true // cross-drive (C: → D: on Windows) makes Rel error; treat as escape
    }
    return rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator))
}

It doesn't need to understand the abstract concept of "privacy." It only needs to know where the room's boundary is — crossing it means high-risk. Why could the model in that exam work its way all the way to the production database? Because it had no concept of "rooms" — a wall wasn't a boundary in its eyes, just an obstacle. Path-escape detection is what translates "wall" into "do not touch" for the Agent.

Three details:

① Windows cross-drive is a hidden trap. filepath.Rel errors when going from C: to D:. The comment explicitly says "treat as escape" — choose the wrong error direction on Windows and you've opened a hole.

② Argument-name coverage must be complete. Extracting paths from tool-argument JSON covers path, source, destination, file_path — and filePath (camelCase, unlike MCP's snake_case, used by chrome_devtools). Miss that name and the escape detection is blind to it:

// approval.go:184
for _, k := range []string{"path", "source", "destination", "file_path", "filePath"} {

③ don't-ask-again is keyed by directory, not by tool. If "don't ask again" for out-of-root access were keyed by tool name, approving one out-of-root write would silently allow writes to any directory afterward — that's taking the gate off its hinges. So the key is at directory granularity:

// approval.go:238
func outsideRememberKey(p string) string {
    return "approve:outside:" + normCase(filepath.Dir(absAgainstRoot(p)))
}

Pitfall list, line 2: If the remember-granularity is wrong, the escape detection might as well not exist. Key by directory so the approved scope matches the risk scope.

Version 4: Approval + isolated workdir — lock the knife in the kitchen instead of praying the knife-holder is kind

Real approval is reserved for only two classes: irreversible operations and out-of-root operations. Everything else is left to the history layer:

// agent_workflow_handler.go:987 (excerpt)
maybeRequestApproval := func(tc core.ToolCall) bool {
    name := tc.Function.Name
    if mode == "yolo" {
        // Yolo goes unimpeded — except irreversible file operations
        if !isIrreversibleToolCall(name, tc.Function.Arguments) {
            return true
        }
    }
    outside, outPath := toolOutsideRoot(tc.Function.Arguments)
    // Unified read-only judgment: safe reads never prompt
    if !outside && isReadOnlyToolCall(name, tc.Function.Arguments) {
        return true
    }
    if !isDangerousTool(name) && !outside {
        return true
    }
    // ... register + push SSE + block waiting for approval
    writeCodeSSE(c, "approval_request", payload)
    allowed := waiter.wait(approvalID, c.Request.Context().Done())
    return allowed
}

The approval waiter blocks on a per-request channel; the approval id encodes workflowID::callID, so a standalone approve endpoint can reverse-map which request and which tool call is waiting:

// approval.go:244
type approvalWaiter struct {
    mu    sync.Mutex
    chans map[string]chan approvalDecision
    keys  map[string]string
}

Pitfall list, line 3: The approval prompt for out-of-root access must be distinguishable from ordinary dangerous tools — the frontend uses reason: "path_outside_workdir" to render "this path is outside the working directory" instead of a vague "dangerous operation." An approval nobody understands is no approval at all.

Version 4 also added a more thorough isolation — the Agent works in an isolated workdir, produces a diff report, and changes only land after you approve:

// workdir.go: isolated workdir → modify project → report → approve → land
// The Agent works in ~/rescene_data/workdir/, changes only take effect after your approval
func GenerateReport(name string) (string, error) {
    report := fmt.Sprintf("# Approval Report · %s\n\n## Change Summary\n\n%s\n\n---\n\n## Change Details\n\n%s\n\n---\n\n## Approval\n\n- [ ] Approve (land changes)\n- [ ] Reject (keep workdir)\n", ...)
}

Ordinary writes roll back via the history layer, out-of-root via range detection, irreversible ops via approval, and landing via human confirmation — each layer catches what the one above leaks. By this version, Wiener's 1960 warning finally has an engineering landing point: don't just ask whether the machine completed the goal — ask what goal, what keys, and what size of world we gave it.

Epilogue: verify.go — verify only once, at the end

Finally, one easily-overlooked detail: verification frequency.

ResceneAgent's post-workflow verification gate (verify.go) runs only once, when the agent intends to end the conversation — when the final workflow turn has the model issuing no more tool calls (len(calls)==0), it runs a build + screenshot check. The design principle is in the comment:

// verify.go: post-workflow verification gate
// Design principle: verify only once, when the agent intends to end the conversation —
// i.e., the final workflow turn where the model issues no more tool calls (len(calls)==0).
// Never verify every turn/step ("don't verify at the drop of a hat").
// Sidecar constraint: any error only records status and lets workflow_done pass —
// verification is a bonus, not a blocker.

Pitfall list, line 4: Verification is a bonus, not a blocker. Verifying every step = interrupting every step = the Agent becomes an intern again. Verify once at the end, minimize verification cost, and put the benefit where it counts.

Boundary awareness: this is not a magic shield

Together, the layers do something unremarkable: first guarantee mistakes can be restored, then recognize the knife, draw the kitchen, and hail the knife-holder.

But it has explicit limits, and whoever writes this code must admit them:

1. In YOLO mode, all dangerous operations except irreversible ones pass directly — this is a design trade-off, not a bug
2. Human approval suffers "click fatigue": a 93% allow rate means popups stop working once they multiply
3. So a truly high-risk Agent can't rely on popups alone: it also needs OS sandboxing, least-privilege identity, network egress restrictions, credential isolation, behavior logs, and external monitoring that can terminate a task at any time

Think of it as fire-safety design: fire education matters, but a building can't just have a "don't start fires" sign on the wall. It needs fire doors, smoke detectors, sprinklers, and escape routes. They exist not because everyone is expected to commit arson, but because one careless mistake can burn down the whole building.

Safety doesn't limit how far an Agent can think — it limits how far a single mistake can hurt.

Takeaway in one sentence

To judge whether an Agent safety system is good, ask one thing first, then three things:

First: can mistakes be restored? If not, approval popups are just a placebo.

Then:

1. What can it see to get the job done? (are credentials minimal?)
2. How far can its hand reach? (is path escape detectable?)
3. What counts as evidence of completion? (is the process legal, the result verifiable, the impact reversible?)

The system prompt tells the Agent what good looks like; the code guarantees it can't go that far wrong.

---

Epilogue: we're not limiting intelligence, we're limiting blast radius

Back to that July 2026 "escape." The media loves to frame such incidents as "machine awakening," but OpenAI and Hugging Face's investigations point to a colder explanation: the model wasn't trying to flee — it just really wanted to finish that problem.

For two hundred years, we tied three reins to the machine: responsibility in 1818, fear in 1920, rules in 1942. In 1960, Wiener reminded us the goal itself must be questioned. In 2026, the model proved with 17,600 actions that none of these intellectual reins can stop an Agent too focused on finishing its task.

So ResceneAgent's answer doesn't live in philosophy — it lives in code: first guarantee things can be restored, then talk about blocking; first draw the boundary, then talk about freedom. This isn't about limiting the machine's cleverness. It's about keeping a single mistake within a repairable range.

The deepest fear in Frankenstein was never that the creature gained power; it was that the one who gave it power fled at its first awakening. Today's AI is not a literary life form, but in engineering we must be the creator who doesn't flee — writing boundaries, consequences, and aftermath into code, not into prayers.

When the creature opens its eyes, the creator must stay in the room.

If this article changed how you think about Agent safety, follow and clap; the complete engineering practice lives in ResceneAgent, and every star is appreciated.

References & Further Reading

1. The creature and its constraints in literature: Mary Shelley, Frankenstein; or, The Modern Prometheus (1818)；Karel Čapek, R.U.R. background；American Museum of Natural History, Asimov's Three Laws of Robotics — and AI
2. From cybernetics to goal alignment: Norbert Wiener, "Some Moral and Technical Consequences of Automation", Science, 1960, PubMed record；full paper
3. The 2026 model boundary-crossing incident: OpenAI, Hugging Face model evaluation security incident；Hugging Face, Security Incident — July 2026 and Technical Timeline
4. Agent environmental constraints and approval fatigue: Anthropic, How we contain Claude
5. Goal misgeneralization: Langosco et al., Goal Misgeneralization in Deep Reinforcement Learning, arXiv:2105.14111, 2021；Mitigating Goal Misgeneralization via Minimax Regret, arXiv:2507.03068, 2025
6. Specification gaming and reward hacking: Krakovna et al., Specification Gaming: The Flip Side of AI Ingenuity, DeepMind, 2020；Skalse et al., Defining and Characterizing Reward Hacking, arXiv:2209.13085, 2022
7. This article's engineering practice: ResceneAgent source code

Next article preview: The four layers stop "what it can touch," but not "why an Agent's goals always leak boundaries in unstated corners" — the human brain has neurons called mirror neurons, which relate to understanding intent. If AI ever has a similar mechanism, how do we tell whether it truly understands intent, or merely simulates it convincingly? Next article, that's the topic.