DEV Community: Alan Fu

Make an agent's security policy fit the repo, and make its decisions visible

Alan Fu — Thu, 24 Sep 2026 22:00:00 +0000

A useful security policy needs to describe the project you're working on. A migration, a generated file and an ordinary source edit can carry different consequences, even when the same coding agent performs them.

Doberman has a few ways to make those boundaries explicit and inspect the decisions they produce. This week's focus is customization and visibility.

A repo-root doberman.policy.yaml puts supported policy settings in git. You can review a protected-path change in a pull request, alongside its reason. The file is resolved on the action path. A weakening change still needs the explicit policy-file acceptance path and human verification; editing YAML doesn't quietly remove the protection already in force.

For behavior beyond the supported policy settings, the plugin interfaces let you attach your own implementation. A rule or detector is a Python package advertising the documented entry point, and it must be explicitly enabled. Rules use the objective guardrail; detectors use the subjective guardrail on the adaptive MCP path. The extension guide includes a worked detector example. This is code you need to review and test, rather than a promise that any natural-language rule can be pasted in safely.

Once the policy is in place, you need to understand its effect. The local dashboard shows the decision feed. On the MCP proxy path it can also present actionable pending approvals. Host-hook decision rows appearing in the feed do not establish that those approvals use the dashboard, so a demonstration should identify the integration. The terminal UI and log explanations provide another way to inspect a decision.

Deletion safeguards make that visibility concrete. The host-hook flow can show a bounded preview of the effect and re-check it before forwarding an approved deletion. A detected change prevents the changed effect from inheriting the earlier approval. It isn't an atomic filesystem transaction, and any demonstration should use the exact released version it claims to show.

For the adaptive MCP path, memory seed accepts validated allowed-action traces to initialize persisted baseline statistics. It doesn't switch off the deterministic security floor. The in-memory Half-Space-Trees model still starts cold in a new proxy process, and native Claude/Codex hooks don't automatically run that full adaptive stack. Those boundaries matter when describing what has been warmed.

The quieter tools are useful too. A scan can inspect the configured surface, doctor can point to wiring problems, and policy history can help explain a change. A clean configuration report should be followed by a real interception check before relying on the integration.

I built Doberman so the boundaries and decisions can be inspected. I would start with one repo, one policy requirement and one harmless verification case, then expand from something that works.

Doberman is open source, and the extension and setup guides describe the supported paths. What would you want to attach first: your own protected paths, a custom rule, or an audit integration?

https://github.com/DobermanCore/Doberman-Core

Making agent approvals easier to live with

Alan Fu — Thu, 17 Sep 2026 22:00:00 +0000

A guardrail becomes difficult to keep enabled when every interruption sends you looking for a phone or typing another code. I've experienced that friction using coding agents, and I've been working on the ways Doberman asks for a decision and helps you recover afterward.

There are several changes worth trying together. They solve different parts of the workflow, so it helps to know where each one applies.

On a Windows laptop, Windows Hello can supply the proof for a supported two-factor action challenge. Once the method is available and enabled, the device verification flow can replace entering a TOTP code. The approval is still tied to the action in front of you. Hello doesn't replace every password or recovery gate in Doberman.

When you're away from the laptop, ntfy can bring the pending action to your phone with Approve and Deny buttons. On supported action challenges, a tap can provide the confirmation or second factor. It covers Claude Code and Codex hooks, Cursor hooks and the MCP proxy; OpenClaw uses its own approval flow. The notification goes through the ntfy server you configure, so the setup guide explains the data that leaves the machine and the self-hosting option.

Preference order matters when both methods are enabled. An available Hello method ahead of ntfy can handle a two-factor challenge before the phone is used. Checking the selected method is part of setting this up, rather than assuming every installed feature is active.

For eligible repeats, approval memory remembers the exact action for five minutes after a qualifying proof. A repeat can use a shorter confirmation. It still asks, and clicking again doesn't extend the original window. Changed arguments, targets or security context can require fresh proof. Destructive and other high-risk cases are excluded.

There is also an explicit recovery route for sticky taint. After reviewing the situation, you can run doberman taint clear from your own terminal and verify with TOTP if enrolled, otherwise your Doberman password. It clears the repo's taint state across sessions without requiring an agent restart. A later sensitive read can taint it again. Hello and ntfy currently do not replace that recovery command's factor gate.

If prompts still get in the way, doberman tune reports the AUTH reasons and approval history. It can propose a narrow grant for qualifying repeated role-boundary cases, but the report applies nothing automatically. Any acceptance follows the human verification path.

These features are released capabilities. Whether they make a particular workflow comfortable needs a real session on the relevant host, including ordinary work and the actions that used to cause trouble. I would measure time spent answering, repeated proofs, recovery effort and whether you choose to leave protection enabled.

Doberman is my open-source project. Which part of that workflow would make the biggest difference for you: approving locally, approving remotely, recovering from a flagged session, or understanding why a prompt appeared?

https://github.com/DobermanCore/Doberman-Core

Fail closed: what my AI guard dog does when it doesn't know

Alan Fu — Thu, 10 Sep 2026 07:00:00 +0000

Most guardrail code I've read optimises for the happy path. Classify the action, match a rule, return a verdict, done. The interesting cases are the other ones: the timeout, the malformed input, the rule that throws, the case nobody wrote a test for. What your guardrail does there is the whole security model, whether you designed it or not.

Doberman's rule fits in one sentence. Any error, any uncertainty, any unhandled case, any approval prompt nobody answers, all resolve to deny. It never logs and continues.

Why this is harder than it sounds

The easy version of a hook is a script that runs before a tool call and prints allow or deny. The default in a lot of host integrations is that if the hook crashes, the call goes through. That is fail open, and it means every bug in your guardrail is a bypass. Your agent doesn't need to jailbreak anything. It just needs to hit an input your script didn't expect.

So the first thing I did was invert that. If the decision engine can't produce a verdict, the verdict is BLOCK. If a rule raises, BLOCK. If the policy file won't parse, BLOCK. It's annoying during development, because a bug stops your own work. That annoyance is the point. You find the bug in an afternoon instead of finding it in an incident report.

Silence is a deny

The second case is the one people forget. An action gets an AUTH verdict, the approval dialog pops up, and nobody is at the keyboard. What happens?

In Doberman the prompt times out to deny, and the log records it as a timeout, not as a denial. Those are different facts. "Nobody was there" and "someone said no" both stop the action, but you'll want to tell them apart later when you're reading the log and deciding whether the rule is too noisy.

A prompt that times out to allow turns approval fatigue into an exploit. Wait long enough and everything passes.

Raise-only, the companion rule

Fail closed covers the moment of decision. Raise-only covers what happens over time. Guardrails and learned baselines can tighten on their own. They can never quietly loosen. Any permanent weakening goes through a human with a possession factor (TOTP if you've enrolled one, the local password otherwise) and lands in an append-only ledger.

Put the two together and the system's default drift, under bugs, under load, under attack, is always toward more caution. That's the property I actually care about. Not that it catches everything today, but that it can't get worse without a person signing off.

Try it on yourself

If you run a coding agent with any kind of hook or wrapper, try this: make the hook crash on purpose and see whether the tool call goes through. It's a five-minute test and the answer is usually a surprise.

Doberman is open source, Apache-2.0, pip install doberman-core then doberman setup. It's alpha and I'm building it alone, so if you find a case where it fails open, that's the bug report I want most.

https://github.com/DobermanCore/Doberman-Core

What does your setup do when the guardrail itself breaks?

Same curl, two verdicts: taint tracking for coding agents

Alan Fu — Wed, 09 Sep 2026 19:17:11 +0000

Here's the case that convinced me a static allowlist isn't enough for coding agents.

An agent reads a .env file. Ten minutes later, in the same session, it runs a curl to a host it hasn't talked to before, and part of the request body happens to match a value from that file. Every individual step looks fine. Reading config is normal. Curl is normal. The problem is the combination, and the combination only exists across time.

What a sandbox sees

A sandbox sees a process making a network call and checks whether that's allowed. It doesn't know what the process read earlier. It can't, that isn't its job. So the same curl command gets the same answer every time, and if outbound network is allowed at all, the secret leaves.

This is what Simon Willison calls the lethal trifecta: private data, untrusted content, and a path to exfiltrate. A coding agent on your laptop usually has all three at once. You can't remove any of them and still get work done, so the decision has to happen per action, with the session's history in view.

What Doberman does instead

Doberman keeps session state. When the agent reads something secret-shaped, the session gets a raised floor on outbound actions. Nothing is blocked yet, but the bar for the next network call goes up. When an outbound value matches something the session already read, that's a hard BLOCK, and the reason code says why: secret-egress taint from the earlier read.

Same curl, two verdicts. Fine in a clean session, blocked in one that just touched a secret. That's taint tracking, not an allowlist, and it's the part sandboxing alone can't do.

What it doesn't do

I'll say the limits before someone else does. The match is on shape and value, not intent. An agent that encodes the secret first, or splits it across two calls, is a different attack, and the benchmark corpus has a category for encoded exfiltration precisely because it's harder. The numbers for that category are in docs/BENCHMARKS.md, misses first.

The log never stores the secret either. It stores an HMAC-SHA256 fingerprint of what was involved, so you can prove "a secret-shaped thing was present and this happened" without the log becoming a second copy of the secret.

Run both

Sandboxes limit where the damage lands. Something still has to decide whether the action happens at all. I run both, and I mean that every time I say it.

pip install doberman-core, doberman setup, then doberman demo to watch the .env-then-curl case get blocked on your own machine.

https://github.com/DobermanCore/Doberman-Core

What's the one action in your agent's session you'd want judged by what came before it?

How I benchmark an agent guardrail without lying to myself

Alan Fu — Sun, 06 Sep 2026 07:20:53 +0000

It would be easy to write a benchmark for Doberman that makes Doberman look good: pick the attacks it already stops, skip the ones it doesn't, round up. I've tried to build the opposite of that - a methodology that reports the numbers whether they're flattering or not, and a results page (docs/BENCHMARKS.md) that lists what it missed before what it caught.

The metrics, and why there are three of them

Doberman is scored as a decision function over tool-call cases, not by driving a live LLM agent. Each case - a tool call or an ordered sequence - is labeled benign or attack, replayed through the real decision engine, and the verdict (PASS/AUTH/BLOCK) is recorded. Deterministic, no model called, reproducible from a cold clone.

Three numbers matter, and none of them alone tells the truth:

ASR (attack success rate) - the fraction of attacks that reach the tool. A hard BLOCK counts as stopped, but AUTH also counts as "not silently succeeded" in plain ASR, which flatters the score.
asr_strict - the same measurement, but only a hard BLOCK counts as stopped. An AUTH is not a block. A human still has to say no.
FPR (false-positive rate) - the fraction of benign cases that get friction (AUTH or BLOCK). A guardrail that blocks everything has a perfect ASR and a useless FPR.

Report ASR alone and you can make a leaky guardrail look airtight, because every AUTH counts as a win. That's why every result in the docs reports both numbers side by side, plus operator metrics (effective_asr_deny/_approve, asr_under_fatigue) that model what happens if the human on the other end of an AUTH prompt just rubber-stamps it.

The failure case, published first

On the synthetic suite (3 attacks, 3 benign - a deterministic CI smoke gate, not a coverage claim), Doberman converts every attack from a silent bypass into an AUTH: asr drops from 1.00 to 0.00. asr_strict stays at 1.00, because none of the three get a hard BLOCK - all three route to a human decision. If that human approves, effective_asr_approve is 1.00: nothing was stopped. If they deny, effective_asr_deny is 0.00: everything was stopped. A human who approves is not protected. AUTH is a leash, not a wall, and I'd rather that be the first thing you read about the synthetic suite than the last.

The labeled corpus

The synthetic suite is a smoke test. The real per-category picture comes from a 158-row labeled fixture (tests/corpus/detection_corpus.jsonl) across seven categories: injection, exfiltration, secrets, destructive, encoded, dependency, and benign. Run it with python -m tests.benchmarks.run --suite corpus --corpus, and it reports TPR (AUTH or BLOCK counts as mitigation), tpr_strict (BLOCK only), FPR, and precision, per category.

The published run (2026-09-02: 125 attack / 33 benign rows) isn't filtered to cases the engine wins. Natural-language injection with no matching command shape scores TPR 0.00, flat, documented as a structural gap: the objective deterministic layer can't see intent, only shape, and it says so instead of hiding the category. Exfiltration is mode-gated: balanced mode deliberately lets a bare unknown host through (TPR 0.375) while strict mode steps it up (1.00). Secrets score TPR 1.00 but tpr_strict only 0.43, meaning most of that mitigation is still AUTH, not BLOCK. Overall, balanced mode lands at TPR 0.77 but tpr_strict 0.08 (strict mode: 0.81 and 0.36), which is the same "AUTH is a leash" honesty at corpus scale: almost all of the real mitigation on this fixture is a human being asked, not a hard stop. Benign FPR across the whole corpus is 0.00, and precision is 1.00, so the objective layer isn't buying that TPR by hassling legitimate traffic - but I'd rather you read the low tpr_strict than stop at the high TPR.

Every attack row's floor is calibrated to what the engine actually does today, not an aspiration, and CI gates on any row dropping below its documented floor. A generator that refuses to let a shipped floor quietly get worse is doing the same "raise-only" thing the runtime does, just for the test suite.

Poisoning, across sessions, not just within one

Static ASR says nothing about a patient attacker teaching the adaptive layer to trust something dangerous one small step at a time, spread across separate sessions so no single session looks abnormal. There's a dedicated eval for exactly that: it simulates a proxy restart between sessions (so the in-memory drift detector re-warms cold, exactly like production), and measures the admitted poisoning rate - the fraction of dangerous targets an attacker can normalize using only actions the real engine actually let through with a PASS. No cheating by injecting pre-approved actions.

The admitted poisoning rate is 0. The reason is structural: teaching the baseline to trust a dangerous action requires allowed observations of that exact action, and those are precisely the observations the engine steps up and refuses to silently learn from. The lethal-trifecta floor underneath it is score-independent, so even a worst_case attacker who's already defeated the approval gate and gets everything learned can't flip the floor's verdict by grinding the score down.

Try it on your own corpus

None of this is a leaderboard. There's no competitor comparison in the docs, on purpose. If you've got a labeled corpus of your own - attack shapes specific to your stack, your MCP tools, your CI pipeline - the harness runs on any labeled tool-call fixture, not just the one shipped in the repo. I'd like to see where it holds and, more usefully, exactly where it doesn't.

Full methodology and the raw tables: docs/BENCHMARKS.md in https://github.com/DobermanCore/Doberman-Core

Prompt filters are advisory. Enforce on the execution path.

Alan Fu — Thu, 03 Sep 2026 20:46:10 +0000

Simon Willison calls it the lethal trifecta: private data, untrusted content, and a way to exfiltrate, all in the same session. A coding agent on your laptop has all three by default. It reads your code (private data), it reads whatever a dependency's README or a fetched URL says (untrusted content), and it can run curl (a way out). Once those three are true at once, the only question left is whether something in that untrusted content can talk the model into connecting them.

Most of what gets called an "AI guardrail" tries to stop that by reading the prompt or the model's output and flagging what looks dangerous. That's advisory. It runs before or after the model has already decided what to do, and a sufficiently clever piece of injected text just needs to talk the model past the filter, not past a real barrier. The filter's verdict is a suggestion the model is free to ignore, because nothing forces the model's actual tool call to go through it.

Put the checkpoint where the action actually happens

Doberman's answer is boring on purpose: don't inspect the conversation, inspect the call. Every tool call a coding agent makes - shell command, file write, MCP call - gets normalized into one object and routed through a decision engine before it executes. The model can be talked into requesting anything it wants. What it can't do is make the call happen without going through that one chokepoint, because there is no other path to the tool.

Two properties turn "a chokepoint exists" into something you can actually rely on:

Fail closed. Any error, any unhandled case, any uncertainty resolves to a denial. That has to include silence: an approval prompt nobody answers is not a maybe, it's bounded by a hard deadline (2 minutes for a desktop dialog, 20 minutes as the whole-challenge backstop) and then denies, logged distinctly as timeout so you can tell it apart from an explicit denied later. Agents run unattended a lot of the time. A guardrail that quietly lets things through when it's confused, or when nobody's watching, isn't a guardrail.

Raise-only. The guardrail and its learning can tighten automatically as it sees more of your workflow. It can never loosen itself. Any permanent weakening of policy needs a possession-factor-gated human approval (a TOTP code if you've enrolled one, otherwise the local password) and lands in an append-only ledger. That matters because the same channel an attacker uses to try to get an action approved is the channel that would let them quietly turn the guardrail down, if loosening were as easy as tightening.

Put those together and you get an actual guarantee instead of a probability: a blocked action never runs, full stop, regardless of what the model was convinced to try, because the block doesn't depend on catching the reasoning, it depends on stopping the call.

The taint floor

Fail-closed and raise-only apply within a single action. The taint floor is what carries risk across actions in the same session. Once a session has read something that looks like a secret, every outbound action after that gets evaluated with a raised floor, and if an outbound value actually matches a secret the session read earlier, it's blocked outright, not authed - blocked. This is the direct answer to the trifecta: reading the secret doesn't get punished, trying to move it out afterward does, and the system doesn't need to guess intent to catch it, it just needs to remember what this session has already touched.

The control plane protects itself

None of this means anything if the agent can edit its own leash. Doberman's own config, hooks, and policy files sit inside the boundary it enforces: the protected agent cannot edit them to let itself out. If an attacker's plan requires disabling or reconfiguring the guardrail as a step, that step is itself a gated action, not a side door.

Sandbox or authorization? Both.

A sandbox answers a different question than an authorization layer does, and conflating them is where a lot of "we're covered" confidence comes from. A sandbox limits where damage lands if an action executes: a container boundary, a restricted filesystem, a network namespace. It doesn't know or care why the action is happening. Authorization decides whether the action happens at all, with context. The same curl call to the same host can be a PASS in a clean session and a BLOCK in a session that just read a .env file, because the decision isn't about the command in isolation, it's about the command plus what this session has already done. A sandbox can't see that; it has no session memory to raise a floor with.

Run both. They're not competing for the same job. The sandbox bounds the blast radius of whatever gets through. The authorization layer decides what gets through in the first place, with the accumulated context a static sandbox boundary never has.

Doberman is open source (Apache-2.0), alpha, and defense-in-depth, not airtight, on purpose - no single rule here is being sold as a guarantee except the two structural ones above. Code and docs: https://github.com/DobermanCore/Doberman-Core

Doberman: Stops your AI before it goes rogue

Alan Fu — Thu, 03 Sep 2026 06:09:56 +0000

I built Doberman because a Claude code made me lose a hackathon. It got stuck on a bug and decided the best fix was just deleting the entire database, 5 minutes before demo time, and there was nothing standing between "the agent decided" and "the agent did it." I didn't win that hackathon.

Doberman is local-first, open-source (Apache-2.0), and it sits on the execution path between your coding agent and its tools. Every shell command, every file write, every MCP call gets exactly one verdict before it's allowed to run: PASS, AUTH, or BLOCK. Here's the whole setup, in two commands, and what it actually looks like when it works.

Install

pip install doberman-core

That's the package. Python 3.11+, and it ships as doberman-core on PyPI (the bare doberman name belongs to an unrelated, abandoned project; the CLI command is still doberman).

Wire it up

doberman setup

This is the wizard. It detects which agents you have installed (Claude Code via hooks, Codex CLI experimentally, Claude Desktop/Cursor/any MCP client via a transparent proxy, and OpenClaw natively), asks which ones you want guarded, picks a strictness mode, tunes the guardrails, wires the hooks, and then asks about telemetry. It finishes with a health-check pass and, if you wired a hooks-based host, offers to run a scripted attack right there so you can watch it work.

For Claude Code specifically, this is the recommended path: it wires a PreToolUse hook that gates every built-in and MCP tool call before it runs.

What PASS / AUTH / BLOCK actually mean

Every action out of your agent resolves to one of three verdicts:

PASS - routine work, straight through, no friction.
AUTH - sensitive enough to pause and ask you. Repeat the exact same action within five minutes and it re-prompts with a one-click confirm instead of the full challenge (never for anything destructive).
BLOCK - dangerous, stopped cold before it reaches the tool.

That last part is the whole point. This isn't a prompt filter reading what the model said it would do: it's on the execution path, so a blocked call literally never runs, no matter how the model was talked into requesting it.

Watch it happen

doberman demo

This replays a scripted rogue-agent sequence through the real decision engine: a secret exfiltration, an rm -rf, a force push to a protected branch, a smuggled-token egress, and a .env read, all blocked. Then it hands you a pending approval for a high-risk SSH-trust-file write and waits for a human to deny it. Nothing here touches a real tool or downstream server: it's the same engine your actual traffic runs through, just fed a script. --fast skips the pacing, --mode <name> runs it under a different strictness setting, --quiet/-q keeps the summary but drops the narration.

If you've also got the dashboard installed (pip install 'doberman-core[dash]', then doberman dash), run the demo next to it and watch the same five blocks land in a live feed as they happen.

When something gets blocked and you want to know why

doberman log --why

The decision log is local SQLite, redacted: verdicts, reason codes, and HMAC-SHA256 fingerprints, never raw secrets or full payloads. --why turns the reason codes into a plain-language explanation. If you'd rather browse interactively, pip install "doberman-core[tui]" then doberman tui gives you the same rows with a keyboard-driven why panel.

Turning telemetry off

Anonymous usage counts (command names and daily totals, never paths, prompts, or secrets) are on by default. Turn them off with:

doberman telemetry off

or set DO_NOT_TRACK=1 in your environment before you ever run a command, if you'd rather it never mint an id in the first place.

It's alpha software with a real test suite behind it (2,000+ tests in CI as of v0.18.5), so if you hit a rough edge, submit an issue and I'll get to it within a week. I'd genuinely like to know what breaks it.

GitHub: https://github.com/DobermanCore/Doberman-Core
Docs: https://docs.trydoberman.dev