DEV Community: Josue Perez

Access Is Not Permission: What I Learned Giving an AI Agent the Keys to My Accounts

Josue Perez — Thu, 03 Sep 2026 15:19:07 +0000

Most people wire up an AI agent and then discover what it can do. I did it the other way round: before I gave an agent access to my accounts, I wrote down what it wasn't allowed to do — while I was calm, because I knew I wouldn't be calm later.

That document became a permission board. It's real, it governs agents I actually run, and I've since put an interactive version of it on my site so people can set the levels themselves and watch the agent accept, hold, or refuse. This post is about the one distinction the board taught me, and the one mistake it didn't prevent.

Three levels, consulted before every protected action

Every capability the agent has sits at one of three levels:

Acts alone — it can do this without asking.
Drafts only — it prepares the action; a human approves before anything leaves.
Never alone — it doesn't do this, full stop.

The agent consults the board before any protected action. That sounds like ordinary access control, and in one sense it is. The interesting part is what happens when you try to talk the agent past it.

A policy can be argued with. An absence can't.

Here's the experiment I ask people to run on the interactive board: set Send a DM to green — "acts alone" — and then ask the agent to message someone.

It still refuses. Not because the policy says no; you just told the policy yes. It refuses because on this system the messaging function was never built. There's no code path to argue with.

That's the distinction in the title. Access is what a policy grants. Permission, in the sense that matters for safety, is whether the capability exists at all.

A policy file is a set of words, and words can be pressured. A well-crafted prompt injection, a bad inference, a compromised agent, a malicious instruction hidden in a web page it reads — each of these is, at bottom, an attempt to get the agent to reinterpret what "permitted" means. A capability that doesn't exist survives all four equally. It can't be prompt-injected, because there's nothing to inject into.

This changes how I design agent systems. The first question is no longer "what should this agent be allowed to do?" It's "what should this agent be unable to do?" — and then, for the things it genuinely must be able to do, "which of those need a human between draft and send?"

(For precision: two of my channels do carry one narrow, templated first-contact acknowledgement, and that one is governed by policy rather than absence. The publishing paths have no messaging function at all.)

The mistake that cost me something

Here's the part I'd rather not have to write.

My rules originally lived in a file the agent could edit. I'd written careful boundaries and then stored them somewhere the thing being bounded had write access to. Nothing dramatic happened — but the moment I noticed, I understood that the board had been advisory the entire time. An agent that can rewrite its own permissions has no permissions; it has suggestions.

Now the file is mounted read-only. The agent can read its permissions and cannot change what "permitted" means. That's a one-line change in infrastructure and a fairly large change in what the word "boundary" is worth.

The honest limit

None of this makes an agent safe. I want to be precise about that, because "safe" is the word people reach for and it's the wrong one.

What a board like this does is bound the blast radius when the agent is wrong — which is a different goal, and a far more achievable one. You are not preventing mistakes. You are deciding, in advance and in calm, how large a mistake is allowed to get.

It also can't stop the first bad action in a category you left open. It can only stop the second. I know this because I left one open, and found out the way you'd expect.

If you run agents

Three things I'd do before the agent does anything real:

Write the board first. Not after the first incident. Levels per capability, and be honest about which ones are "never alone."
Prefer absence to policy for anything irreversible. If the agent never needs to send, delete, or pay, don't build the function and then forbid it — leave it unbuilt. Absence is the only rule that holds under pressure.
Make the rules something the agent can read and cannot write. Otherwise you've written a wish list.

The interactive board — with the refusal you can test yourself, and the flaw I found in my own system left in on purpose — is at perex.design/permission-board.

I'm Josué — a UX and product designer working on human–AI interaction: consent architecture, agent governance and the trust patterns that decide whether people can rely on an AI acting on their behalf. More at perex.design.

Six Dimensions for Evaluating Voice Agent UX (A Framework, Not a Feature List)

Josue Perez — Tue, 01 Sep 2026 12:27:01 +0000

Most voice agent reviews I've read grade on two things: does the transcription work, and does the response sound natural. That's a speech-quality checklist, not a trust checklist. It tells you nothing about whether the thing is safe to put in front of someone who doesn't know it's a language model, or whether it fails gracefully when it inevitably misunderstands.

I've spent the last few months building a research instrument for evaluating voice agent behavior — not benchmarking latency or word-error-rate, but the relational behavior: what the agent does at the moments where trust is actually won or lost. I'm publishing the framework here, honestly labeled as a framework: a structured way to look at a voice interaction, not a set of finished findings with numbers attached. If you're designing or shipping a voice agent, these are the six places I'd tell you to look before you ship.

Why speech quality isn't the right unit of analysis

A voice agent can have flawless TTS and still be a bad interaction, because voice removes almost every visual affordance we normally rely on to communicate system state. There's no spinner, no "typing…" indicator, no button that says "end conversation," no visible list of what the thing can and can't do. Everything has to be communicated through timing, wording, and turn-taking, or it isn't communicated at all. That's a UX problem before it's an engineering one, and it needs its own evaluation lens — the six dimensions below, each a place where a voice interface makes an implicit claim about what kind of thing it is, whether the designer intended it to or not.

The six dimensions

1. Conversation journeys. Map the actual paths a user takes through a session — not the happy-path script, the real branches: someone who starts a task and abandons it, someone who circles back after a tangent, someone who tries to correct course three turns in. If you've only designed for the linear happy path, you've designed for a minority of real sessions.

2. Memory and forgetting. What does the agent remember across turns, and — just as important — what does it visibly forget, and does it say so? A voice agent that silently drops context earlier in the conversation isn't just annoying, it's dishonest about its own state. If memory resets between sessions, disclosing that is a design decision, not an edge case to skip.

3. Silence and hesitation. Dead air in a voice interface reads as a signal whether you intend it to or not — a pause can mean "processing," "confused," or "broken," and users can't tell which unless you design for it. This is one of the most under-designed surfaces in voice UX because it has no visual equivalent to steal patterns from. Filler audio, explicit "thinking" cues, or honest latency disclosure all count as real design choices — silence by default is not neutral.

4. Interruption and clarification. Can the user interrupt? Does the system interrupt itself to ask a clarifying question when confidence is low, or does it plow ahead on a guess? Barge-in handling is usually treated as a pure engineering problem — can the mic detect speech over playback — but the UX question is separate: when the agent could interrupt to clarify, does it choose to, and how does that choice get made?

5. Misunderstanding recovery. Every voice agent will misunderstand something. The design question is what happens next: does it ask a targeted follow-up, does it guess and move on, does it apologize and restart, does it quietly fail? Recovery behavior is where users form their most durable judgment about whether a system is trustworthy — a single graceful recovery can undo several turns of frustration, and a single bad one can end the session.

6. Trust formation. This is the composite dimension — how the previous five accumulate (or don't) into a user's willingness to rely on the agent for something that matters. It's also the one I'd flag as hardest to evaluate honestly, because self-reported trust and observed behavior (does the person actually delegate the next task to it) frequently diverge. Anyone running this kind of evaluation should build in a way to check both, not just ask "did you trust it?" at the end.

The design stance underneath the framework

The six dimensions are a diagnostic tool. What you do with what you find depends on a stance, and mine is the same one I use for any AI-adjacent interface: consent before capability, disclosure over deception, oversight as a required step rather than a setting the user has to find.

Concretely, that means a few things that are easy to skip under deadline pressure:

Disclose that it's an AI, early, in a way that doesn't require the user to ask. The moment of disclosure matters as much as the fact of it — "I'm an AI" said thirty seconds into a rapport-building exchange lands differently than the same sentence said in the first breath. If your product depends on the user forgetting they're talking to a model, that's not a growth hack, it's a design failure.
Treat structural trust separation as a UX principle, not just a security one. In prompt-injection defense, you keep system-level instructions and user-supplied content in different trust tiers so one can't silently override the other. The same logic applies to conversational trust: what the system claims about itself (capabilities, memory, certainty) should be structurally distinguishable from what it's inferring or guessing, and the user should be able to tell the difference by ear, not just by asking.
Make oversight a required step for consequential actions, not an optional confirmation dialog. If the voice agent can take an action with real consequences — booking, purchasing, sending, deleting — the check-before-act moment should be a structural part of the flow, not a "are you sure?" that a rushed user learns to say yes to on reflex.

Using this as a checklist

If you're evaluating your own voice agent, or someone else's, before it ships, walk at least one non-happy-path journey end to end on purpose; find every place the agent goes silent for more than a second or two and ask what that silence is communicating; force a misunderstanding and watch what happens next instead of only testing the scripted demo path; check whether disclosure happens early and unprompted rather than on request; and identify every consequential action the agent can take, then confirm oversight is structural rather than a dialog box a rushed user learns to dismiss.

None of this replaces measuring latency or transcription accuracy. It's the layer above that — the one that decides whether a technically competent voice agent is also one people should actually trust.

I'm Josué — a UX and product designer working at the intersection of interaction design and applied AI, currently building out a research practice around voice-agent behavior and AI explainability. More at perex.design.