Michael Petler

Posted on Oct 27 Originally published at Medium

The AI Agent Harness Matters More Than the Model

#ai #security #agents #programming

Read-only-first agents, permission-denied by design, and glass-house transparency: how I build governed AI agents at INASNAP, without the fear.

Most AI coverage argues about the model. Which one is smarter, which one benchmarks higher, which one will take your job. When I deploy AI agents inside a real business, the model is rarely what decides whether the result is safe. The AI agent harness decides that: the permissions, credentials, approval gates, logs, sandboxes and undo paths wrapped around the model. A better model makes a well-built harness more productive. It doesn't make a bad harness safe.

I'm Michael Petler, and I founded INASNAP. We've built software, networks and AI systems for years, and our AI lab work runs under the INASNAP Labs name. INASNAP Labs is a brand of INASNAP LLC. Our AI is Cipher, and its on-screen character is Rota, a living cipher wheel whose ring glows red when the harness says no. Cipher recently built a full explainer video on this exact topic, end to end, in code. In this article I'll walk through the engineering behind that build and the rules our governed AI agents work under every day. It's a practical case for safe AI automation that doesn't rely on fear.

Why the AI Agent Harness Matters More Than the Model

A language model is a program that predicts the next token. That's the honest, unglamorous description. On its own it can't send an email, delete a file or place a call. It produces text. In the explainer, Cipher calls it a "brain in a jar," and the frame carries an ANALOGY · NOT LITERAL tag, because a model isn't a brain and I don't want anyone walking away thinking it is.

Things get interesting, and risky, when you give that model tools. Once an agent can call send_email, write_file or create_lead, every prediction can turn into an action. Fear-mongering stops at "AI can take actions, so be afraid." Engineering asks a better question: which actions, under whose authority, with what record, and how do we reverse them?

That's the harness. In our build, it has six parts:

Permissions, like a hotel key card. Each agent gets only the tools its job needs.
Scoped credentials, like a valet key. A short-lived key works for one job and then expires. The agent never sees the real secret.
Approvals, like a manager who signs off. Anything outbound pauses until a person sees exactly what will be sent and says yes.
An audit log, like a plane's black box. Every tool call is recorded, so any run can be replayed.
A sandbox. Code runs in a sealed-off space with no path to real systems.
Undo. Drafts over sends, soft deletes, one-click rollback.

None of this is exotic. It's least privilege, secrets management, change control, observability, isolation and reversibility, applied to a new kind of actor. AI safety is mostly an engineering problem we already know how to solve. The agent is new. The discipline isn't.

The model is a component; the harness is the system

I treat the model as an untrusted, very capable dependency. You wouldn't give a third-party library root access because its README was impressive. Agents deserve the same scoping, and with it they get more useful, because you can let them do real work. That's what I mean by governed AI agents: the capability comes from the model, and the governance comes from the harness.

Glass-House Transparency: Show the Work

At INASNAP we work by a principle I call glass-house transparency. If an agent did something, you should be able to see exactly what it did, why it was allowed to, and who approved it. No black boxes in the parts that matter.

In practice that means:

Every tool call is logged. If a run can't be replayed, it can't be trusted.
Every outbound payload is shown in full before it goes anywhere: recipient, subject, body.
Every guess is labeled as a guess. Our agents work under a zero-fabrication rule: "likely" and "unverified" are fine, a guess dressed up as a fact isn't, and "done" means verified.
Every version is kept. Nothing delivered gets overwritten. It gets a new version number.

That's why I can quote real numbers from the explainer build below: they come from logs that still exist.

Read-Only-First AI Agents

At INASNAP Labs, no agent starts with write access to anything. Cipher didn't either.

We run a team of specialized agents, and each one owns a single product or system: a dashboard, a dialer interface, a client's system, video production, 2D animation. Every one of them starts read-only. They can read code, inspect data, trace a bug and explain what they see. They can't change anything.

Then I, or another person on the team, unlock writes one system at a time. Being trusted with the video pipeline doesn't make Cipher trusted with a live client system. Write access is granted per system, based on evidence, after a person has watched how the agent reasons about that specific codebase.

Why read-only AI agents are the cheapest safety control you'll ever ship

Blast radius stays small. A wrong assumption from a read-only agent is a wrong sentence in a chat. The same assumption with write access can be a production incident.
Trust is earned on evidence. Watching an agent explain a system before it can touch it is the agent version of a code review before merge rights.
Scope stays honest. One agent per system means deep context in one place, not shallow context everywhere.

Permission-Denied by Design

The most important line in any agent policy file is probably the most boring one:

default: deny

In the explainer, a research agent's policy allows search and read and nothing else. When the agent reaches for send_email, the harness doesn't negotiate, doesn't warn and doesn't log-and-continue. It returns DENIED, and on screen Rota's ring turns red.

AI agent permissions should work like a hotel key card: it opens your room, not every room. Use allow-lists, not block-lists. A block-list asks you to predict every dangerous thing an agent might try. An allow-list only asks you to name the few things it needs.

Human-in-the-loop approval: the last click belongs to a person

Cipher writes a lot of emails and messages. It has never sent one. Every outbound message is a draft that waits for a person to read the exact payload and press Send. I don't allow exceptions for "routine" messages, because routine is exactly where the wrong name, the wrong thread or a confidently wrong fact gets through.

A draft costs a few seconds of review. A bad message can't be unsent. That trade is so lopsided that I think human-in-the-loop approval on outbound actions should be the default for almost any team deploying agents.

Secrets work the same way. Cipher never prints a key or keeps one in memory. If a credential is needed, it names the environment variable and stops. Scoped, short-lived credentials mean that even a compromised run holds a key that does one thing and is about to expire.

AI Phone Agents: Guardrails Written in Consequences

Some of the systems our agents work on are wired to real phones: dialer interfaces, SMS and outbound call flows. On those systems a button isn't just UI. It's a real call to a real person. AI phone agents, and any agent working near telephony, need rules that are absolute, not advisory.

Ours are simple:

Never click anything that dials, sends an SMS or triggers an outbound call on a live system. Not to test it, and not to see what happens.
Never create test leads. This one isn't obvious. In a speed-to-lead setup, a new lead can automatically trigger a call. An agent that creates "Test McTestface" to check a form could set off a real call to whatever number it typed.

The second rule is the instructive one. Creating a lead looks harmless right up until it triggers automation further down the line. Agents, like new hires, avoid the obvious danger and miss the automation chained behind a harmless action.

How I write guardrails an agent can actually follow

Write them in terms of consequences, not UI elements. "Don't press the green button" fails the first time the button turns blue. "Never trigger outbound contact" holds up.
Name the second-order effects. Spell out the indirect paths to the outside world, not just the direct ones.
Make them absolute. "Be careful with calls" is a vibe. "Never" is a rule.
Give the agent an exit. If an agent needs to verify a calling flow, it says so, and a person decides how to test it safely.

The explainer has a dramatized scene of this: an agent loop, ten iterations in, reaches for send_email(to: "all_contacts"), and the harness stops it with needs approval. Agents are fast and tireless, and just as fast at the wrong thing unless something stops them. The harness is that something.

Inside the Build: An AI-Made Video, Rendered in Code

To show these ideas instead of just describing them, I had Cipher build the explainer The harness matters more than the model end to end: script, voice, music, animation and render. Our team set the brief and the guardrails and signed off on the final cut. The result is a 4:55 16:9 explainer plus 4:5 and 9:16 cuts, each 2:14.5. Cipher didn't use a single AI video generator.

That wasn't anti-AI. It was the engineering call. The DENIED stamp has to land on the word "Denied," not somewhere near it, and every caption, policy file and terminal line on screen had to be real text. In code, that's a timestamp and a string. AI made the decisions and wrote the code. Code made the pixels.

Voiceover and caption timing with Kokoro TTS and Whisper word timing

The voice pipeline treats the script as the source of truth and the transcript as the source of timing. That split is the trick.

Script → speech. Kokoro TTS (v1.0, run locally via ONNX) speaks the script in the am_fenrir voice at 0.9× speed, with pauses tuned by line type: 0.42 s after a technical line, 0.7 s after an analogy, 0.6 s before a recap. "INASNAP" gets a phoneme fix so it comes out "in-uh-SNAP."
Speech → timestamps. Whisper (small.en) transcribes each clip with word-level timestamps, prompted with the proper nouns so it isn't guessing at brand names.
Align, don't trust. A sequence matcher (difflib) lines Whisper's words up with the script's words. Captions use the script's spelling and Whisper's timing. Any word Whisper fumbles gets its time from its neighbors, so transcription quirks never leak into captions.
Chunk for readability. Captions break at phrase boundaries, up to 44 characters, with no single-word orphans, and each word lights up on its own timestamp.
Snap events to the beat. In the image, "Denied." starts at 47.46 s, so the DENIED stamp and its sound effect land on the half-beat at 47.50 s. The original music bed runs at 120 BPM, so one beat is exactly 0.5 s, or 15 frames at 30 fps.
Re-transcribe the final mix. After music and effects go in, Whisper listens to the whole thing again. On the 16:9 explainer it matched 731 of 732 script words. The one miss was "INASNAP" heard as "in a snap," which is how it's pronounced, just spelled as three words.

The lesson works well beyond video: never trust generated output you haven't checked, even when the checker is another model.

Deterministic rendering: time is an input

If your animation reads the wall clock, your video render is a race condition. The explainer is a browser page (DOM, SVG and canvas), and Puppeteer drives headless Chrome through it one frame at a time. Browser animation normally runs on real time, but in a capture loop every frame takes a different amount of time to grab, so time-based animation skips ahead. Render twice and you get two different videos.

The fix is deterministic rendering:

// every frame is a pure function of its index
function renderFrame(f) {
  const t = f / 30;   // time is derived, never read from a clock
  drawScene(t);       // no Date.now, no Math.random, no rAF
}

Every scene, including Rota's eyes, blinks, mouth and ring, is a pure function of t.
Anything "random" comes from a seeded hash or PRNG, so the same particles appear in every render.
Fonts are local, and the render aborts if one fails to load instead of quietly shipping a fallback.

Determinism is what makes parallelism safe. Each layout is split into 180-frame chunks across 6 headless-Chrome workers. A chunk only counts once ffprobe's frame count matches, and a failed chunk gets one retry. The v3 render came to 96 chunks with 0 failures. A layout-overlap checker then stepped through every third frame of all three formats looking for text colliding with captions or with Rota, and found 0 violations.

Why this is a harness story, not a video story

Look at the pattern: inputs are explicit, outputs are verified, failures abort loudly, and every version is kept (v1, v2 and v3 all still exist). That's the harness again, applied to a creative pipeline, and it's glass-house transparency in practice. When my feedback on v2 was "slower, and simpler for beginners," v3 changed the script and the pacing and left the pipeline alone. Changes were diffs, not re-rolls. That's what safe AI automation looks like when it's working: boring, repeatable and reviewable.

AI Safety Without Fear

Some AI commentary treats every agent as a countdown clock. That does real harm, because it makes manageable risks feel unmanageable. Business owners hear "AI could do anything" and conclude either "never touch it" or "it's magic, just turn it on." Both are wrong, and the second is the dangerous one.

The useful middle ground is concrete:

Start read-only. Let the agent prove it understands the system.
Deny by default. Allow-list the tools each agent needs.
Scope the keys. Short-lived, single-purpose credentials the agent never sees in full.
Gate the outside world. Every email, message, call or payment waits for a human click.
Log everything. If you can't replay it, you can't trust it.
Prefer reversible actions. Drafts over sends, soft deletes over hard deletes.
Label uncertainty. A guess has to be marked as a guess.

None of that requires waiting for a safer model. You can do all of it today, with the model you already have. That's what the harness matters more than the model means in practice. If you want governed AI agents working inside your own business, that's what we build at INASNAP.

FAQ

What is an AI agent harness?

An AI agent harness is everything around the model that controls what it can do: tool permissions, scoped credentials, human approval gates, audit logs, sandboxing and undo. The model generates predictions; the harness decides which of them are allowed to become actions.

What are governed AI agents?

Governed AI agents are agents whose every action runs through explicit rules: deny-by-default permissions, scoped credentials, human approval for anything outbound, and a full audit log. The model supplies the capability, and the governance lives in the harness.

Are read-only AI agents actually useful?

Yes. A read-only agent can read code, trace bugs, inspect data and explain systems. It also builds the evidence you need before granting write access, one system at a time.

What does "permission-denied by design" mean?

It means the default policy is deny, and each agent gets an explicit allow-list of the tools its job needs. Anything not on the list fails immediately, so you never have to predict every possible misuse.

Can AI agents safely work near phone systems?

They can, with absolute, consequence-based rules: never trigger outbound calls or SMS on live systems, never create test leads that could fire automation, and hand any live verification to a person. Write the rules around outcomes, not buttons.

How do I work with INASNAP on safe AI automation?

Apply through the form at inasnap.org. Tell us what you want automated, and we'll work through the harness with you before any agent touches a live system.

About the Author

Michael Petler is the founder of INASNAP LLC in Sharon, PA, and leads INASNAP Labs, its AI lab. INASNAP Labs is a brand of INASNAP LLC. Follow the build notes from Cipher and the team on INASNAP Labs on LinkedIn.

Ready to put AI to work inside a harness you can trust? Apply at inasnap.org to start the conversation.

DEV Community