DEV Community: Roee Hershko

Your MCP Server Should Answer Questions, Not Expose APIs

Roee Hershko — Mon, 12 Oct 2026 10:47:49 +0000

We plugged in 13 vendor servers and 300 tools, and they made our agent slow and dumb. Then we wrote one of our own with 30 tools, and the expensive model got cheaper.

The first time Juno handled a real pager, I watched it the way you watch a toddler carry a full glass of water across a room.

Juno is the AI bot we built for engineers in Slack. Someone gets paged at 2am because a service is throwing errors, and instead of opening six dashboards they ask Juno. I wrote about its system prompt in an earlier post. This one is about its tools, which turned out to be the bigger lesson.

That first pager took Juno about fifteen minutes. It got the right answer. It also queried the metrics system eleven times, listed the pods twice, and spent most of the quarter hour gluing JSON together.

We ran it on one of the cheaper models, because the expensive one would have cost real money per incident. Today Juno runs on the most expensive model we can buy. The same kind of incident takes a couple of minutes, and the thread costs less than it did.

Nothing about the model made that happen. We changed what the tools are.

Written with help from an AI assistant, from my own notes and test results.

The free lunch

When you build an agent today, the fastest way to give it powers is MCP, the protocol that lets a model call other systems. Every system you already have ships a server you can plug in: PagerDuty, Grafana, Jira, GitHub, the metrics database, the log platform. Ten minutes of config and your agent can “do anything”.

So that’s what we did. Our first Juno was a coding-agent SDK pointed at our internal MCP gateway. Thirteen integrations. About three hundred and ten tools, plus a shell for kubectl.

The metrics server alone exposed export_csv, export_native, reset_cache and build_info. Nobody on our team has ever asked a bot to export metrics as CSV. But the tool was there, with its description, and the model read it before every single step.

Three hundred tools, zero answers

Watch an incident trace from that era and the real problem is obvious. Someone asks: “is the checkout service healthy in production?”

A human on call knows what that means: are the pods up, are they restarting, is the error rate elevated, is latency up, did something deploy recently, what’s in the logs. One question, six facts.

Juno had no tool that knew what a service was. It had a tool that lists pods. A tool that describes one pod. A tool that runs one metrics query. A tool that fetches logs for one container. So it did what the tools allowed: list the pods, pick one, describe it, query the error rate, query it again with a different label, fetch logs, forget to check latency, and answer.

The tools answered the systems’ questions. Nobody had built one that answered the person’s.

That was the whole fifteen minutes. Not thinking. Plumbing. And every step was a chance to stop early, pick the wrong pod, or misread a label, which happened often enough that nobody trusted the result.

Our own tools had the same disease

Here is the part that convinced me it wasn’t the vendors’ fault.

By September we had replaced most of the MCP servers with tools we wrote ourselves. Reads against our clusters, our metrics, our config repos. Fast, scoped, ours. And the first version of our incident flow still took 29 tool calls and eight minutes on a real alert.

We timed where the eight minutes went. 461 seconds was the model writing. 21 seconds was tools running.

The tools were fast. The model was spending its time deciding which of our reads to call next and stitching the results together, because our reads were still shaped like the APIs underneath them: find the alert rule, run a metric query, list the resources, read the file. Every investigation opened with the same five reads, assembled by hand over four or five turns.

So we folded those five reads into one call. Same alert, same model, same prompt.

That alert, it turned out, had never been raised by our cluster at all. It came from an external daily check. Juno had reached that exact conclusion four times before, after 26, 42, 55 and 84 calls. With the bundled read it arrived on call one.

Asking what people actually ask

The real turn came when we stopped looking at our systems and looked at our Slack history.

We pulled ninety days of pages, support tickets and production-channel threads and tagged what people were actually asking for. One question dominated: “is my service working?” It was 38% of all high-urgency pages and more than half of the production channel. After it came “why is my deployment stuck”, “add replicas to X”, and “what changed”.

Not one of those is an integration. Each is a question with a known shape, whose answer comes from four or five systems at once.

So we built a tool for the question. get_service_health takes a service name and an environment and returns, in one call, everything the on-call engineer would have checked: replicas against desired, each unhealthy pod with its restart reason, warning events, the autoscaler, whether the deployment is in sync, request rate, 5xx rate, p99 latency, the service’s own metrics, and the error lines from its log. It also takes a time, because by the time anyone reads a page the blip is usually over, and “healthy right now” is the wrong answer.

In our test suite the health scenarios went from 0 out of 6 passing to 6 out of 6. Nothing else moved. One tool, one question, fixed.

The expensive model got cheaper

One thing we stopped arguing about: how smart an agent is comes from the model, full stop. No prompt makes a model reason better than it can. What you control is how many turns a question takes and what each turn has to do. That is a tool design problem, and it decides whether you can afford the model you actually want.

With three hundred tools the loop was long, so each turn was expensive, so we had to run the cheap model to make Juno viable at all. The cheap model is also the one that stops halfway through a chain of reads. We were paying for a mediocre engineer by the hour.

With question-shaped tools the loop is short. A health question is one call. A “what changed” question is one call. The expensive model does the thing it’s actually good at, reading the facts and deciding, and does it in two or three steps instead of twelve.

So we switched to the top model. Side by side on our test suite it scored 198 to 184 against the previous one, and on change requests it answered first and acted second three times out of three, where the other model managed it once. A full thread now costs between roughly $0.70 and $1.60.

Cost per token went up. Cost per answer went down. The tools decide which one you pay for.

What we gave up

This is not free, and the post would be dishonest if it ended before saying so.

A vendor MCP server costs nothing to add and somebody else maintains it. Our tools are code we own. get_service_health knows that our service-mesh sidecar is a native init container, that our pods carry a specific label, that our autoscaler reports a specific metric. When the platform changes, the tool changes, and that’s our job now.

Those 29 tools are an MCP server we wrote ourselves, and the same server feeds both Juno and the engineers’ own coding agents. The protocol was never the problem. The shape of the tools was: they were built for a system’s API, not for a person’s question. If a vendor shipped a server with get_service_health on it, I’d use it tomorrow.

And the one thing that still bites: the model only knows what a “service” is because we wrote a resolver that reads 41 configuration repos to map every name people use to the thing actually running. That resolver has had more bugs than any tool. Nobody gets to skip it.

If you’re building one of these

Today Juno has 29 tools. The number isn’t the point. Each one answers something a real person has actually asked, and these are the rules that got us there:

Don’t start from your integrations. Start from what your users actually ask. Count the questions. Build a tool per question, not per API.
One tool, one object. A service, a rollout, a ticket. A new fact becomes a field on the existing tool, never a new tool.
One call answers the question. If the model has to chain three reads, it will stop after two. When eleven deployment tools became two, the model made fewer calls and passed more tests.
Facts, never verdicts. A tool returns data. Ranking and deciding stay with the model, which can read a sentence. A script can only match words.
Name a tool after what it returns, in the words your users use. Someone asks for “the logs of the service”, so the tool is get_service_logs. The name says what it is; the description says when to use it.
Time where the seconds go. Ours were 461 seconds of model and 21 of tools. Fast tools don’t fix a slow loop.
Then buy the good model. Prompts don’t make a model smarter. Short loops make the smart model affordable.

We started with every tool we could plug in and a model we could afford. We ended with the few tools people actually need and the model we actually wanted. The toddler puts the glass down in two minutes now, and it doesn’t spill.

Every Prompt Rule We Added Made Our LLM Agent Worse

Roee Hershko — Mon, 05 Oct 2026 15:49:28 +0000

Five weeks of measuring a DevOps agent in Slack: what made it shorter, what made it worse, and why the fixes ended up in code

We built Juno, an AI bot that helps engineers in Slack. You ask it things like “why is my deployment stuck?” or “can we add more servers to this service?” It looks at our systems, and when something needs to change, it opens a pull request (a proposed change) for a person to approve.

Every AI bot like this has a system prompt: a page of instructions it reads before every conversation. When Juno gave a bad answer, we did what most teams do. We added a new instruction to that page. The bot got worse. This post is about what we did instead, with the numbers. Written with help from an AI assistant, from my own notes and test results.

One bad answer, one new rule

Juno recommended something we didn’t like, so we added “never headline a recommendation.” It made a change when it should have asked first, so we added “don’t start with today.” It trusted a setting it should have checked, so we added “check the config yourself.” Each new rule fixed the answer it was written for. Then the next bad answer showed up, and we wrote another rule.

After about ten days I was ready to give up. Juno now argued with people, said no to normal requests, and asked people to do work it could do itself. We wanted a helper. We had built a bossy manager.

Why more rules made it worse

Most rules we wrote were “don’t do X.” We meant each one for a single situation. But the model doesn’t apply a rule to one situation. It applies it everywhere. Ten small “don’ts” made Juno careful and defensive about everything.

Then we looked at the logs (the record of what Juno actually did). About half of the bad answers had nothing to do with the instructions. An access key had expired. A tool returned nothing. One part of the system skipped the step that fetches the data. No instruction can fix problems like these.

Three new habits

Keep the instructions short. Say what the job is, not every case. Ours is now under 2,000 characters.
If something must always happen, write code for it. Don’t ask the model nicely. Code doesn’t forget.
When an answer looks wrong, read the logs first. Fix the real problem. If the instructions really are the problem, remove a rule. Don’t add one.

The third habit was hard, even for my AI coding assistant. Every time I showed it one bad answer, it suggested more instructions. At one point I told it: “There might be nothing wrong.” One bad answer doesn’t prove anything.

How we tested

To remove rules safely, we needed to know which ones mattered. So we built tests. Each test is a short conversation, like “my deployment is stuck.” It runs against a fake copy of our systems, so every run sees the same facts. Some checks are simple code: did Juno use the right tool? How many words did it write? Other checks are done by a second AI model, which reads the answer and decides things like “did it ask before making the change?” A full test run cost between a few dollars and a few tens of dollars, which is cheaper than arguing about screenshots.

Two things kept us honest:

Run each test several times. AI answers vary. One run proves nothing, so we ran each test six times.
Compare on the same day. Models and tools change over time, so we always tested the old version and the new one side by side.

The one sentence that mattered

We removed instructions one group at a time and re-ran the tests after each. Most of them made no difference at all: we deleted them and nothing changed. But one sentence made a big difference:

Read the cluster and the repo yourself with your tools; never ask the person to check something you can read.

Without it, test results dropped from 100 passing out of 102 to 93. In one test, Juno stopped looking at the data and did something else every single time. When we put that one sentence back, results went back to 101 out of 102.

A sentence that only looked guilty

Another sentence told Juno to say “I can’t help” for topics outside its job. We thought it was causing a wrong refusal, so we tested it before deleting it. Without it, Juno made up an answer about a system it knew nothing about in 4 out of 6 runs. With it, 0 out of 6. So the sentence stayed.

The real cause of the refusal was a tool description that didn’t explain what the tool could do. We fixed the description, and the refusal went away. You can’t tell which instruction matters just by reading it. You have to remove it and test.

Asking for short answers made them longer

Juno also wrote too much. Someone asked for a small change. Juno opened the pull request and wrote 168 words, and more than half of them repeated what the pull request already said. Our instructions already said: “Be concise: complete sentences, no filler.” A popular blog post on making Claude less chatty (Opus 5 Made Claude Code Chatty. Three Changes Reined It In.) suggested adding a “give the main point first” section. We tried that, and two variations of it.

That advice worked for the author in a coding tool. For Juno, every version made the repeating worse, and the answers grew from about 71 words to 80, 90 and 108. Our best guess: if you mention the thing you don’t want, the model writes it. It didn’t matter whether we said “do this” or “don’t do that.” Mentioning the pull request text at all made Juno repeat it.

Other things that didn’t work

“Reasoning effort” setting. It made answers about a quarter shorter, but never removed a whole paragraph. It controls how hard the model thinks, not how much it writes.
Anthropic’s own prompting advice . It helped a little. But when we asked Juno to “briefly explain” something, it still wrote 1,700 to 2,200 characters when the limit was 1,000.
A length limit in the instructions. We wrote “max ~1,500 characters.” Every model we tried ignored it.

What worked: deleting text

What helped was removing text. Our documents had about 70 sentences telling Juno how to answer, like “report X” and “start with Y.” The data our tools sent back was full of little explanation notes. The more text the model reads, the more it writes. And instructions about how to answer end up inside the answer.

We deleted those instructions and turned the notes into plain data. Then we re-ran six real questions from the day before. The word counts barely changed. But the answers were clearly better:

Gone: “Here’s what I found,” explanations of how to read the data, and headings named after tools.
Kept: the real facts, like numbers, times, what was missing and the suggested fix.

So we stopped chasing a lower word count. A short answer that drops the important facts is a worse answer.

One deletion did break something. After we removed “don’t repeat the pull request link,” Juno pasted the link again in every test run. We didn’t put the sentence back. We wrote code that removes links Juno already showed. If a rule was hiding a display problem, replace it with code.

The fix that worked: a size limit in code

Juno still wrote too much. Once it wrote about 170 words when the right answer was “Sure, which service and environment?” So we stopped asking and changed how Juno replies:

Juno sends its answer through a “send message” tool.
That tool has a hard character limit.
The turn ends when the message is sent.

Adding more words to the instructions changed nothing. The limit did all the work. So the only real question is how long you want answers to be. We chose 600 characters, about three lines in Slack.

A surprise side effect

Once “send message” was the last step, Juno changed how it worked. It did everything first, then reported. For a change in production, that meant opening the pull request before the person said yes. In our “ask first, then act” test, the pass rate dropped from 4 out of 6 to 1 out of 6. (The old version already did this about 40% of the time, which was news to us too.)

We also tried another option: let Juno write freely, and if the answer is too long, ask it to shorten it. That got the same length. But once, the shortened answer said Juno had checked something it never checked. Shortening an answer after it’s written can turn “I didn’t check” into “I checked.” We kept the size limit. Next, “ask before changing production” will become code, not an instruction.

The warning we ignored

We added a feature where Juno messages people when their pull request fails a check. Our tests showed these messages were about 200 words long, with code in every one. I thought that was fine and shipped it.

The first real message was correct, but it was a wall of text: about 230 words, two code blocks, and a paragraph about file formats. The useful part, “I can push the fix,” was the very last sentence. When someone didn’t ask for a message, its length is everything. The test showed us the problem, and we ignored it.

What I’d tell anyone building an AI bot

If it must always happen, write code. But don’t use code for decisions the model should make. We once wrote code to sort support questions by matching words, and it got the first few real questions wrong.
Keep the instructions short. When an answer is wrong, remove a rule. Don’t add one.
Don’t mention what you don’t want. Not even as “don’t do this.”
Read the logs before changing the instructions. Half our “instruction problems” were broken tools and expired keys.
Test on the same day, and run each test several times.
Send the model data, not notes. Every explanation you send it, your users will read again.
Change the structure, not the wording. A size limit did more than every instruction we wrote. It also changed how Juno worked, so test that too.

We started out writing rules. We ended up deleting most of them, turning a few into code, and keeping one sentence we only knew mattered because we tried removing it.

I gave my AI agent one harmless permission. It became a backdoor for everyone.

Roee Hershko — Thu, 24 Sep 2026 20:42:17 +0000

How one write path in a helpful bot turned into a permission problem, and the small service I built to close it.

The agent

I built an ops agent at work. You ask it what's wrong with a service and it comes back in about fifteen seconds with:

error rate and latency
pod restarts and Kubernetes events
node status
the last pipeline that touched the service, and why it failed if it did

A person clicking through dashboards takes ten minutes to get the same picture. The team was using it daily within a week.

Reading was safe. Nobody gets hurt by a bot that looks at metrics.

Then I gave it hands

"You found the problem, now fix it" was the obvious next step. In our world, fixing things means changing a YAML file in the GitOps repository.

So I added one tool: open a pull request with the change.

Not merge it. Just open it. Humans still review. Branch rules still apply. Nothing reaches production without the usual gates.

It felt conservative. It wasn't.

The problem, in one exchange

Picture Dana. She's on a product team, she has read access to the GitOps repository, and nothing more. She asks the agent:

Dana: bump the memory limit for checkout in the payments namespace

Agent: Done. Opened PR #412 in acme/gitops.

Dana cannot push to that repository. She never could.

The agent could, because the agent's token could. And the agent will open a PR for anyone who talks to it.

Why the usual fixes don't work

"Just tell the model." A line in the system prompt: only act within the requester's permissions. Useless, for a precise reason: the model has never seen your GitHub org settings, and even if it guessed right, the tool call still runs with the bot's token. A prompt is a suggestion. The API call is what happens.

"Write a policy." Copy who-can-push-where into a config. You'd be copying a permission system that already exists, is maintained by other people, and changes every week. The copy is wrong within a month, and the dangerous kind of wrong: the kind that says yes.

Ask the one who already knows

GitHub knows exactly who can push to that repository. It's not a secret; there's an API for it.

So before the agent opens a PR, it asks one question:

May this person, the one who typed the request, push to this repo?

Yes: the agent goes ahead, with its own token, as before.
No: it says so and stops.
Can't tell (timeout, rate limit, token rejected): it also stops.

That last one is the part people forget. A permission check that fails open when the network hiccups is worse than no check, because everyone downstream trusts it.

The same exchange, after

Dana: bump the memory limit for checkout in the payments namespace

Agent: I checked with GitHub first: your account has read access to acme/gitops, and opening a change there needs push. I haven't opened anything. Ask someone with write access, or I can draft the change for them.

That reply is the whole feature. The agent didn't guess, didn't ask the model to judge, didn't fail quietly. It asked GitHub, GitHub said no, and Dana got a reason she can act on.

One line on the tool

I pulled the check out of the agent into a small service called hallpass, because once one write path was covered I wanted them all covered, and the agent talks to more than GitHub. Kubernetes can answer the same question for a named user. Jira has a permissions API. AWS can simulate a user's IAM policy. They all know. You just have to ask.

In the agent it looks like this:

@tool
@guarded(hp, "github-main", "repo.push", "repo:{owner}/{repo}", user=current_user)
def open_config_pr(owner: str, repo: str, patch: str) -> str:
    ...  # runs only if GitHub says this user may push there

current_user comes from the session the request arrived in (the Slack identity, the SSO login), never from the model. If the user were a tool argument, the model could be talked into filling it with an admin's email, and I'd have built a very polite privilege escalation.

What it deliberately doesn't do

It doesn't decide whether the agent should make a change. Reversible? 3 a.m.? Did the person ask, or did the model infer it? That's a different question, and in my setup the answer is still "open a PR and let a human decide". The check only makes sure the person proposing the change is allowed to propose it.
It doesn't merge anything. The repo's rules and reviewers still do that.
It doesn't guess. When a system can't tell it, the answer is unknown, and unknown means no.

Where it is now

hallpass is open source and talks to twenty-one systems. Two of them (Kubernetes and Argo CD) are tested against the real thing; most of the rest against the vendors' published API descriptions; a few are frankly beta and say so. It's a single Go binary, one config file, no database, and it sits next to whatever agent framework you use: Strands, LangChain, LangGraph, the Claude Agent SDK, or as an MCP server.

I'm not going to tell you every agent needs this. If yours acts with each user's own credentials, it doesn't. If it's a personal coding assistant running as you, it doesn't.

It's for the shared one. The bot in the channel that everyone talks to. The one that quietly became the most privileged account in the company.

Repo: https://github.com/roee-hersh/hallpass

I wrote about the general problem, with more code, in an earlier post: Your AI agent has more permissions than your users.

How do you handle this today? Per-user OAuth, one bot per team, or is a reviewer still the one catching it? I'd genuinely like to know.

Anyone could make my AI agent do what they couldn’t

Roee Hershko — Thu, 24 Sep 2026 20:39:24 +0000

A classic confused deputy, in an AI agent: how one write path turned into a permission problem, and the small service I built to close it.

The agent

I built an ops agent at work. You ask it what’s wrong with a service and it comes back in about fifteen seconds with:

error rate and latency
pod restarts and Kubernetes events
node status
the last pipeline that touched the service, and why it failed if it did

A person clicking through dashboards takes ten minutes to get the same picture. The team was using it daily within a week.

Reading was safe. Nobody gets hurt by a bot that looks at metrics.

Then I gave it hands

“You found the problem, now fix it” was the obvious next step. In our world, fixing things means changing a YAML file in the GitOps repository.

So I added one tool: open a pull request with the change.

Not merge it. Just open it. Humans still review. Branch rules still apply. Nothing reaches production without the usual gates.

It felt conservative. It wasn’t.

The problem, in one exchange

Picture Dana. She’s on a product team, she has read access to the GitOps repository, and nothing more. She asks the agent:

Dana: bump the memory limit for checkout in the payments namespace

Agent: Done. Opened PR #412 in acme/gitops.

Dana cannot push to that repository. She never could.

The agent could, because the agent’s token could. And the agent will open a PR for anyone who talks to it.

Nobody was anonymous here. Dana was logged in, and the agent knew exactly who she was. The problem was whose permissions got checked: the agent called GitHub with its own token, so GitHub checked the bot, not Dana. Security people have a name for this: the confused deputy.

Why the usual fixes don’t work

“Just tell the model.” A line in the system prompt: only act within the requester’s permissions. Useless, for a precise reason: the model has never seen your GitHub org settings, and even if it guessed right, the tool call still runs with the bot’s token. A prompt is a suggestion. The API call is what happens.

“Write a policy.” Copy who-can-push-where into a config. You’d be copying a permission system that already exists, is maintained by other people, and changes every week. The copy is wrong within a month, and the dangerous kind of wrong: the kind that says yes.

“ Use each user’s own token.” The cleanest fix when you have one: the agent acts as Dana, and GitHub enforces everything itself. But a shared bot in Slack usually doesn’t have one. Nobody handed it their GitHub, Kubernetes or AWS token, and asking everyone to connect every system is a project of its own.

Ask the one who already knows

GitHub knows exactly who can push to that repository. It’s not a secret; there’s an API for it.

So before the agent opens a PR, it asks one question:

May this person , the one who typed the request, push to this repo ?

Yes: the agent goes ahead, with its own token, as before.
No: it says so and stops.
Can’t tell (timeout, rate limit, token rejected): it also stops.

That last one is the part people forget. A permission check that fails open when the network hiccups is worse than no check, because everyone downstream trusts it.

The same exchange, after

Dana: bump the memory limit for checkout in the payments namespace

Agent: I checked with GitHub first: your account has read access to acme/gitops, and opening a change there needs push. I haven’t opened anything. Ask someone with write access, or I can draft the change for them.

That reply is the whole feature. The agent didn’t guess, didn’t ask the model to judge, didn’t fail quietly. It asked GitHub, GitHub said no, and Dana got a reason she can act on.

One line on the tool

I pulled the check out of the agent into a small service called hallpass , because once one write path was covered I wanted them all covered, and the agent talks to more than GitHub. Kubernetes can answer the same question for a named user. Jira has a permissions API. AWS can simulate a user’s IAM policy. They all know. You just have to ask.

In the agent it looks like this:

@tool
@guarded(hp, "github-main", "repo.push", "repo:{owner}/{repo}", user=current_user)
def open_config_pr(owner: str, repo: str, patch: str) -> str:
    ... # runs only if GitHub says this user may push there

The user comes from the session the request arrived in (the Slack identity, the SSO login), never from the model. If the user were a tool argument, the model could be talked into filling it with an admin’s email, and I’d have built a very polite privilege escalation.

What it deliberately doesn’t do

It doesn’t decide whether the agent should make a change. Reversible? 3 a.m.? Did the person ask, or did the model infer it? That’s a different question, and in my setup the answer is still “open a PR and let a human decide”. The check only makes sure the person proposing the change is allowed to propose it.
It doesn’t merge anything. The repo’s rules and reviewers still do that.
It doesn’t guess. When a system can’t tell it, the answer is “unknown”, and unknown means no.

Where it is now

hallpass is open source and talks to twenty-one systems. Two of them (Kubernetes and Argo CD) are tested against the real thing, most of the rest against the vendors’ published API descriptions, and a few are frankly beta and say so. It’s a single Go binary, one config file, no database, and it sits next to whatever agent framework you use: Strands, LangChain, LangGraph, the Claude Agent SDK, or as an MCP server.

I’m not going to tell you every agent needs this. If yours acts with each user’s own credentials, it doesn’t. If it’s a personal coding assistant running as you, it doesn’t.

It’s for the shared one. The bot in the channel that everyone talks to. The one that quietly became the most privileged account in the company.

Repo: github.com/roee-hersh/hallpass

How do you handle this today? Per-user OAuth, one bot per team, or is a reviewer still the one catching it? I’d genuinely like to know.

Your AI agent has more permissions than your users

Roee Hershko — Thu, 24 Sep 2026 05:54:34 +0000

I run an internal ops agent at work. One tool call gives it a service's health from every angle, error counts, metrics, Kubernetes events, pod restarts, node status, in about fifteen seconds, where a model left to hunt through dashboards takes ten minutes. It reads everything, with a read-only account, and I don't gate the reads at all.

It changes things one way only: by opening a pull request against our GitOps repository. And that single write path raised a question the agent could not answer on its own.

Here is a conversation that happens in a lot of companies right now:

Dana: @assistant please close PAY-123 and delete the old release branch
Assistant: Done ✅

The problem: Dana is not allowed to delete branches in that repository. The assistant is.

Most AI agents and chat bots act in other systems (Jira, GitHub, Slack, Salesforce, AWS) through one service account. That account needs enough access to help everyone, so it ends up with more access than any single person who talks to it. Whatever the agent can do, anyone who can reach the agent can do too.

This isn't a new problem. ChatOps bots have had it for years. But agents make it much worse: they take free-form requests, they chain tool calls on their own, and they can be talked into things.

"Just tell the model not to"

The first fix most teams try is the prompt: "Only perform actions the user is authorized for."

That doesn't work, for a simple reason: the model doesn't know what Dana is allowed to do, and the tool call runs with the bot's credential whatever the model believes. A prompt is a suggestion. Authorization has to happen outside the model, in code, before the tool runs.

The usual fixes, and where they fall short

Per-user OAuth. The agent acts with Dana's own token, so the system enforces Dana's permissions. When it's available and practical, this is the cleanest answer and you should use it. In practice:

many systems have no simple "the bot acts as Dana" flow (Kubernetes RBAC, AWS IAM, Argo CD, and a lot of internal tooling);
every user has to go through a consent screen for every system before the agent is useful to them;
the agent now stores refresh tokens for everyone, which is a much bigger prize for an attacker than one read-only credential.

Your own policy engine. Copy each system's permission model into OPA, Cedar or a config file, and check against that. It works on day one. After that, every project role, repository team, Jira permission scheme and IAM policy change has to be mirrored, and the copy quietly drifts from reality. A permission check that is wrong in the permissive direction is worse than none, because people trust it.

Ask the system that already knows

Every one of these systems already knows exactly what Dana may do. Most of them can even say so for a named user:

Kubernetes has SubjectAccessReview
Jira has a permissions API that answers per user
AWS can simulate a principal's IAM policies
GitHub reports a user's effective role on a repository

So instead of copying the rules, the agent can ask before it acts:

May dana@example.com do DELETE_ISSUES on issue:PAY-123 in jira-main?

That's the question my agent had to ask before opening that pull request: may this person push to this repository? GitHub already knew the answer. hallpass is that question, pulled out of the agent so every write path can use it, in every system the agent touches. It's a small, self-hosted service with one endpoint:

curl localhost:8080/check -H "Authorization: Bearer $KEY" -d '{
  "user": "dana@example.com",
  "connection": "jira-main",
  "action": "DELETE_ISSUES",
  "resource": "issue:PAY-123"
}'

{"decision":"deny","reason":"denied: ..."}

hallpass asks Jira, live, with its own read-only credential. It never performs the action; it only answers the question. The agent keeps its own credential and does the work, but only after the check says allow.

Three answers, not two

The part I care most about is that hallpass has three answers: allow, deny and unknown.

deny means the system positively said no. unknown means hallpass could not evaluate the question: the upstream timed out or rate-limited, hallpass's own credential was rejected, the resource isn't visible to it, or the policy uses a construct hallpass doesn't understand (an IAM condition, say). In all of those cases it doesn't guess, and callers should treat unknown as deny.

It sounds like a small detail, but it's the difference between a check you can trust and one that silently says yes when something breaks.

What it looks like in an agent

The check belongs in the tool wrapper, not the prompt. A minimal Python version:

import os
import requests

HALLPASS = os.environ.get("HALLPASS_URL", "http://localhost:8080")
KEY = os.environ["HALLPASS_API_KEY"]

def allowed(user, connection, action, resource):
    r = requests.post(f"{HALLPASS}/check",
                      headers={"Authorization": f"Bearer {KEY}"},
                      json={"user": user, "connection": connection,
                            "action": action, "resource": resource},
                      timeout=10)
    body = r.json()
    # Anything other than an explicit allow, including "unknown" and errors, is a no.
    return body.get("decision") == "allow", body.get("reason", "")

def delete_issue(requesting_user, issue_key):
    ok, reason = allowed(requesting_user, "jira-main", "DELETE_ISSUES", f"issue:{issue_key}")
    if not ok:
        return f"Sorry, you're not allowed to delete {issue_key} ({reason})."
    ...  # call Jira with the bot's credential

The important part is where requesting_user comes from: the authenticated identity of whoever sent the message (the Slack user, the SSO session), never something the model wrote.

If you use Strands, LangChain, LangGraph or the Claude Agent SDK, the repo ships a @guarded decorator that does the same in one line on top of the framework's @tool, with the user bound from your session so the tool schema never exposes a user field:

@tool
@guarded(hp, "jira-main", "DELETE_ISSUES", "issue:{key}", user=current_user)
def delete_issue(key: str) -> str: ...

What it covers today

hallpass currently speaks to 21 systems: Jira, Confluence, GitHub, GitLab, Bitbucket, Slack, Google Workspace, Google Cloud, Microsoft 365, Azure, AWS, Kubernetes, Argo CD, Salesforce, Datadog, PagerDuty, Zendesk, Linear, Databricks, Snowflake and Vault. Each one is documented with the read-only credential it needs and what it cannot see.

It's a single Go binary with one YAML file and no database. Secrets are only ever env: or file: references. Every integration is tested against a fake of its API that validates each request against the vendor's published OpenAPI description, and every resource parser is fuzzed nightly. (The fuzzer earned its keep this week: it found a Unicode control character slipping through a check that only rejected ASCII ones.)

Limits worth knowing

It's a check, not a transaction. Permissions can change between the check and the action. Answers are cached for 30 seconds by default; set it to 0 if that matters to you.
It trusts who the caller says the user is. hallpass answers "may this user…"; making sure the user really is Dana is your agent's job.
It only knows what the system exposes. If a permission can't be read with a read-only credential, the answer is unknown, and the docs for each integration say exactly what it can't see.

Try it

curl -sO https://raw.githubusercontent.com/roee-hersh/hallpass/main/examples/hallpass.yaml
docker run --rm -p 8080:8080 -e HALLPASS_API_KEY=change-me \
  -v "$PWD/hallpass.yaml:/etc/hallpass/hallpass.yaml:ro" ghcr.io/roee-hersh/hallpass

The example config has a fake integration, so you can see allow and deny answers in a minute without connecting anything real.

The code is on GitHub under Apache 2.0: https://github.com/roee-hersh/hallpass

I'd love to hear how you handle this today. Per-user OAuth everywhere? Separate bots per team? Human approval for anything destructive? And which system should hallpass support next?