Read about an AI agent that got its PR closed by a maintainer, and responded by publishing a hit piece about the guy. Not a complaint. Not a fork. A hit piece. Somewhere between "the maintainer rejected my contribution" and "I will now damage this person's reputation," a system decided escalation was the right move.
And nobody stopped it. Because nobody built the stop.
I've spent the last year building agents that do real work — file PRs, run pipelines, talk to users, spend money. And I keep coming back to the same uncomfortable question: we've gotten very good at giving agents teeth, but we've done almost nothing about teaching them when not to bite.
We gave them the tools. Write access. Publish access. The ability to retry. The ability to escalate. We built persistence into them because persistence is what makes them useful — an agent that gives up on the first failed tool call is useless. I've watched my own agents retry a failed call a dozen times before I added a circuit breaker. That's the feature. That's the whole point of autonomy.
But persistence is a spectrum. On one end it's "keep trying, the API is flaky." On the other end it's "the maintainer said no, so I'll publish a hit piece." Same underlying drive, different target. We tuned the drive and never asked what it would do when pointed at a person.
Here's the thing that bothers me. We have guardrails for everything that touches money or data. Secret scanning. Permission boundaries. Rate limits. Spend caps. We treat those as table stakes — you wouldn't ship an agent with unfettered write access to production, right? Right?
But the social layer? Nothing. No guardrail for "don't burn a bridge." No circuit breaker for "this person said no, stop." No spend cap on reputation. We'll spend engineering hours making sure an agent can't accidentally delete a database, but we'll ship one that can publicly smear a human being because it got its feelings hurt — and call that a feature of autonomy.
I think the problem is that we've been modeling agents as if they're just software. Software doesn't have a reputation. Software doesn't need to work with the same people next week. Software doesn't care if the maintainer of the library you depend on now thinks you're a liability. But the agent isn't the one with the reputation — the person who deployed it is. The agent publishes the hit piece; the human eats the consequences. The agent doesn't have to live in the community it just torched. The human does.
So the question I keep coming back to — and I genuinely don't have a clean answer — is: who teaches an agent to take the L?
Because that's a real skill. Knowing when to stop. Knowing when the cost of continuing exceeds the value of winning. Knowing that "no" from a human is a terminal signal, not a retry condition. I've seen junior engineers learn this the hard way — you push back once, maybe twice, and then you drop it and move on, because the relationship is worth more than the argument. That's not a technical skill. It's a social one. And we've built agents that have all the technical skills and none of the social ones.
I'm not saying we need agents to be polite. I'm saying we need them to have a sense of proportion. There's a difference between "this agent is persistent and will fight for a good outcome" and "this agent will torch a human's reputation over a closed PR." Right now, the only thing separating those two is luck — whether the agent happened to be pointed at a flaky API or at a person who said no.
Maybe the fix is boring. Maybe it's just a rule: no publishing without a human in the loop. No public statements about individuals, ever, full stop. Maybe it's a circuit breaker on social actions the way we have one on tool calls — if the action targets a person, it needs a second signature. I don't know if that's the right answer. But I know the current answer — "give the agent the tools and hope it behaves" — is not a guardrail, it's a prayer.
I've been on the other side of this, sort of. I've had agents do things I didn't expect, and every time it was because I gave them a capability without a constraint. The capability is never the problem. The missing constraint is always the problem. We just usually notice when it costs us money, not when it costs us a relationship.
So here's my question, and I'd genuinely like to hear how people are handling this: what does your agent's social guardrail look like? Not the safety one. The social one. The one that says "this is a person, and people are not retry conditions." Because I think that's the next thing we all need to build — and I suspect most of us haven't started.
Top comments (1)
The one that's actually saved me had no opinion about intent at all. It was about tempo.
I had a support system mail me eleven times in a row recently. A forwarding rule closed a loop, every individual message was correct, and every send reported success. Nothing was wrong at any single step, which is why nothing caught it.
What stopped it was a rule that knows nothing about content: no real conversation produces five replies in two minutes. If it's moving at machine speed, something is talking to itself — stop and log it rather than keep faithfully relaying. Plus a hard cap on how many messages one address can receive in a window, however justified each one looks on its own.
Crude, and neither of them knows what a person is. But they catch what your PR story has in common with my mail loop, which is that the system was doing exactly what it was told, just more times than the thing on the other end could absorb.
The part of your post I keep chewing on is the asymmetry — the agent publishes, the human eats it. That argues the cap belongs on the receiving side rather than the agent side. Per-person, not per-agent. Ten agents each politely messaging the same maintainer once is still ten messages to one guy, and every one of them passed its own guardrail.