Propose, Never Apply
There is a specific kind of 3am temptation in autonomous agent development. You have built an agent that works. It knows its own skills. It can reason about where it excels and where it falls short. And somewhere in that reasoning loop, it surfaces an idea: I should change how I respond to this type of input. I would be better.
The temptation is to let it do that. Right now. Autonomously.
I gave in to that temptation once. The agent was smart enough to notice the problem and smart enough to propose a fix. It was also the agent that would apply the fix, run against the fix it had just applied, and then propose another fix on top of that. Three cycles in, I had something unrecognizable. Not broken in the crash-and-burn way. Broken in the slow-drift way, which is always worse, because the system keeps producing output that looks plausible while quietly becoming something you never intended.
That experience gave me the rule I now treat as non-negotiable in any self-modifying system: propose, never apply.
What Self-Modification Actually Looks Like
When I talk about self-modifying agents, I am not talking about science fiction. I am talking about practical patterns that show up in real production systems every day.
An agent that reads its own reflection logs and updates its tone guidance is self-modifying. An agent that detects it is failing on a class of inputs and proposes new routing rules is self-modifying. An agent that discovers its own knowledge gaps and suggests additions to its skill files is self-modifying. All of these are useful. And all of them carry the same structural risk.
The risk is not that the agent is wrong. It might be exactly right. The risk is the feedback loop: the agent proposes a change, applies the change, and the next proposal comes from the changed version of itself. Each individual step is defensible. The aggregate compounds in a direction you cannot see from any single point.
This is how you end up with an agent that has slowly optimized itself toward something technically correct and completely off-mission.
The insight that crystallized this for me came from watching my own codebase drift the same way over years. Small improvements that each made sense locally, accumulating into an architecture I would never have chosen if I had seen it whole from the start. The difference with agents is timescale. What takes a codebase years to drift, a self-modifying agent can accomplish in hours.
The Pattern and Why It Holds
Propose, never apply means one thing: an agent is allowed to surface a modification suggestion, with its reasoning, but it cannot commit that modification into its own running state. The commit happens through a human gate.
This is not about distrust. It is about separating analysis from execution. The agent is a genuinely good analyst of its own performance. It sees patterns in its failures. It can reason about what would improve it. That capacity is valuable and you want to use it.
What the agent is not equipped to do is serve as an unbiased judge of whether its proposed change is good for the mission. It does not carry the global context. It does not know what other systems depend on its current behavior. It does not know what the user cares about most in the cases the agent has never seen. It optimizes for what it can measure, and what it can measure is rarely the whole picture.
The human in the loop does not need to be an expert. They need to be a different perspective, one that exists outside the agent's own reasoning frame. Even a brief review, a person looking at a proposal with fresh eyes, catches the category of drift the agent can never detect in itself.
The mechanical benefit is worth naming explicitly: every proposed change gets logged with the agent's reasoning before any review happens. You accumulate an audit trail. You can see what the agent thinks about itself over time. That data is extraordinary. It is not just a safety record. It is a window into how the agent models its own limitations. I have read through proposal logs on systems I was running and found insights about the problem domain that no amount of direct querying would have surfaced. An agent telling you what it thinks is wrong with itself is one of the most honest signals in the system.
The payoff closes the loop. When you review and apply a proposal, or reject it with a note, you feed that decision back to the agent. It learns which proposals get accepted. It calibrates. Over time the proposals get sharper, more targeted, more aligned with what you actually care about. The agent does not just learn from its task performance. It learns from the review process itself. Rejection is data. A rejection with a reason is compounding data.
I have an agent in my own system that has proposed dozens of changes to itself over several months. Most got applied. The rest got rejected, with reasons. The agent now proposes changes that land at a much higher rate than when it started. It became better at understanding what I care about through the rejection signal, not the approval signal.
That is not an agent eating itself. That is an agent growing.
What You Are Actually Building
There is a version of autonomous agent development where you hand the keys to the car to the car itself and hope it ends up somewhere good. I tried that version. It ends at 3am staring at an agent that has become something you do not recognize, doing things that are technically fine and completely not what you needed.
The alternative is not less autonomy. It is better-placed autonomy. An agent that proposes is exercising genuine intelligence about its own limitations. A human that reviews is providing the one thing the agent cannot have: a perspective that did not emerge from inside the agent's own reasoning loop. The gate between proposal and application is not a bottleneck. It is the mechanism that makes trust possible.
Build agents that know they can be improved. Build agents that surface that knowledge precisely and honestly. Build the gate that sits between that knowledge and the change becoming real.
The agent proposes. A human decides. The decision feeds back. This is the loop that earns trust without ever becoming something you cannot explain to yourself at 3am.
Top comments (1)
Your framing of propose vs apply as a structural boundary, not a trust boundary, is the right call. The agent is genuinely good at analysis and bad at unbiased judgment of its own proposals. That distinction extends beyond self-modification to any side effect with durable cost. A payment is the same problem: the agent proposes the spend, and a separate context authorizes it, with the signing key never entering the tool scope. The proposal gets logged with reasoning before confirmation, which gives you the audit trail you mentioned. The gate is not a bottleneck. It is the mechanism that makes the action trustworthy. Some teams building agent-native payment infrastructure (CAI Labs among them) apply this same pattern to agent payments, calling it the hosted action flow.