In the first week of September three arXiv papers landed on the same nerve:
- HarnessDev (2609.01437) asked whether LLMs can create and evolve their own agent harness.
- SafeEvolve (2609.02786) had agents co-evolve their own safety policy from experience.
- PatchBench (2609.04075) measured how well agents patch real security vulnerabilities — and found the usual proof-of-concept check inflates solve rates by 1.83×.
Read together, they say one thing: agents are starting to modify the thing that runs them. Their own tools, their own guardrails, the code they operate on.
Every one of those papers leaves the same question open. Who approves the patch?
An unreviewed self-patch is a supply-chain risk you invited in
We spent a decade learning not to curl | sh from strangers. Then we learned to pin dependencies, sign builds, and review every PR. A self-improving agent that writes a tool and runs it in the same breath undoes all of that in one step — and it does it inside your perimeter, with your credentials.
The tempting answer is "don't let agents write tools." But that throws away the capability that makes agents useful: the thing your agent needs is almost always discovered mid-task. A PDF encryptor that doesn't exist. A library it can't import. A file format nobody wired up. Freeze the toolkit and the agent either fails or — worse — fakes it.
So the real question isn't whether agents should grow. It's how the growth gets governed.
What a governed self-patch looks like (a real run, unedited)
I've spent four months on an open-source runtime called Systemu that takes one specific position on this. Last week I watched it work end-to-end on a boring, real task: "encrypt these PDFs with a password." No such tool existed.
Prefer watching? The whole loop in 60 seconds, built only from the real, unedited screenshots of the run: systemu-demo.mp4
1. The agent asks — it doesn't fail, it doesn't fake.
The running agent issues a REQUEST_HARNESS — "provision what I lack." One card lands in the Inbox:
HIGH — Forge tool: pdf_encrypt
Apply password protection to a PDF file and save the encrypted version to a new path.
What approve does: generates + reviews the tool code, then enables it.
Safe default: Skip
The safe default is Skip. Do nothing, and nothing happens.
2. You review the spec, then the code — before anything exists.
Approving the card doesn't run anything. It generates the tool and shows you the source with a plain warning: this does not prove the code safe — read it before approving. The LLM reviewer can flag problems, but it can only ever push the decision toward deny or escalate. It cannot open a hole.
3. Dependencies gate separately.
The generated tool wanted pypdf. That's its own approval. No silent pip install, ever.
4. The first run is gated too.
Even after the tool is deployed, its first execution lands as an action card marked dangerous-until-proven, with the actual arguments shown. Safe default: Deny. Re-forging the tool re-gates it.
5. Deployed, with provenance.
Minutes later the Build page shows pdf_encrypt — DEPLOYED · AGENT-BUILT · dry-run PASSED. The toolkit grew by one. Every step is in the decision ledger, attributed to the run that made it, revocable in one click.
That's the whole loop: gap → request → govern → grow.
Two ideas underneath it that I think are non-obvious
A self-requested capability is more dangerous than a pre-provisioned one. The agent chose it. So it's gated more strictly, not less — the opposite of how most frameworks treat "the agent decided it needs X."
Judgment can only downgrade toward safe. When an ambiguous request needs an LLM judge, the judge may deny or escalate, never grant beyond policy. A judge fault fails to escalation, not to grant. Your safety doesn't depend on the model being right.
Does the governance kill the capability?
That was my fear, so I measured it instead of asserting it. A Capability-Gap Benchmark: tasks that are impossible without acquiring a missing capability, across six capability families (tool, skill, file/secret access, compute budget, sub-agents, MCP servers), across 5 models from 5 vendors, graded by an external oracle rather than the system's own verifier.
- Frozen toolkit (classic push harness): 6%
- Governed pull (full Governor, human gates): 61%
- 179 trials, McNemar p = 2.8 × 10⁻¹⁰
Governance isn't the brake on self-improving agents. It's the steering. Preprint on Zenodo.
Honest limits
- The forge step is only as good as the model. Flash-tier models write noticeably worse tools.
- First install pulls a lot of dependencies; the dashboard extra is heavy.
- The evaluation is mine. I would genuinely like someone to try to break the 6% → 61% result — the trial logs are available.
- It is a solo project from Chennai. Treat it as a serious beta, not a product.
Try it
pip install "systemu[dashboard]"
systemu init && systemu start
MIT licensed. The 22-second walkthrough is at the top of the README: https://github.com/rameswaran-mohan/project-systemu
If your AI patched itself tonight — would you know?
Top comments (2)
The approval-queue framing rings true. I run a handful of cron-scheduled agents on a small VPS and the pattern that emerged was less "should the agent be allowed to grow" and more "the approval happened once, months ago, and nobody has re-read it since." The first approved tool was genuinely reviewed. The twentieth was a rubber stamp because the queue had five cards and I wanted my PDFs encrypted. So the governance question I keep running into isn't the gate itself, it's drift: the thing I approved is no longer bit-for-bit the thing that executes.
Curious how Systemu handles that layer. Is an approved forge tool pinned to a hash, so a later edit forces re-approval? Or does an approval cover the capability ("PDF encryption") and trust future implementations? The second model is far more usable and far more frightening at the same time.
The PatchBench number about PoC checks inflating solve rates 1.83x is the part I'd want more people to internalize. An agent that runs its own proof-of-concept and grades itself is exactly the setup where "it works" and "it demonstrably works" diverge, and the grader is the least monitored piece of the whole pipeline.
One honest question from production: the inbox card says what approve does, but does anything surface what approve costs? I've had cases where approving a tiny helper dragged in a chain of implicit dependencies nobody saw until something broke. Capability diffing between "approved state" and "actual runtime state" feels like the missing tool here.
Thanks Baptiste — the drift point is the real one. Where Systemu stands today, honestly:
Pinned to a hash. "Always allow" remembers the exact tool body (a sha1 of the implementation bytes) plus its declared effect set. Edit or re-forge the tool and the signature changes, so it asks again. Same idea for MCP: each tool definition is hash-pinned when you attach a server, so a server that quietly changes a tool loses its trust. And even a remembered tool still asks if a call's arguments score higher than the one you reviewed. One gap I've written down in the code: the host a tool talks to isn't part of the signature yet, so a tool blessed against one host is blessed against all. That's on the list.
The rubber stamp. The batch card only covers tools whose effects are fully classified as safe (reads, screenshots, clipboard). Anything that runs a shell, deletes, sends, or writes to the network can't be bulk-approved at all — it asks every time, arguments in front of you. That keeps card #20 from rubber-stamping the dangerous ones. It does nothing for fatigue on the safe ones. I don't think anything does yet.
Cost. No. The forge card says what approving does; the dependency then shows up as its own card after the code is generated — you see pypdf, but only after. There's no diff between "approved state" and what's actually on disk and installed. You're right that it's the missing tool, and most of the pieces exist already (body hashes, a dependency approval store, the decision ledger). I'll build it. If you have a concrete case from your VPS, open an issue and I'll use it as the test.
On PatchBench: agreed, which is exactly why the benchmark is graded by an external oracle, never by the agent's own verifier.