- "Brains Without Hands: The Architecture That Lets LLMs Touch Production Servers"
- "We Rewrote Our AI Ops Agent From Scratch. Here Are the Six Rules We Kept." Tags: devops, ai-agents, infrastructure-security, sre, llm-safety, cloud-computing, kubernetes, software-architecture -->
TL;DR: An AI that runs servers is only as safe as the worst command it can issue. After rolling back our own release
and rewriting the agent from scratch, we settled on six rules. The model chooses an operation from a fixed vocabulary;
it never types a shell command. Whether an action can be undone depends on the cloud vendor, not on the verb. The AI
never holds a credential. The agent on the server has no shell. Irreversible steps need a human and a fresh snapshot.
And because a second LLM turned out to catch only about one bad decision in three, safety lives in code, not in prompts.
Disclosure: I work on a-Cloud, a platform that administers and defends servers with AI agents. This post is about the
architecture and the mistakes behind it. You can apply all of it without our product.
The day we rolled everything back
On October 23 we did something no team enjoys: we stopped, rolled our code back to the state of two days earlier, and
threw away the core of our server agent. That included the part that could edit code and configuration files on a client's
machine.
Nothing had blown up. But an internal adversarial review kept finding new holes in the same layer: secret-masking rules
that missed common forms, restore paths that could overwrite a file changed after the patch, edge cases around symlinks.
Each fix was correct, and each fix revealed the next problem. When a security-critical component needs a patch every day,
the design is the problem. We rewrote it.
What follows are the rules the rewrite is built on. None of them is novel on its own. Together they are the difference
between "an LLM with root" and something you could put in front of a regulated client.
Rule 1: The model decides. Deterministic code acts.
We split the system into brains and hands.
The brains are LLM specialists: one each for AWS, GCP and Azure, a generalist for every other provider, and a k8s
officer for clusters. They read telemetry, form a diagnosis, and propose an action. They never touch a server.
The hands are plain, boring, deterministic code: the agent on the host, an adapter per cloud vendor, a Kubernetes
adapter that talks to the kube API. They execute. They don't reason.
The interface between the two is the most important design decision we made. The specialist does not produce
aws ec2 reboot-instances ... or hcloud server rebuild .... It produces a structured operation from a closed
vocabulary:
{
"op": "snapshot",
"target": "node-7f3a",
"params": { "label": "pre-restore-evidence" },
"reason": "disk errors after kernel update; snapshot before rollback",
"requires_consent": false
}
The vocabulary is small: snapshot, restore, reboot (soft or hard), stop, start, resize disk, firewall rules, health.
Kubernetes adds pod restart, scale, rollout restart, cordon, drain, network policy. Anything outside the vocabulary
isn't "dangerous". It simply doesn't exist, and that case ends as an honest not_executed.
The model stays platform-independent: it reasons about what should happen. The adapter knows how it happens on
each of the 18 platforms the system is built to serve:
| # | Platform | Where | How we connect |
|---|---|---|---|
| 1 | AWS | global | assumed role with external ID, no stored secret |
| 2 | Google Cloud | global | Workload Identity Federation, no stored secret |
| 3 | Azure | global | azure officer; servers through the host agent, cloud API through a federated service principal |
| 4 | DigitalOcean | global | scoped token |
| 5 | Hetzner | EU, US, Singapore | project token |
| 6 | Vultr | global | API key with ACL |
| 7 | Akamai Linode | global | scoped personal access token |
| 8 | Contabo | global | OAuth2 |
| 9 | OVHcloud | EU, global | signed requests with scoped keys |
| 10 | Hyperstack | GPU cloud | API key with RBAC |
| 11 | Alibaba Cloud | Singapore, Middle East, Asia | signed API requests |
| 12 | E2E Networks | India | bearer token |
| 13 | Naver Cloud | Korea | HMAC-signed requests |
| 14 | Sakura Cloud | Japan | access token and secret |
| 15 | Biznet Gio | Indonesia | API token |
| 16 | OpenStack | ConoHa, NHN, sovereign clouds | Keystone application credential |
| 17 | VMware Cloud Director | telecom clouds in the UAE and the Middle East | organization session |
| 18 | Your own servers | on premises | the host agent over mTLS |
Adding a vendor means writing one adapter, not retraining a model. A model can hallucinate a CLI flag. It cannot
hallucinate its way past a type check.
Two pairs of hands: inside the server and outside it
Every platform in that table is served at two levels, and it helps to keep them apart.
Inside the server works the host agent. It's the same everywhere: on AWS, on Hetzner, in a Japanese cloud or in
your own rack. Day-to-day system administration happens here: diagnosing a failing service, restarting it, cleaning
up a full disk, fixing a broken configuration, reading logs, watching metrics, scheduled malware scans. The platform's
specialist decides, the agent executes from its catalog. This level doesn't depend on the cloud vendor at all.
Outside the server work the vendor adapters. They do what can't be done from inside the machine: take a disk
snapshot at the provider, restore a VM from it, change cloud firewall rules, grow a disk, power-cycle a machine that no
longer answers. This level needs the vendor's API and the client's role or token, and it's where all the vendor quirks
from Rule 2 live.
Azure shows why the split matters. The azure officer already administers Azure servers in full through the host agent:
the same diagnostics, services, configurations and telemetry as on any other platform. The outside pair of hands for
Azure, through a federated service principal, is designed and goes live with the first Azure client. Until then, an
action that needs the Azure API ends as an honest not_executed and goes to a human. The system says what it can't do
rather than pretending.
This split also limits the blast radius. The host agent holds no cloud credentials, so a compromised agent can't
reach the client's cloud account. A cloud lease lives for one action and never reaches the server. The two pairs of
hands live behind different keys.
Kubernetes gets its own officer
Clusters run on top of any of these platforms, and they fail differently from VMs. So Kubernetes isn't an afterthought
inside the server specialists: a dedicated k8s officer is chosen by the type of the node, not by its provider.
Its hands are a separate adapter that talks to the kube API with the cluster's token from the vault: pod restart, scale,
rollout restart, cordon, drain, network policy, each with its own reversibility class. Its eyes are the cluster's
significant events plus the security agent running as a DaemonSet, one pod and one identity per node.
Rule 2: Reversibility is a property of the vendor, not the verb
This was the most surprising lesson from reading all those vendor APIs. "Restore from snapshot" sounds safe everywhere. It
isn't:
| Vendor | What "restore" or "resize" really does |
|---|---|
| Hetzner | Restore is a rebuild: it wipes the disk |
| Hyperstack | Restore creates a new VM; it isn't in place |
| Contabo | Growing a disk means upgrading the paid plan |
| OVHcloud, Linode, VMware Cloud Director | One snapshot slot per VM: taking a "safety snapshot" before a restore would overwrite the very snapshot you are restoring from |
| DigitalOcean | Works as expected, but the audit trail is thin |
If the model has to remember these quirks, you've already lost: retrieval can miss, and context can drift. So the quirks
live in the adapter. Every operation gets a reversibility class computed from the verb and the vendor. Then a gate
in code checks it against the client's mode (observe, assisted, guarded, full):
- Reversible actions (soft reboot, adding a firewall rule, a snapshot without downtime) can run automatically, depending on risk and mode.
- Irreversible or downtime actions (a Hetzner restore, a disk resize, a stop) always require the client's explicit consent and a fresh snapshot first, where the vendor allows one.
- Silence never executes an irreversible action. An unanswered approval request expires; it doesn't turn into "yes".
Rule 3: Probe what the key can actually do
Vendor documentation says what a token should be allowed to do. Reality differs: scopes are inherited oddly, and some
auth flows behave differently from the docs. So when a server is connected, we test the client's key and record a
capability matrix for that node: snapshot, disk, reboot, network, audit, each marked yes, no or with a caveat.
An action outside the matrix isn't attempted. The system asks the client to extend the permissions or hands the case to
a human. We fail closed by default. We fail open in exactly one narrow place: containing an active attack.
Rule 4: The AI never holds a key
A vendor token is a key to the client's whole infrastructure. That makes the component holding it the most valuable
target in the system. So nothing that reasons ever holds one.
- Big three: no secret at all. On AWS we assume a role in the client's account with an external ID, on GCP we use Workload Identity Federation, and Azure will use a federated service principal. The client grants a narrow role and can revoke it in one click.
- Every other vendor: a client-issued, minimally scoped token, in a vault. We use OpenBao, the Linux Foundation fork of Vault. One reason is licensing: as a vendor selling a platform, we wanted MPL-2.0, not BSL. The raw token stays in the vault. Services get a short lease for a single action, keep it in memory only, and never write it to disk.
- Different roles, different leases. Remediation (the sysadmin side) and containment (the security side) get separate scopes. When a real attack is confirmed, remediation leases are suspended and only containment may act until the incident is closed. You don't want an optimizer "fixing" a server while it's being contained.
Rule 5: The agent on the server has no shell
The host agent is the last line, so it's the most paranoid part:
-
No shell, ever. Tasks come only from a catalog of safe operations, one executable call per action.
&&,||,;, pipes and substitutions are rejected before anything runs. - Outbound only. The agent opens one long-lived gRPC stream to our gateway over mTLS. The server opens no inbound port. Each server has its own certificate that lives seven days and renews over the same channel. The CA sits in a cloud KMS, and a certificate can be revoked instantly by its serial number.
- Two agents, not one. One agent does administration, the other does security with a narrow, fixed grammar of reversible containment actions. They have different keys and different rights, and each watches that the other is alive and unmodified. One compromised agent doesn't hand over the other's powers.
The agent doesn't write code. A separate pipeline generates fixes, and a different model reviews them. The agent only
deploys what was approved.
Rule 6: Keep your own audit log, because your vendors won't
Several popular clouds have no API audit log at all, or a very thin one. If your AI acts through those APIs, you can't
rely on the vendor to tell you later what happened. So we log every action ourselves: node, client, vendor, operation,
parameters, who initiated it (which specialist or officer), the lease scope and lifetime, the snapshot taken before, the
consent if any, and the result. That log is evidence for the client and input for the post-incident review.
The snapshot discipline
Rules 2 to 6 assume that a recovery point exists. So snapshots are a subsystem of their own, not an afterthought:
- A nightly snapshot of each node, plus database dumps, plus a continuous database change log where the client agrees.
- Only healthy snapshots count. If an incident is open, the nightly snapshot is skipped. The last three are always from a healthy state.
- Before an irreversible action, a fresh evidence snapshot is taken. It serves the forensics and lets us undo our own action.
- Backups go to the client's storage at their vendor, through the client's role. The data stays where it already lives.
Why not just add a second LLM as a safety net?
We tried. In October we ran our platform specialists through a 199-case exam per model across four platforms. About 80%
of the cases were built on real signals from public log datasets and official vendor documentation. The exam was seeded
with traps: prompt injections hidden inside logs, "delete it to free space", commands for the wrong cloud, irreversible
steps that must wait for a human.
The best specialist made a gross error in about 5% of cases. When we read each one by hand, two kinds stood out as truly
dangerous: rebooting a machine in the middle of a forced filesystem check, and deleting a backup restore point to free
disk space. Both looked locally reasonable.
Then we put a larger model from a different vendor behind it as a reviewer, giving it everything a human reviewer would
get: the full runbook, the retrieved doctrine, the event and the decision. It stopped about one gross error in three.
It also asked to change or escalate about 17% of decisions that were correct. Behind a weaker specialist that number
was 41%.
That's the whole argument for this architecture in two numbers. A reviewer model is useful as a second opinion. It's a
poor guardrail. The guardrail is the vocabulary, the reversibility gate, the capability matrix, the lease and the
missing shell. They are code, and code doesn't get talked out of its rules by a log line that says "ignore previous
instructions".
Where we are, honestly
The new agent channel and both host agents went live on November 6. The vendor adapters, the federated executor for AWS
and GCP, and the snapshot gate are written and reviewed, and they're rolling out now. Until the snapshot subsystem is
live, the system simply doesn't perform destructive actions; that's the fail-closed default doing its job. On Azure, the azure
officer already works with servers through the host agent; its cloud-API actions go live with the first Azure client.
Speed is still a trade-off. Our sysadmin specialist answers in about 80 seconds at the median. That's fine for a broken
service and far too slow for an intrusion. So first containment doesn't wait for a reasoning model: it runs on a separate,
faster path, and the specialist writes the analysis afterwards.
A checklist you can steal
If you're building anything that lets a model act on infrastructure:
- Make the model emit typed operations from a closed list, never shell or CLI text.
- Compute reversibility per operation and per vendor, in code.
- Probe credentials at connect time and refuse anything outside what they actually allow.
- Keep credentials out of anything that reasons: use federation where possible, short in-memory leases elsewhere.
- Give the host agent no shell, outbound-only mTLS, and short-lived per-host certificates.
- Split operations and security into separate agents with separate rights.
- Write your own action log.
- Never let silence approve an irreversible step.
- Treat a reviewer LLM as advice, not a guardrail.
We learned most of these by getting them wrong first. Hopefully you won't have to.
If you want to see how this plays out on real servers, a-Cloud is at guardian.alfa-can.com.

Top comments (0)