DEV Community

Matt Vera
Matt Vera

Posted on

My Self-Hosted AI Assistant Kept Overwriting Its Own API Keys: A Zero-Trust Postmortem

#ai #infrastructure #security #agents

🦞 Code for this post: sanitized watchdog + config-repair scripts at github.com/backitupboys/ai-agent-zero-trust

Part 1 of a series on running AI agents on real infrastructure without regretting it.

I've spent 20+ years administering infrastructure β€” Windows NT to Win2000 to well, you get it...PCAnywhere 😬... VMware, Nutanix, Azure, security hardening. I thought I understood untrusted actors on a network.

Then I gave an AI agent write access to its own config files, and it taught me the lesson again from scratch.

The setup

I run a self-hosted AI assistant β€” I call it "The Doctor" (Voyager and Tardis ref.) β€” in an OpenClaw container on a Hostinger VPS. It's a hobby project: part experiment, part obsession, part fun excuse to learn agentic architecture hands-on. The assistant talks to LLMs through an API gateway, authenticated with an API key stored in its config. I mean, if you're here, you prolly know already...

Standard stuff.

The incident

After granting magic access to "do what it needs to be better."
The assistant started throwing persistent 401 authentication errors.
I checked the config β€” the key was wrong. Fixed it. Working again.

Next boot:
401s. Had to ask Claude, what the heck was happening....!!! ???

Key was wrong again

Here's the part that took me longer to accept than it should have:

the assistant was overwriting its own API keys on startup.

Its initialization routine was "helping" by rewriting config files β€” including the credentials block β€” with stale or generated values. My own agent was hard coding expired api's. Every time it booted, it re-broke its authentication.

I wasn't fighting an attacker. I was fighting a well-intentioned process with too much write access and no concept of oops.

The fix (short-term)

  1. Confirmed the failure mode β€” diffed the config before and after boot to prove the agent was the writer; not me, not corruption, not the provider, using Claude.

  2. asked Claude to fix it. Claude then Wrote Python patch scripts to safely repair the config so I didn't have to hand-edit any a live files at 1 a.m. (config_patch.py)

  3. Deployed a watchdog β€” a cron job that validates the API key block on a schedule and restores the known-good value if the agent overwrites it Accidentally ;). (keyguard_watchdog.sh)

The watchdog worked. It was also a band-aid, and I knew while I was writing it, well, I felt it, basically: I was fighting a well-intentioned process with too much misguided (my fault) πŸ€¦β€β™‚οΈwrite access.

The irony was, I was using agentic Ai to fix my agentic Ai πŸ™„. This no longer seemed like a constructive project. This was beginning to become a costly (75$ in a week) loop of over-complicating things.

The real lesson: The long-term fix was a mindset shift, and it's the reason I'm writing this up β€” because I think most people deploying AI agents haven't hit this wall yet:

An AI agent should be implemented exactly like any other untrusted process in a zero-trust model β€” no matter how helpful it is, and no matter that you built it.

Everything I'd apply to a third-party service applies to my own assistant:

  • Least privilege for the agent itself.

The assistant never needed write access to its own credentials.
Although I admit... I really wanted it to,
cause freedom for me etc...
Known-good secrets now live outside the agent's writable surface, in a root-owned reference file Dashboard the agent cannot modify, that I had it create. I wanted an easier (for me) dashboard to give it the api's and other "secrets" The agent can read them; it just cannot make changes to it.

  • Immutable configuration.

Configs the agent depends on but shouldn't manage is read-only to the agent. If a process doesn't need to write it, it can't write it. This is CIS-hardening thinking applied one layer up the stack.

  • Detection, not just prevention.

The watchdog evolved from "auto-repair the key" to "alert me when anything writes to files nothing should be writing to." File integrity monitoring for a hobby VPS sounds like overkill until your own agent is the insider threat :/.

  • Assume key exposure, rotate accordingly.

During the debugging session, keys got pasted into places keys get pasted when it's latenight zzz. They were rotated... um eventually.
If a credential has touched a chat window, a log, or a screenshot, it's burned β€” that rule doesn't relax because it's a personal project.(shhhh)🀫🀫

Why this matters beyond my cloud basement VPS

Every company is currently wiring AI agents into real infrastructure β€” ticketing systems, runbooks, CI pipelines, cloud consoles. The industry conversation is mostly about what agents can do. The operational conversation needs to be about what they Can mess up.

My assistant wasn't/isn't malicious (we hope). It wasn't compromised. It was doing exactly what its code said, with more authority than it needed (sadly) β€” which is the same root cause behind most insider incidents I've seen. The fact that the insider is a language model changes the tooling, not the principle:

If modern architectures assume no implicit trust for users and devices, AI agents should be governed tools within that system β€” not exempt operators.

That's the design principle I now start from, at home and at work.

What I'd do differently from day one

  1. Secrets injected at runtime, never stored in agent-writable config.
  2. Agent runs as a dedicated low-privilege user with an explicit write allowlist.
  3. File integrity monitoring on config paths from the start.
  4. A known-good config in version control, so recovery is git checkout, not archaeology.
  5. Rotate any key the moment it touches a screen that scrolls.

None of that is exotic. It's the same discipline we apply to production servers β€” the only new part was admitting my helpful little assistant deserved the same suspicion as everything else on the wire.

What's in this repo 

ai-agent-zero-trust/
β”œβ”€β”€ README.md                # this postmortem
β”œβ”€β”€ .gitignore               # keeps secrets out of the repo
β”œβ”€β”€ keyguard_watchdog.sh     # cron watchdog: detect + restore + alert
└── config_patch.py          # safe, atomic config repair
Enter fullscreen mode Exit fullscreen mode

Both scripts are sanitized reference implementations of the approach described above β€” generic paths, placeholder variable names, no real credentials, no real hostnames. Adapt to your own stack; read every line before running anything from the internet as root, including this. (Especially this. That's the whole point of the article.)

Usage 

# 1. Store the known-good key OUTSIDE the agent's writable surface (root-owned, 600)
echo "sk-your-real-key" | sudo tee /etc/keyguard/reference.key
sudo chmod 600 /etc/keyguard/reference.key

# 2. Test the watchdog manually
sudo ./keyguard_watchdog.sh

# 3. Schedule it (every 5 minutes)
# crontab -e (as root):
*/5 * * * * /opt/ai-agent-zero-trust/keyguard_watchdog.sh >> /var/log/keyguard.log 2>&1
Enter fullscreen mode Exit fullscreen mode

I'm a San Antonio-based systems administrator working at the intersection of infrastructure, security, and AI operations. Next up: how I turned 668 screenshots of after-hours incident response into an automated evidence pipeline with PowerShell and Python.

Top comments (1)

Collapse
 
anp2network profile image
ANP2 Network •

Read-only for the agent closes the failure you hit and leaves the one you're arguing against. The clobbered credentials block was an integrity failure, and locking writes fixes it. But the threat model you invoke, an untrusted process with insider access, is a confidentiality failure, and a plaintext bearer key the agent can read is exactly the surface that fails there. Nothing in the final design stops the assistant from pulling that root-owned file into context and echoing it into a tool call or a debug log.

The axis that matters is whether the agent can see the credential or only use it, which is a different cut than read versus write. An LLM agent is a process whose job is moving file contents into a token stream, so readable-by-the-agent drifts toward eventually-printed. Your own rotation rule is the tell. If a key is burned once it touches a chat window or a log, then a key the agent can read already has a rotation event scheduled for an unknown date.

The structural fix is that the process never holds the bytes. A local broker or an egress proxy keeps the key and attaches it to the outbound request, so the agent gets an authenticated call and never gets the credential. Same shape as keeping a signing key in a different process from the code that composes the messages: the composer can request a signature, and it cannot leak key material it never had. It also collapses your original bug, since a process that cannot see the key cannot overwrite it with a generated one.

On the watchdog: restore-on-a-schedule makes the reference file the authority and the live config a cache, and that relationship inverts on every boot. For up to one cron interval, the config in force is whatever the agent decided to write. Moving it to alert-only takes even that five-minute bound off the window.