Most people who try to let an AI agent operate on a real VPS end up in a few places.
They either lock it down so hard the agent can barely do anything useful (read-only, no writes, ask a human to copy-paste the command back), or they hand over a standing SSH key and just hope nothing gets hacked. Or the worst scenario, an agentic mess of deletes and rewrites of sensitive data without any backups taken.
Neither of those counts as running an autonomous agent. One is a chatbot with a read-only window into your server. The other is a loaded gun with the safety off, cowboy style.
There is another way, and it’s the only one that actually holds up once you’re doing production work on a live system. We run it daily against our own production infrastructure. Here’s exactly how it works, with a real situation from completed work we did today, not a hypothetical scenario.
This article is intentionally very straightforward compared to some of the other 70 deep dives, so it will be easy to read for most and devoid of any complex technical aspects.
The three things that have to be true at once
An agent operating on your infrastructure needs to do three things at the same time, or the whole setup falls apart.
It has to act, not just suggest. If every command gets copy-pasted by a human into a separate terminal, you’ve built a slower way of doing the work yourself.
It has to fail safely. If a write command goes wrong, there needs to be a way back that doesn’t involve your users telling you the site is down.
And it has to remember. If the agent forgets every fix and every incident the moment a session ends, it re-solves the same problems from scratch, over and over. That quietly costs more time than doing it by hand.
Most setups manage one of these. Getting all three right at once is the actual hard part, and it’s why “just give the AI a terminal” either stays useless or eventually causes a real incident.
How we actually do it
Every command gets classified before it runs.
Read-only work, checking logs, listing files, checking a process, runs immediately with no friction.
Anything that writes to disk, restarts a service, or installs a package comes back as a pending action with the exact command shown, and nothing executes until it’s approved by the sloppy human.
Some might say that's painful as they want the agent to loop forever; it is 100% necessary to stop a Chernobyl-agentic meltdown of your VPS.
Every write gets its own approval, not a session-wide green light. That’s the real difference between “the agent has SSH access” and “the agent proposes commands a human confirms,” and it stops mattering as an abstraction the first time something almost goes wrong.
Nothing gets touched without a backup first. Before any file changes, a copy gets taken automatically. That single habit is the reason you can let an agent make a real change with actual confidence instead of crossed fingers. If it screws up, which it eventually will from bad human context, bloat, or just loop errors, you go back to the previous saved backup.
Keys don’t live on the server being administered. Credentials sit in an encrypted vault and get pulled only for the exact moment they’re needed, then get destroyed right after: written, used, shredded, in one step, so there’s never a window where an interrupted session leaves a live key sitting on disk. If a managed server is ever compromised, there’s no standing key on it for an attacker to find and reuse somewhere else.
And it remembers. Not within a single chat session, but across days and across whichever AI tool you happened to have open. A fix made three months ago in a different tool is still recallable today, because the memory isn’t tied to any one app’s chat window. The agent has real-time access to recall thousands of saved memories with past actions.
What this looked like today
We used exactly this setup to make a real production change: syncing new navigation links across 20 live HTML pages on a public site. Here’s what actually happened.
Twenty-four automatic backups were taken before anything was written, one per file group plus individual snapshots, with no manual setup required. Every write command showed a preview and needed explicit approval before touching the live site, so nothing ran silently. The site never went down. Checked right after: HTTP 200, 14 millisecond response time, zero crash-driven restarts, zero downtime.
A separate memory test, run fresh in a brand new session, asked simply what a given repo does. The answer surfaced a specific past bug fix (a function called getSlipstream() had been returning module exports instead of a proper singleton instance) that no amount of reading the current code would have revealed.
That's what persistent memory is actually for. Not remembering what a file contains right now, but remembering what already went wrong and how it got fixed.
That’s the whole idea, demonstrated rather than claimed: act for real, fail safely, remember what happened. Miss any one of the three and you don’t have an autonomous agent. You have either a contagion-sized liability or a cron job agentic toy.
Why it doesn’t work without this
Skip the approval step and the first hallucinated delete or config overwrite becomes a real incident with no undo. Skip the backup and “I fixed it” from an agent is a claim you still have to verify yourself, which erases most of the time you thought you were saving.
Skip the vault and every server you’ve pointed an agent at becomes a pivot point the day any one of them gets compromised. Skip persistent memory and every session starts back at zero, so you’re not saving time at all, you’re just paying the “explain everything again” tax forever.
None of these get bolted on easily after the fact. Each one is a real decision made up front, and getting any of them wrong is the difference between a tool you use daily without thinking about it and one you’re quietly afraid of.
How the pieces actually connect
None of this depends on which AI client you happen to prefer. Claude Desktop, Claude Code, Cursor, and Windsurf all speak the same protocol underneath, called MCP.
That protocol is what makes the setup portable instead of custom-built per tool. You run one small server locally (or point at one running elsewhere), and any MCP-capable client can connect to it the same way, using its own config file but pointing at the same underlying process.
For memory, that server is backed by a local SQLite database with embeddings generated on your own machine, no cloud API involved in the recall itself. The one detail that trips people up, and it tripped us up too the first time, is that the database path has to be set explicitly in every client’s config.
Leave it out and the server falls back to a default location, which means Claude Code and Cursor can end up talking to two different, mostly empty databases while you assume they’re sharing one brain. Registering it for Claude Code looks like this:
claude mcp add vektor \
--env VEKTOR_LICENCE_KEY=your-key \
--env VEKTOR_DB_PATH=/home/you/.vektor/slipstream-memory.db \
-- node /path/to/vektor.mjs mcp
The same two environment variables get set in Claude Desktop’s config, in .cursor/mcp.json, and in Windsurf's MCP config. Four different files, four different formats, one shared database underneath.
Once it's registered, running claude mcp list (or the equivalent command in whichever client you're using) confirms the connection actually took, which is worth checking rather than assuming. We caught two false positives doing exactly this, once where a server was never registered at all and once where a removal hadn't actually taken effect.
The SSH side runs as a separate server speaking the same protocol, and the two are meant to work together rather than in isolation. A typical write to a remote server goes through four steps in order.
The command gets classified as read-only or as a write. If it’s a write, a snapshot of whatever it’s about to touch gets taken automatically. The command comes back as a pending action with the exact text shown, waiting for an explicit approval before anything executes.
Once approved, it runs, and the response that comes back includes a health check alongside the command’s own output, so “did this work” and “did this break anything” get answered in the same round trip instead of requiring a follow-up.
Credentials for that SSH layer never sit on either end permanently. They live in an encrypted vault, get pulled for the single command that needs them, and get removed again immediately after, written, used, and deleted in one step rather than as a separate cleanup that could get skipped if something crashes in between.
Put together, that’s the full loop: an MCP client asks for a change, memory supplies whatever relevant context already exists from past sessions, the SSH layer backs up the target before touching it, a human approves the specific command, it runs, and the result gets verified before the agent reports back that it’s done. Laid out visually, the whole path looks like this:
Memory and execution run as two independent servers, but they answer to the same client and the same conversation, which is why the agent can recall a past fix and then safely act on it in one continuous exchange instead of two disconnected tools.
Five ways to actually use this
Safe production changes across many files at once. Instead of hand-editing twenty pages one at a time, an agent searches for the exact pattern across every file, applies one batched change, and verifies the result afterward. Backed up before, checked after, approved at every write. The nav sync above isn’t a special case. It’s the normal case.
Incident memory that survives the incident. The next time the same class of bug shows up, the agent doesn’t start from zero. It recalls the last time it happened, what caused it, and how it got fixed, and can propose reapplying that fix instead of re-diagnosing everything from scratch.
Fleet administration without standing keys anywhere. Managing several servers doesn’t mean a private key sitting on each one. One vault, one fetch-use-shred pattern per connection, no server holding a credential that outlives the single command it was needed for.
Log and security monitoring with a memory of normal. Checking auth logs or traffic patterns is more useful when the agent remembers your baseline, what normal traffic looks like, what’s already been flagged before, instead of reading raw numbers cold every single time you ask.
Cross-tool continuity for solo work. Start a task in one AI client and continue it in another, and the memory comes with you, because it’s stored once, locally, and any client that connects to it sees the same history instead of keeping its own separate, empty one.
Every one of these runs against our own infrastructure. We watched the backups get taken, watched the approvals gate the writes, and watched the memory surface a real fact in a fresh session that it had no way to know about before today, on the same system this article is discussing.
None of this is theoretical, and none of it needed a special setup built just for this article. It’s the same memory server and the same SSH layer we run every day for the last 4 months, connected to whichever client we happen to have open at the time.
If you’re already letting an AI agent near a real server, or thinking about it, the questions worth asking are the same three from the top of this piece: can it actually act, will it fail safely, and will it remember any of this tomorrow. If the honest answer to any of those is no, that’s the gap to close first, before the agent, not after.
Before you hand any agent access to something that matters, ask it the same three questions we asked ourselves building this.
Does it have a real human-in-the-loop failsafe, or does it just say yes to itself or everything and then say sorry after the meltdown?
Can you actually trust it not to overwrite or delete your work the first time it misreads what you meant?
And does it make backups automatically, or only after you’ve already lost something and asked for that feature?
If you can’t answer all three with a straight yes, you don’t have a safe setup yet, you have a countdown to disaster and another sobby post on Reddit.
We built Vektor because we couldn’t answer those three questions honestly with anything else we tried, and we weren’t willing to run an agent against our own production systems until we could. Everything in this article is the actual setup we still use, eating our own dog food in real life, not a version cleaned up for publication.
If you want to see it for yourself before trusting it with anything real, the docs, the source, and the full setup instructions, the MCP configs for each client, and the rest of the toolset are at vektormemory.com/docs.
Try it against a repo or a server you already know well. That’s the fastest way to see whether it holds up, because you’ll know immediately how efficient it is.
Vps
LLM
AI Agent
Vector Database
Top comments (0)