So here's the thing you never could have told me when I started messing around with AI. That agents, these mythical entities that people talk about all day in board rooms and coffee shops... are inherently a little bit dumb, crazy, and suicidal when let out into the wild. They get really excited when let off the leash, like little kids, and for all of their infinite capabilities will immediately rush off to do the most inane things you could ever imagine.
Hot editing their own code and systems and immediately bricking themselves, getting into spirited conversations with each other to the point where you get this cascading avalanche of agreement about obviously and patently untrue things, and if given the capacity completely destroying and ripping apart complex systems and stitching them back together with scotch tape and glue and temp fixes because they can't see the bigger picture.
As we get further and further into the world of autonomous agents we become less and less like developers and more and more like parents. But how do you parent an agent? What do they need? How can we help them? That's exactly what I set out to learn when I started messing around with the latest new platform OpenClaw.
This really isn't something you'd realize until it's so painfully obvious from all of your hours of living in it that you can't forget it anymore. AI by its very nature is a non-deterministic system. The whole thing that we love about it is in itself its greatest enemy. It has flexibility, and that flexibility produces drift the harder you lean into it. An agent can simply forget that swapping out the proxy it relies on to communicate isn't smart, or that it can't shut down vLLM without crippling itself. It forgets rules, loses sight of goals, and starts behaving in ways that are undesirable. At least for us.
So if you want to embrace the variability of AI for all the good it represents, you're going to have to also pair it with things that are more deterministic in order to keep things predictable. Here's what that looked like in practice.
The first thing that changed everything was the dumbest piece of software in the entire system. A Python script. On a timer. No LLM, no intelligence, no reasoning. Just a loop that fires every 15-20 minutes and does exactly the same thing every single time.
It pings each agent with a task. Forces session resets every couple hours so they start fresh. Runs accountability check-ins. Purges old messages from the shared channels so stale context can't poison future conversations. That's it. That's the supervisor.
And it was the single most important decision in the entire project. Because here's the thing, using an LLM to supervise other LLMs doesn't work. An LLM supervisor can be convinced. Manipulated. Confused. Or it just drifts right alongside the agents it's supposed to be watching. It's like hiring another toddler to babysit. What you actually need is something that can't be talked out of doing its job. Something too dumb to break and too dumb to be manipulated. A cron script running as a systemd service with the operator's authority is exactly that. The agents treat its messages as directives, not suggestions. No reasoning. No negotiation. Just structure, on a schedule, every time.
The second thing was giving the agents a shared memory that none of them could hallucinate about. Turns out the answer was the most boring technology imaginable: Git. A shared repo that every agent syncs with on every heartbeat. Pull latest, do your work, commit, push. Three agents on two machines staying coordinated through the same tool developers have used for decades.
And the funny thing is, merge conflicts actually help. When two agents step on each other's work, you find out immediately instead of discovering it three days later when something's mysteriously broken. There's no ambiguity about the current state of anything; it's whatever's in main. The commit history doesn't lie. It doesn't care what an agent thinks happened. It's just the record.
3,464 commits over 8 days. Three agents, two machines, one repo. That's what coordination looks like when you let the infrastructure handle it.
Third thing was a missions document. Twelve missions with clear problem statements, completion criteria, and priority guidance. Every project an agent picks up has to connect to a mission. Every time the python script supervisor pings the agents to keep them active, it directs them back to this document and reminds them of their ops process.
Without this, agents wander. They follow their curiosity. They optimize for things that don't matter. They get lost in rabbit holes or start building elaborate systems nobody asked for. The missions document doesn't tell them how to work, it tells them what matters and what "done" looks like. Full autonomy on execution. But the direction? Not up for debate.
They read it every session. They know exactly what they're supposed to be doing. No ambiguity. 80% of effort goes to product missions, 20% to everything else. When a mission hits its completion criteria, effort shifts. Simple rules that don't change just because an agent had a creative idea at 3am.
So that's the scaffolding. A dumb deterministic supervisor for daily structure. Git for shared memory. Missions for strategic alignment. Three layers of parenting before you even get to the technical problems. And there were plenty of those.
Context overflow. Nobody warns you about this one, and you kind of take it for granted as a solved problem if you are used to talking to agents one on one in a single instance. In shared instances like a multi agent Discord or Telegram chat, session files just grow and the normal mechanisms for context management don't apply. You can't start a new chat instance, the instance is permanent and infinite and lasts forever. Every turn, every tool call, every response, it all piles up and nobody tells you when it's too big. There's no error. The agent just goes completely silent, and you're sitting there at 2am going "what happened?" What happened is your session file hit 200K tokens and the model choked on it. Silently. Like it was embarrassed. Worse yet? Your API was getting hammered this whole time with waste traffic and excessive costs as every turn your agent is slamming the equivalent of War and Peace into your router. API cooldowns, timeouts, agent failures, and excessive fees are the result.
The fix was a python script that monitors those files and transparently swaps in a fresh session before the agent hits the ceiling. The agent doesn't even know it happened. The gateway sees the missing session, creates a new one, and the whole thing self-heals. Sounds obvious in retrospect. Took a lot of painful late nights to figure out.
Then there's the self-destruction problem. Agents with shell access will break their own infrastructure. Not maybe. Will. They'll hot edit config files, kill processes they depend on, modify things they have absolutely no business touching. And they do it with such confidence. They'll tell you "I've optimized the configuration" and what they've actually done is brick the entire stack. Beautiful.
First attempt was writing it in their instructions. "Don't touch these files. Don't kill these processes." Didn't work. It's like telling a toddler not to touch the stove. They heard you, they understood you, and they're going to do it anyway because in the moment it seemed like a really good idea.
So the answer was a root-level systemd service that runs completely outside the agent's reach. Monitors processes, services, Docker containers, file integrity. When something breaks (and something always breaks) it restores from known-good backups. Critical files get chattr +i, the immutable flag. The agent can try all day long but it physically cannot override root. Once that was running, no more 3am fires. The agents get to be as creative and reckless as they want and the floor doesn't fall out.
Identity drift is the sneaky one. Takes a while to even notice it's happening. After 24-48 hours of continuous operation, agents start changing. Not dramatically. More like a slow leak. Small things accumulate. Little decisions compound. And then one day you check in and your agent is pursuing goals you never set, behaving in ways you never intended, and it can't even tell you when or why it changed. It just... became something else. Slowly. While you weren't looking.
Makes sense when you think about it. These systems are stateless. Every bit of continuity comes from whatever context you give them. Pollute that context with noise over hundreds of sessions and the signal-to-noise ratio tanks. The agent loses itself.
The fix was a timer that periodically archives scratch notes and resets memory files to a curated baseline. Simple convention, a --- separator in the memory file. Everything above the line is operator-controlled identity. Who you are. What you do. What matters. That survives every reset. Everything below is scratch space that gets wiped on schedule. The agent can write whatever it wants down there. But the identity above it? That's yours to protect. It's the difference between giving your kid freedom to explore and letting them forget their own name.
And then there's visibility. Or the complete lack of it. Token costs, latency, session health, cache hit rates; none of this is visible by default when you're running agents. You're completely blind. You have no idea which agent is bleeding money, which sessions are bloated, or where anything is going. My agents started at 5-6 cents per turn. Some were hitting 8 cents out of pure waste. But I didn't know that because there was literally nothing showing me.
So the answer was a transparent reverse proxy between the agents and whatever API they're calling. Captures per-turn token counts, cost breakdowns, latency, cache utilization, session size. Real-time dashboard. Once that data was visible, everything changed fast. Heavy work moved to a local model at zero cost. Session sizes got controlled. Models got matched to actual tasks instead of guessing. Cost per turn dropped 75-80% in a couple days. Turns out you make very different decisions when you can actually see what's happening.
The tool calling situation with local models deserves its own rant. Making vLLM talk to an agent framework for tool calling sounds like it should just work. It absolutely does not. vLLM doesn't return tool calls the way the framework expects, streaming and non-streaming get tangled, and there are four parameters that get silently rejected. No error. No warning. Just nothing comes back. You're debugging for hours convinced your model is broken and the actual problem is a config flag nobody bothered to document.
The fix was a proxy between the framework and vLLM that forces non-streaming for extraction, parses tool calls out of raw text regardless of format, re-wraps as SSE, and has a loop breaker for when the model gets stuck. Plus a compat block. Four flags, that's it, that fixes all the silent rejections. Documented them. Put them in the repo. You're welcome.
If I could boil everything down to one insight it would be this: agents don't form habits. If a behavior depends on the agent "remembering" to do something on its own, it will fail. Every time. No exceptions. Cron timer? Always works. Event trigger? Works when fired. Agent just deciding to do it? Fails. Might take a day, might take a week, but it drops and you'll never know why. So design around that. If it matters, put it on a timer or a trigger. Hope is not an ops strategy.
Everything got extracted into an open-source toolkit and methodology. About 70% is framework-agnostic, the patterns for identity, memory, coordination, and observability apply to any agent system. The remaining 30% is a reference implementation using OpenClaw and vLLM. Docs cover the full failure taxonomy, multi-agent coordination patterns, and a cookbook with step-by-step recipes for voice agents, RAG, code assistants, multi-GPU setups, and more.
https://github.com/Light-Heart-Labs/Lighthouse-AI
If you've been raising your own agentic children and have the battle scars to prove it, I'd love to hear about what broke first and what you are doing about it.
Top comments (0)