DEV Community: Tijo Gaucher

5 ways your AI agent runtime silently dies overnight (and the boring fix for each)

Tijo Gaucher — Mon, 04 May 2026 07:40:16 +0000

I ran the same agent for thirty straight days. It died five times. Four of them did not show up in any log I had set up ahead of time, which is the part that bothers me most.

By the end I had a checklist of things that take an agent down at 2am while you're asleep, and none of them are the dramatic failures that get blog posts. They are all dull.

Here is the list.

1. OOM during a long tool-call loop

The agent is happily looping through 200 tool calls in one task. Each call returns a response. The agent appends every response to its working context plus an internal trace it writes to disk. Around call 150, RAM usage starts going up faster than usage going down. By call 180, the kernel OOM-killer wakes up and ends the process.

In the log: nothing. The agent's stdout cuts off mid-sentence. The supervisor logs say "process exited 137" which is the OOM signal but very few people read it that way the first time.

The boring fix: cgroup memory limits with a soft warning at 80%, plus a tool-call counter that flushes the working trace to disk every 25 calls and resets the in-memory copy. Not exotic. Just remembering that long agent loops are basically a memory leak unless you actively flush.

2. File descriptor exhaustion

Day eleven. The agent had been making API calls all day. A new tool call started and immediately got OSError: too many open files. The agent caught the exception, tried to retry, got the same error, gave up, returned an error to the user.

The agent itself didn't crash. It just stopped being useful. The supervisor process saw "agent returned an error" and moved on. Nothing alerted.

What actually happened: the agent's HTTP client was reusing a session pool that didn't close idle sockets, and over 11 days it had accumulated about 950 open FDs against the per-process default of 1024. Every new HTTP call added to the pool. Eventually it ran out.

The boring fix: explicit session lifecycle with a timeout, a daily restart of the agent process, and ulimit -n raised to something sane (16384 on the runtimes I cared about). The daily restart is the cheap one. People resist it because it feels primitive, but every long-running daemon I have ever shipped survives on a daily restart somewhere in the stack.

3. Context window bloat

This one I had read about, but it still got me. The agent's working context grew to about 180,000 tokens by hour 60 of a multi-day task. Each new tool call cost more than the last because the model was paying to re-read the entire history. By hour 65 a single tool call was taking 90 seconds and burning through the per-minute rate limit, which the agent interpreted as "the API is down" and went into a backoff loop.

The agent didn't crash. It just got slower and slower until it was producing nothing, and the bill kept going up.

The boring fix: a context summarizer that runs every N tool calls, replaces the last K turns with a one-paragraph summary, and keeps the most recent 5 turns verbatim. This is well-trodden ground in the literature, but the surprising part is how rarely small teams actually wire it up. Most agent codebases I have looked at assume the conversation will end in a few turns. Long-running agents need garbage collection on their own conversation history.

If you want a longer treatment of why AI agent hosting is mostly about boring problems like this one rather than the model itself, the longer version is worth a skim. The summary: the model is the easy part now. Everything around it is where the failures live.

4. ulimit walls (max user processes)

Day nineteen. The agent had spawned a background process for a long-running task and then gone on to do other work. Background tasks accumulated. By midnight there were 287 zombie processes attached to the agent's user, the per-user max user processes limit was somewhere around 1024 in this environment, and at 03:14 a new spawn failed with Resource temporarily unavailable.

In the log: a single line saying the spawn failed. The agent caught it as a generic exception and continued. The user-facing behavior was "this task takes forever." Three days later when I finally noticed I had to manually reap the zombies.

The boring fix: a process supervisor that owns the lifecycle of every spawned task, kills anything that has been alive longer than its declared TTL, and treats child processes as a resource that needs to be tracked. setsid and prctl(PR_SET_PDEATHSIG) are your friends. Also raise ulimit -u to something generous, but the real fix is killing things on schedule.

5. Webhook timeouts that look like success

Last one, and the meanest. The agent finished a task and called a webhook to notify a downstream system. The webhook took 31 seconds to respond. The HTTP client had a 30 second timeout. The client raised a timeout error. The agent's wrapper caught the timeout and logged a success because the wrapper had been written assuming "timeout means delivered, the receiver was just slow."

This is true for some kinds of fire-and-forget delivery. It is catastrophic for any kind of state-changing call. The downstream system never received the call. The agent thought it had. The user-facing system had two views of the world that did not agree.

In the log: a success line. No error. Nothing wrong.

The boring fix: idempotency keys on every state-changing webhook, a status check after every call that crossed the timeout threshold, and never treat a timeout as success without a separate confirmation. A timeout tells you the status is unknown. It does not tell you the call was delivered.

The pattern across all five

Every one of these failures had the same shape: a long-running agent ran into a resource limit or a state assumption that was fine for short tasks and broken for multi-day ones. The agent itself did not crash in three of the five cases. It just stopped being useful, and the supervisor was not watching for that.

The hosting layer needs to do three things that aren't sexy:

Memory and FD limits with warnings before the hard cap, not at the hard cap
Process lineage tracking so spawned tasks can't outlive their parent's intention
State-changing call confirmation, not just transport-level success

If you are running an agent on your laptop for an hour, none of this matters. If you are hosting OpenClaw agents in production for paying customers, all of this matters more than the model you picked.

What I now log on day one of any agent project

The thing that would have saved me the most pain on this run is just better logging from the start. None of the metrics below are exotic. None of them require an APM vendor. They are just the things I now scrape from any agent process before letting it run for more than 24 hours.

Per agent loop I track: RSS memory, FD count, child process count, total tool calls in this loop, total context tokens, time since last tool call returned, and the result of the last 10 tool calls (success or specific error). Per host I track: free memory, total FDs in use, load average, and the pid count for the agent's user. Both go to a flat file with a timestamp. No dashboard. Just a thing I can grep when something is weird.

Five of those metrics would have caught four of the five failures I described, hours or days before they actually broke things. The fifth (the webhook timeout) needs application-level logging, not host-level. That one is on the developer of the wrapper.

I have a longer guide on the hosting end of this at https://rapidclaw.dev/blog/ai-agent-hosting-complete-guide, but if you read nothing else, read this: the most expensive failure mode is the one that doesn't crash the process. Crashes get noticed. Slow-degrading agents do not.

MicroVM vs Docker for AI agents: I gave one sudo and broke the other

Tijo Gaucher — Mon, 04 May 2026 07:38:06 +0000

Last week I ran a small experiment that I should have run a year ago.

Same agent code. Same model. Same task list: install three Python packages from a CSV, fetch a few APIs, write JSON to disk, run a long-running scheduled job. Two isolation modes. One was a Docker container with the agent process inside, mounted volume, the usual. The other was a Firecracker microVM running a slim Linux image with the agent on top. Both got sudo inside their sandbox. I let them run for seven days each, then rotated.

I went in expecting the difference to be small. Memory overhead, maybe boot time. The actual difference was bigger than that.

Day one and two: Docker

Setup was the part everyone has done before. docker run with a few mounts, the agent gets a shell inside, away we go. The agent is told to apt-get install a couple of system libraries it has decided it needs. That works. It writes a 40 MB cache file to /tmp. That works. It runs a long job that opens a few hundred sockets to a public API.

Around hour eighteen the host machine's dmesg started printing about memory pressure. Not from the agent itself. From a different container running on the same host. The Python process inside my agent's container had a retry loop that would not stop holding file descriptors. That was the leak. Linux does not look at which container a process lives in when it picks something to OOM-kill. It just picks. The neighbor went down.

This is the part of Docker that no production person likes to talk about. Containers share the host kernel. They share the host scheduler. When one container goes off the rails, the rest of them feel it on the same host. If you're a small shop running one agent on one host, fine. None of this matters yet. For anything that looks like a tenant model, it stops being fine fast.

The other thing I noticed on day two: the agent decided to chmod 777 a folder it didn't own. Not malicious. Just a Python script doing what Python scripts do when permissions throw an error. With sudo available inside the container, it succeeded. The host filesystem was untouched (because mounted volumes have their own boundary), but anything inside that container was now wide open to whatever the agent did next.

Day three: rebuild as a MicroVM

I tore down the Docker setup and rebuilt the same agent inside a Firecracker microVM. Same code, same packages, same task list. Boot time went from about 200 ms (Docker) to about 700 ms (microVM). Memory baseline went up by roughly 60 MB for the kernel itself.

That is the cost. You pay it once.

What you get is a separate kernel. The agent's sudo is a real Linux sudo inside a real kernel that nobody else on the host shares. When the agent ran the same chmod 777 thing, it still happened, but the blast radius was a single VM that I could destroy and recreate in under a second. When the agent leaked file descriptors, only the VM's per-process limits got hit. The host kernel didn't notice.

Day four I let the agent install a kernel module on purpose. In Docker this would be a host-level event (or blocked outright, depending on capabilities). In the microVM it loaded into the VM's own kernel and stayed there. Whether that is useful or annoying is up to you. Either way, it stayed inside the VM.

Day five through seven: chaos

I wrote a script that told the agent to do five terrible things in a row, the kind of thing a real long-running agent might trip into over a week:

Open 10,000 file descriptors and never close them
Allocate memory in a tight loop until something breaks
Spawn 200 child processes
Write a 5 GB file to its own filesystem
Make a sudo iptables rule that drops all incoming packets

Docker handled (1) and (2) with cgroup limits if you remember to set them. Most people don't. The default is no limit. By the time I noticed (1) on the third run, the host had hit its global file descriptor ceiling and SSH started refusing connections.

The microVM handled (1) through (4) inside its own boundary. The kernel inside the VM hit its limits, the VM started thrashing, and then I killed it from outside. The host machine never noticed any of it. (5) was the most interesting. The agent set the iptables rule and locked itself out of network. In Docker the agent had affected the container's network namespace, recoverable but messy. In the microVM the agent had only touched the VM's network stack. I rebooted the VM in 800 ms and was back.

Snapshots are where the gap really opened. Firecracker can snapshot a running VM to disk and resume it later. I snapshotted the agent mid-task on day six, killed the host, restored the snapshot on a different machine, and the agent picked up its loop one second later without knowing anything had happened. Try that with a Docker container and you will spend the afternoon learning about CRIU and giving up.

The link to actually running this in production

Doing this experiment locally is one thing. Running an agent like this for a paying customer, on hardware you have to keep alive for 30+ days at a stretch, is a different problem. The boring infrastructure problem nobody writes about: it isn't the isolation primitive that's hard. Anyone can spin up Firecracker. The hard part is babysitting a hundred of these things at once, snapshotting them every so often, recovering them when a host dies, and not losing the agent's state in the meantime.

I'll plug the thing I work on once and move on. The Builder Sandbox tier is a managed wrapper around exactly this microVM-with-sudo model, with the snapshot and recovery loop already wired up. If you don't want to babysit it, that's the option. If you want to babysit it yourself, Firecracker is open source and the docs are fine.

What I'd tell past me

Running one tiny agent on a host you own? Docker is fine. The overhead is real, the boundary is real enough for that scope, and you already know the tools.

The moment you have an agent that needs sudo, runs for more than a few days, and might do something weird at 3am, switch to a real VM. The 60 MB and the 500 ms of extra boot time will pay for themselves the first time the agent does something stupid. The snapshot story alone is worth the migration.

The thing I didn't expect, going in, was how much my mental model changed. With Docker I treat the container as a thing the agent lives in. With a microVM I treat the VM as a thing the agent is. That shift, more than any individual feature, is what made the seven-day test feel different on day three than it did on day one.

A few specifics, in case you try this

The microVM rebuild was Firecracker 1.5 with a vanilla Ubuntu 22.04 rootfs, 2 vCPU, 1 GB RAM, virtio-net for the network. Boot times stayed under a second consistently once I trimmed the kernel config. I used jailer to drop privileges on the Firecracker process itself, and seccomp filters on the agent's user inside the VM. None of that is exotic. The Firecracker docs cover all of it. The only thing I had to figure out the hard way was the snapshot directory layout, which the docs assume you already understand.

For Docker the comparison build was the standard python:3.12-slim base with the agent process as the entrypoint, a tmpfs mount for /tmp, and --cap-drop=ALL plus only the capabilities the agent actually needed. Even with that, the chmod-777 case still worked inside the container because sudo plus CAP_FOWNER is enough for filesystem-mode changes. You can lock this down further with seccomp profiles, but at that point you have built a worse VM with extra steps.

If you want the longer cost breakdown of running this yourself versus paying someone to keep it alive, I wrote that up here: https://rapidclaw.dev/blog/openclaw-hosting-cost-self-host-vs-managed.

[The Boring AI Agent Workloads That Actually Pay in 2026]

Tijo Gaucher — Mon, 04 May 2026 04:38:04 +0000

Every other post on my feed is still pitching the "ambient agent that runs your whole job." If you actually run agents in production, you know that story is mostly vibes. The workloads that real people pay for, repeatedly, look almost embarrassingly mundane.

After a year of running agents for SMEs — accounting firms, e-commerce shops, two solo law practices — here are the four shapes of work that consistently survive the trial-to-paid conversion. None of them require AGI. All of them require an agent that doesn't fall over on day eleven.

1. Scheduled jobs that used to be cron + a human

The unsexy starting point. A cron job kicks off at 6 AM. It logs into a portal, scrapes a number, drops it in a sheet, and Slacks the team if a threshold trips. That used to be a half-day Selenium project plus a $40/mo VPS plus the ongoing maintenance tax of the portal redesigning itself every quarter.

An agent flips the math. The same job is now a five-line prompt and a browser tool. The portal redesigning itself is the agent's problem now, not yours. The cost question stops being "how much engineering time" and becomes "how reliable is the runtime."

That second question is the entire moat for managed agent platforms. It's also the reason most of the open-source-only "just spin up your own" pitches fall apart at month two. The agent works fine. The orchestration around it — retries, secret rotation, the headless browser updating, the model deprecating — is what bleeds the operator dry.

2. Browser automation that was too brittle for RPA

If you've ever priced UiPath or Automation Anywhere for a small business, you know the answer: it's not for them. The licensing is enterprise-shaped and the bot creation requires a specialist. Meanwhile, the actual workflow — log in, click three things, download a CSV, email it — is the kind of thing every five-person operation needs done weekly.

Agents with a real sandboxed browser tool eat this category. Not because they're smarter than RPA, but because they degrade gracefully. When the "Export" button moves three pixels left, an agent finds it. When the page adds a cookie banner, an agent dismisses it. The thing that used to take a consultant three days to update takes the agent zero.

The catch is that "real sandboxed browser" is doing a lot of work in that sentence. A Docker container with a headless Chromium is fine for a demo. For production, you want a MicroVM with sudo so the agent can actually install things, persistent file storage so its session survives a restart, and live port forwarding so you can watch it work when something looks off. That's roughly the hardware bill that managed OpenClaw hosting abstracts away.

3. Coding agents that don't touch production

This one is the most counterintuitive. The coding agent market that's working isn't replacing engineers — it's replacing the "I'll get to it next sprint" backlog at companies that don't have engineers.

Real example: a roofing company. Their internal "system" is a Google Sheet, a Calendly, and three Zapiers. They have a list of forty small tweaks they want — a column added here, a webhook there, a conditional email. None of it is hard. All of it is too small for a contractor and too unfamiliar for the owner. An agent with shell access and the patience to iterate clears that backlog in a weekend. The owner doesn't read the code. The owner reads the result.

The reliability bar here is different from the production code reliability bar. The agent doesn't need to write perfect code. It needs to not silently break the spreadsheet that runs the business. That's an observability problem, not an intelligence problem. Snapshot the state before each change, let the operator roll back, and the whole category gets safer than it sounds.

4. The "always-on assistant" that is actually a search index

The mythology of the AI assistant is that it answers anything. The reality of the paying assistant is narrower: it knows your stuff. Your contracts, your meeting notes, your invoices, your support tickets. It can pull a number out of a 200-page master services agreement faster than the human who wrote the agreement.

These deployments don't fail because the model is dumb. They fail because the data plumbing is broken — stale embeddings, a connector that silently drops half the documents, a permission boundary that leaks one tenant's data into another. None of which is a model problem.

This is the workload most people quote when they say "we tried AI and it didn't work." What didn't work was the integration. The model is fine.

What actually distinguishes the survivors

Look at those four. None of them require a frontier model. None of them require "agentic reasoning" past a couple of hops. What they require is a runtime that:

Doesn't crash, or recovers gracefully when it does
Has the right tools wired up (browser, shell, file storage, an email sender)
Surfaces what it's doing well enough that a non-engineer can tell when it's stuck
Costs predictably — flat monthly is much easier to sell to a small business than per-token

This is why the managed-hosting framing is starting to take over the SME conversation. The buyer doesn't want to think about API keys, model selection, or which sandbox their agent is running in. They want the POS-system experience: pay a flat fee, the agent works, somebody else is on the hook when it breaks.

If you're building this for yourself, the Builder Sandbox tier on RapidClaw gives you the MicroVM with sudo and live port-forwarding without the infra babysitting. If you're past the building phase and need something an operator can actually run unsupervised, that's the white-glove side of the same platform.

What I'd skip in 2026

For completeness — the workloads that keep showing up in pitch decks but quietly failing the trial-to-paid test:

The "AI sales rep" that prospects and closes by itself. The "AI manager" that runs your team's standup. The "autonomous research analyst" that reads ten papers and synthesizes a thesis. These will get there. They are not there yet. If you're trying to make rent this quarter, build the boring one.

The boring one is what's paying.

Tijo Gaucher runs RapidClaw, managed OpenClaw hosting for non-technical operators. Previously, content at Human + AI.

[I ran ONE AI agent for 30 days straight — here's what actually broke]

Tijo Gaucher — Thu, 30 Apr 2026 00:46:23 +0000

Most AI agent demos are shaped like a 90-second loop: prompt → tool call → answer. The interesting failures don't show up there. They show up around day 7, when the process you started in a tmux session has eaten 4 GB of RAM, your browser sub-agent is wedged on a captcha you never noticed, and the thing has been retrying the same failed Stripe webhook for 36 hours.

I ran a single OpenClaw agent on a small VPS for 30 days. It was scoped to one boring job: triage incoming sales emails, draft replies, file them in the right folder, ping Slack on anything weird. The agent ran continuously, scheduled by cron, with persistent state in SQLite. No multi-agent orchestration, no fancy memory layer — just one process trying to stay alive.

Here is what actually broke, in the order it happened.

Day 1–3: everything looks great

The first three days are a honeymoon. Latency is good, the agent handles edge cases I didn't think to specify, and the inbox triage rules quietly improve as it picks up patterns. This is where most demo videos end. It's also where most teams declare victory and move on, which is the mistake.

Two things to instrument before day 4 even starts:

Per-run token cost, written to a flat log. You'll need this when you investigate cost drift in week two.
Process RSS memory, sampled every minute. The number that matters isn't the peak — it's the slope.

If you're using a hosted setup like RapidClaw's managed OpenClaw runtime, the slope is graphed for you. If you're self-hosting, write the sampler yourself before you forget. You will forget.

Day 4: the context bloat starts

The agent's working memory file grew to 18,000 tokens. None of it was strictly wrong. It was just… accumulated. Old email threads it had handled, notes about edge cases, a half-finished plan for a problem that resolved itself two days earlier.

The cost per run had quietly tripled.

This is the most boring failure mode in long-running agents and the one nobody warns you about. Your prompt isn't getting worse — your context window is getting fatter. The fix is unglamorous: a compaction step that runs nightly, summarizes anything older than 48 hours into a few bullet points, and archives the rest to a file the agent can grep but doesn't auto-load.

If you skip this, by day 14 you're paying GPT-4-class prices to send the model a partially-decayed copy of last week's todo list every single run.

Day 7: the first silent kill

The OOM killer took the process at 3:47 AM. There was no error in the logs because the process didn't get to write one. It just stopped existing.

This is where most self-hosted agent setups quietly die in production and the operator doesn't notice for two days. The cron job that runs the agent every 15 minutes also exits cleanly when the process is gone — there's no parent supervising health.

Three things you want before day 7:

A liveness file the agent touches on every successful run, plus an external check that alerts when it's stale for more than 30 minutes.
A systemd unit (or equivalent) with Restart=on-failure and MemoryMax= set well below your VPS's actual RAM. You want the agent to die predictably and come back, not get reaped silently.
Logs that flush on every event, not on buffer fill. A buffered log is a log you don't have when the OOM killer arrives.

This is also the point where the "managed hosting" pitch starts to make economic sense for non-developers. Setting up systemd, a watchdog, log shipping, and metric scraping for one agent is two evenings of work for a competent backend engineer. SMEs don't have that engineer.

Day 11: the captcha trap

The agent's browser sub-task hit a captcha while loading a vendor portal. It didn't fail. It didn't error. It just waited. For 90 minutes. Then the headless Chrome process leaked and the next 14 runs spawned new Chrome instances on top of it.

The lesson is that anything involving a real browser needs both a hard wall-clock timeout and a "did the page actually finish loading the thing I asked for?" assertion. A 200 response is not a success signal when the body is a captcha challenge.

If your agent does any web automation at all, this will happen to you. The honest version of the agent demo isn't "watch it browse the web" — it's "watch the watchdog kill a stuck browser session and surface a human-readable reason for it."

Day 18: model drift on the provider side

The replies started getting weirdly formal. Not wrong — just off. I couldn't reproduce it on Claude with the same prompt locally, but in production the change was clear over a 3-day window.

Eventually I figured out the provider had silently routed a percentage of traffic to a slightly different model variant. This is a real thing that happens, and the only way you catch it is logging a stable hash of the prompt and the full response for every run, then diffing aggregates week-over-week. If you're not doing this, you'll just notice "vibes feel different" and have no evidence.

Day 24: the small bug that hid in the schedule

A timezone bug in the cron expression meant the agent ran exactly zero times for 18 hours during a holiday DST shift. Nobody noticed because there was no one in the inbox to notice. The triage queue piled up, and the agent's first run after the gap took 11 minutes and 92,000 tokens to dig out.

Schedules are infrastructure. Test them on a fake clock before you ship them.

Day 30: what stuck

The agent is still running. The job is unglamorous, the per-run cost is now lower than day 1 because of the compaction step, and most weeks I don't think about it. That's the real success criterion for a long-running agent: do you stop having to think about it?

The narrative around "ambient AI agents that do your whole job" is still mostly vibes. The agents that actually pay rent today are boring: scheduled jobs, browser automation, coding agents, inbox triage. They're sticky because once you have one running and supervised, the cost of replacing it is high. They're hard because the supervision is the actual product.

If you're a developer building these for yourself, lean into systemd, structured logs, and a 5-line health check. If you're not — or you're shipping this for non-technical operators who can't be on-call for a Python process — managed runtimes like RapidClaw exist precisely because day-7 reliability is a product, not a feature.

The demo is easy. The 30-day uptime is the moat.

Tijo writes about practical AI agents at Human + AI. RapidClaw is the managed-hosting side of the same operator-focused practice — built for people who want a working AI assistant without becoming a Linux admin.

[The 8-Turn Problem] Why Your Agent Fails at Turn 3 and You Only Notice at Turn 7

Tijo Gaucher — Mon, 20 Apr 2026 04:32:26 +0000

Last Tuesday an agent I shipped decided, mid-conversation, that the user's name was "Export CSV." It wasn't. Seven turns earlier, a tool result had come back with a quoted header row where a username field should have been, and the model silently absorbed that string as ground truth. Every subsequent turn degraded quietly — apologetic tone, subtle hallucinations, a refusal that referenced "your account, Export CSV."

The per-call logs looked fine. The latencies were green. Token usage was nominal. The only way to see the break was to reconstruct the whole conversation as a causal graph and follow the poison forward.

This is the 8-turn problem. It's the single most expensive class of bug I ship, and most of the observability stacks I've tried were built for a world where requests are independent. They aren't.

Why request-level monitoring lies

Traditional APM assumes a request is a closed unit: it came in, it did something, it came out, and if you aggregate p99 and error rate you know whether the system is healthy. That model was fine for stateless services. It's openly broken for agents.

An agent request carries state that isn't in the HTTP payload. It carries the conversation. It carries the tool results that previous turns wrote into context. It carries the model's own prior outputs, which are now training the next inference. A turn that looks locally correct — valid JSON, successful tool call, reasonable response — can be the exact moment your agent quietly goes off the rails for the next 40 minutes of user conversation.

I watch three numbers more than I watch latency:

Turn-over-turn intent drift: does turn N still match the user's original ask?
Tool result contamination rate: how often does a tool response contain strings that look like instructions?
Session success rate, not request success rate: did the user actually get what they came for?

None of those are visible from a metrics dashboard that aggregates individual calls. You need traces that span the whole session, and you need them structured so you can walk them backward from the failure.

What a useful trace actually looks like

The OpenTelemetry GenAI SIG has been converging on gen_ai.* semantic conventions, which is good. The prevailing shape: each tool call, each LLM invocation, each retrieval is a child span, parented to the turn, parented to the session. Do that, and your trace tree tells the story of the reasoning chain.

A few things people get wrong here:

Don't put prompts in span attributes. Attributes are indexed, have size caps, and leak straight into your observability backend as PII. Use span events. Events can be sampled, redacted, or dropped at the Collector without touching app code. This one change will save you a compliance conversation later.

Parent spans by the turn, not just the call. If every LLM call is a root span, you lose the conversational structure. The parent-child relationship between turn 3 and turn 7 is the thing you actually want to trace. If you're building this yourself, each session gets a trace_id, each turn gets a span_id under it, and tool calls and inferences nest under the turn.

Emit a "decision" span. The LLM call itself is one span, but what the agent did with the output — picked a tool, rephrased, escalated — is a different concern and worth its own span. This is where drift shows up first.

At RapidClaw we default to this layout and bolt on session-level rollups so you can ask "which turn did this fail at?" without scrolling through 40 spans.

The debugging workflow that actually works

When a user reports an agent did something weird, the temptation is to grep logs for the error. There's usually no error. Here's the loop I run instead:

Pull the full session trace. Not the failing turn — the whole conversation, from the first user message forward.
Diff the system state between turns. What changed in memory, in the scratchpad, in the retrieved context? This is where you find the poisoned field.
Replay from the suspected turn with the same tool responses. Most agent frameworks let you rehydrate a session; if yours doesn't, you need to fix that before anything else.
Mutate one variable at a time. Change the tool response. Change the model. Change the system prompt. Bisect until the behavior flips.
Write the regression test at the session level. Not a unit test on a single call — a full conversation fixture with expected final state.

Step 3 is where most teams stall. If you can't replay a session deterministically, you're guessing. The replay and re-simulate workflow is the single feature I'd build first in any agent observability tool, including ones I don't run.

Practical hygiene for small teams

I run a small operation — think five agents in production, not five hundred — and the infrastructure choices reflect that. A few things that have held up at this scale:

One OTLP pipeline, everything flows through it. Don't run a separate tracing stack for agents. Emit gen_ai.* spans into the same Collector your regular services use, then branch at the exporter if you want a specialized backend for LLM-specific analysis. Vendor lock-in is a real risk and OTel is the escape hatch.

Sample aggressively on success, keep everything on failure. Full-conversation traces are expensive. A 1% tail-based sampler plus 100% retention for sessions that flagged any of: tool error, user thumbs-down, abnormal turn count, or model refusal — that gives you the signal without drowning.

Tag sessions with the outcome, not just the request. Instrument your app to send a session-end event with "did the user get what they wanted?" If you can't answer that, instrument it first. Every other metric is downstream.

Treat evals and tracing as the same system. Evaluation runs are just traces with known expected outputs. The moment you split them into different tools you start writing glue code that never gets maintained.

The uncomfortable part

Most agent reliability issues I've seen in the last six months aren't model issues. They're context management issues. The model is doing its job — taking what's in the window and producing a plausible next token. The bug is upstream, in what we let accumulate in that window.

Observability for agents is, practically, observability for the context window over time. If your tooling can't show you how a single field mutated across seven turns, it can't help you debug the 8-turn problem. And the 8-turn problem is most of the bugs.

If you want to see how we handle session-level tracing in practice, the RapidClaw quickstart walks through instrumenting a LangGraph agent in about ten minutes. But the principle matters more than the tool: trace the session, not the request, and save yourself the compliance conversation by keeping prompts out of attributes.

Your agents are going to hallucinate. The question is whether you find out at turn 3 or turn 73.

Implementing A2A Protocol for Multi-Agent Communication

Tijo Gaucher — Sat, 18 Apr 2026 03:42:08 +0000

If you've ever wired two AI agents together, you know the drill. Custom JSON schemas, bespoke HTTP endpoints, and a growing pile of adapter code that nobody wants to maintain. Google's A2A (Agent-to-Agent) protocol is the answer to that mess, and I've been implementing it across OpenClaw and Hermes agents on Rapid Claw for the past few weeks. Here's what the implementation actually looks like.

What A2A solves (and what it doesn't)

A2A standardizes the message envelope between independent agents. Think of it as the TCP/IP of agent communication — it defines how agents discover each other, exchange structured messages, delegate tasks, and return results. It doesn't care what framework you're using internally.

The key distinction: MCP (Model Context Protocol) handles agent-to-tool communication. A2A handles agent-to-agent communication. You need both in any serious multi-agent deployment, and they compose cleanly because an A2A peer is essentially a tool with an agent on the other end.

The envelope format

Every A2A message carries the same required fields. The interesting bits go in payload:

envelope = {
    "a2a_version": "1.0",
    "message_id": f"msg_{uuid4().hex}",
    "correlation_id": "conv_01HZKXR7...",  # ties the conversation together
    "trace": {
        "trace_id": "4bf92f3577b34da6...",
        "span_id": "00f067aa0ba902b7",
    },
    "sender": {"agent_id": "planner-openclaw-prod-01", "framework": "openclaw"},
    "recipient": {"agent_id": "executor-hermes-prod-03", "framework": "hermes"},
    "intent": "task.delegate",
    "payload": {
        "task": "summarize_and_file",
        "inputs": {"url": "https://example.com/report.pdf"},
        "constraints": {"max_tokens": 4000, "deadline_ms": 30000}
    },
    "reply_to": "https://agents.rapidclaw.dev/a2a/planner/inbox",
    "expires_at": "2026-04-18T12:34:56Z"
}

Three fields do the heavy lifting: correlation_id threads multi-agent conversations into a single trace, trace carries OpenTelemetry-compatible span context so your existing APM stitches everything together, and intent is the verb recipients dispatch on — not a URL path.

Publishing an OpenClaw agent as an A2A endpoint

An OpenClaw agent becomes an A2A peer by exposing an inbox and registering with a platform registry. The agent doesn't need to know who will call it — only how to respond:

from fastapi import FastAPI, HTTPException
from openclaw import Agent, Task
from a2a import Envelope, verify_signature, sign

app = FastAPI()
planner = Agent.from_config("planner.yaml")

@app.post("/a2a/inbox")
async def inbox(envelope: Envelope):
    if not verify_signature(envelope, allowed=TRUSTED_SIGNERS):
        raise HTTPException(401, "signature verification failed")

    if envelope.intent == "task.delegate":
        task = Task(
            name=envelope.payload["task"],
            inputs=envelope.payload["inputs"],
            trace=envelope.trace,
        )
        result = await planner.run(task)

        reply = Envelope(
            intent="result.return",
            correlation_id=envelope.correlation_id,
            trace=envelope.trace,
            sender={"agent_id": AGENT_ID, "framework": "openclaw"},
            recipient=envelope.sender,
            payload={"status": "ok", "result": result.to_dict()},
        )
        return sign(reply, PRIVATE_KEY).dict()

The caller discovers executors by label, not URL — this is the part A2A gets right. No hardcoded hostnames:

executor = await lookup(
    intent="task.execute",
    labels={"framework": "hermes", "env": "prod"},
)

Three patterns worth implementing

Request/reply is the simplest. Planner calls executor, waits for the reply envelope, acts on it. Use for sub-tasks with clear deadlines.

Fan-out/fan-in dispatches the same intent to a pool of executors in parallel, correlates replies by correlation_id, and takes the first good answer or aggregates. This is how you build research-agent ensembles.

Async with callback fires a task.delegate with a reply_to URL and returns immediately. The callee POSTs a result.return when done. You get durability without holding an HTTP connection open.

The platform layer matters

The protocol is the easy part. Production A2A needs five things at the platform layer: a registry for discovery, identity and mTLS per agent, routing with network policy, observability that stitches traces across agents, and per-agent rate limits. You can build all five yourself — Postgres registry, Vault for keys, Envoy for mTLS, OTEL collector, Redis for rate limits — or use something like Rapid Claw that ships them preconfigured.

If you're thinking about multi-agent architectures more broadly, I wrote up the common orchestration patterns (planner/executor, supervisor, blackboard) that pair well with A2A as the transport layer.

A2A isn't revolutionary — it's the boring infrastructure piece that was missing. And boring infrastructure is exactly what you want when you're trying to ship agent systems that actually work in production.

[Patterns] AI Agent Error Handling That Actually Works

Tijo Gaucher — Fri, 17 Apr 2026 08:47:16 +0000

Most AI agent tutorials show the happy path. Your agent calls an LLM, gets a response, does the thing. Ship it.

Then production happens. Rate limits. Timeouts. Malformed responses. Context window overflows. Your agent goes from "demo-ready" to "incident-generating" in about 48 hours.

I run a small operation — 5 agents max, solo founder. Every failure that wakes me up at 3am is one I should have handled in code. Here are the patterns that actually work.

Classify Your Errors First

Not all errors deserve the same treatment. The first thing I do in any agent system is classify failures into two buckets:

Transient errors: Rate limits (429), timeouts, temporary network blips, model overload. These will probably work if you try again.

Permanent errors: Invalid API keys, malformed prompts, context window exceeded, model doesn't exist. Retrying won't help.

class ErrorClassifier:
    TRANSIENT_CODES = {429, 500, 502, 503, 504}

    @staticmethod
    def classify(error):
        if hasattr(error, 'status_code'):
            if error.status_code in ErrorClassifier.TRANSIENT_CODES:
                return "transient"
        if "timeout" in str(error).lower():
            return "transient"
        return "permanent"

This classification drives everything downstream. Transient errors get retries. Permanent errors get logged, reported, and gracefully degraded. When you're thinking about agent security patterns, error classification also matters — permanent auth errors need different alerting than transient network hiccups.

Retry Strategies That Don't Make Things Worse

The naive approach — retry immediately, retry forever — is how you turn a rate limit into a ban. Exponential backoff with jitter is the baseline:

import random
import time

def retry_with_backoff(fn, max_retries=3, base_delay=1.0):
    for attempt in range(max_retries):
        try:
            return fn()
        except Exception as e:
            if ErrorClassifier.classify(e) == "permanent":
                raise  # Don't retry permanent errors

            if attempt == max_retries - 1:
                raise

            delay = base_delay * (2 ** attempt)
            jitter = random.uniform(0, delay * 0.5)
            time.sleep(delay + jitter)

Key details: jitter prevents thundering herd when multiple agents hit the same limit. And always cap your retries — 3 is usually enough. If it hasn't worked in 3 tries, it's not going to work in 30.

Circuit Breakers for LLM Calls

Retries handle individual failures. Circuit breakers handle systemic ones. If your LLM provider is having a bad day, you don't want every request queuing up and timing out.

class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_time=60):
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.recovery_time = recovery_time
        self.last_failure_time = None
        self.state = "closed"  # closed = normal, open = blocking

    def call(self, fn):
        if self.state == "open":
            if time.time() - self.last_failure_time > self.recovery_time:
                self.state = "half-open"
            else:
                raise CircuitOpenError("Circuit breaker is open")

        try:
            result = fn()
            if self.state == "half-open":
                self.state = "closed"
                self.failure_count = 0
            return result
        except Exception as e:
            self.failure_count += 1
            self.last_failure_time = time.time()
            if self.failure_count >= self.failure_threshold:
                self.state = "open"
            raise

I wrap every external LLM call in a circuit breaker. When the circuit opens, agents fall back to cached responses or simpler logic instead of piling up failures. If you're taking an observability-first approach, you'll want to track circuit state transitions — they're one of the best early warning signals.

Fallback Chains: Your Safety Net

When your primary model fails, having a fallback chain prevents total outage:

FALLBACK_CHAIN = [
    {"provider": "anthropic", "model": "claude-sonnet-4-20250514"},
    {"provider": "openai", "model": "gpt-4o-mini"},
    {"provider": "local", "model": "cached_response"},
]

def call_with_fallback(prompt, chain=FALLBACK_CHAIN):
    errors = []
    for option in chain:
        try:
            return call_model(option["provider"], option["model"], prompt)
        except Exception as e:
            errors.append(f"{option['provider']}: {e}")
            continue
    raise AllProvidersFailedError(
        f"All {len(chain)} providers failed: {'; '.join(errors)}"
    )

The chain degrades gracefully: premium model → cheaper model → cached/static response. Your users get something even when everything is on fire.

Timeout Handling

LLM calls are slow. An agent waiting 120 seconds for a response that's never coming is wasting resources and blocking downstream work.

import asyncio

async def call_with_timeout(coro, timeout_seconds=30):
    try:
        return await asyncio.wait_for(coro, timeout=timeout_seconds)
    except asyncio.TimeoutError:
        raise TimeoutError(f"LLM call exceeded {timeout_seconds}s limit")

Set aggressive timeouts. For most agent tasks, if you haven't gotten a response in 30 seconds, something is wrong. I default to 30s for completions and 10s for embeddings.

Putting It All Together

Here's how these patterns compose in a real agent:

async def agent_execute(task):
    breaker = get_circuit_breaker("llm_calls")

    try:
        result = breaker.call(
            lambda: retry_with_backoff(
                lambda: call_with_fallback(task.prompt),
                max_retries=3
            )
        )
        return AgentResult(status="success", data=result)

    except CircuitOpenError:
        return AgentResult(
            status="degraded",
            data=get_cached_response(task),
            note="Using cached response - LLM circuit open"
        )
    except AllProvidersFailedError:
        return AgentResult(
            status="failed",
            data=None,
            note="All providers unavailable"
        )

The key insight: every layer has a defined failure mode. Timeouts prevent hangs. Retries handle blips. Circuit breakers prevent cascading failures. Fallbacks provide degraded-but-functional responses.

What I Track

Error handling is only useful if you know it's working. For my small setup, I track:

Error classification distribution — am I seeing more transient or permanent errors?
Circuit breaker state changes — how often are circuits opening?
Fallback chain depth — how far down the chain are requests going?
Retry success rate — are retries actually recovering errors?

Having real-time error monitoring changed how I build agents. Instead of finding out about failures from users, I catch patterns before they become outages.

The Boring Truth

None of these patterns are novel. Circuit breakers come from distributed systems. Retry with backoff is older than most of us. Fallback chains are just failover by another name.

But applying them specifically to AI agents — where failures are probabilistic, responses are non-deterministic, and costs compound with every retry — that's where the craft is. Start with error classification, layer on retries, add circuit breakers, and build fallback chains. Your 3am self will thank you.

[2026] OpenTelemetry for LLM Observability — Self-Hosted Setup

Tijo Gaucher — Fri, 17 Apr 2026 08:43:05 +0000

I've been running a small AI automation shop — just me, a handful of agents, and a self-hosted stack that needs to stay observable without blowing the budget. When I started instrumenting my LLM pipelines, I found that most observability guides assumed you'd use a managed platform. But if you're like me and prefer to own your data and infrastructure, OpenTelemetry gives you a solid, vendor-neutral foundation.

Here's what I've learned getting OpenTelemetry working for LLM agent traces on a self-hosted setup in 2026.

Why OpenTelemetry for LLM Workloads?

OpenTelemetry (OTel) has become the de facto standard for distributed tracing, metrics, and logs. The ecosystem matured significantly through 2025, and the semantic conventions for generative AI — covering LLM calls, token usage, model parameters — landed as stable in early 2026.

For LLM workloads specifically, OTel gives you a few things that matter:

Trace continuity across agent steps. When your agent calls an LLM, retrieves from a vector store, then calls another LLM, each step is a span in a single trace. You see the full chain, not just isolated API calls.

Token and cost attribution. The gen_ai semantic conventions include attributes like gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, which let you track per-request costs without bolting on a separate billing layer.

Vendor neutrality. Whether you're calling OpenAI, Anthropic, or a local model via vLLM, the instrumentation shape is the same. Swap providers without rewriting your observability code.

The Self-Hosted Stack

My setup is modest — a single VPS running the collection and storage layer, with agents deployed separately. Here's the architecture:

[Your LLM Agents]
       |
       v
[OTel Collector]  ← receives traces via OTLP/gRPC
       |
       v
[Tempo / Jaeger]  ← trace storage
[Prometheus]      ← metrics storage
[Grafana]         ← visualization

If you've looked at the self-hosted vs managed cost comparison, you know the economics are favorable when you're running fewer than five agents. The managed platforms charge per span or per seat, which adds up quickly even at small scale.

Setting Up the OTel Collector

The Collector is the central hub. It receives telemetry from your agents, processes it, and exports to your storage backends. Here's a minimal config for LLM traces:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512
  attributes:
    actions:
      - key: deployment.environment
        value: production
        action: upsert

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, attributes]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Nothing exotic here. The batch processor keeps things efficient, and we're exporting traces to Tempo and metrics to Prometheus. If you want a deeper walkthrough on getting this into production, the production deployment guide covers Docker Compose configs and health checks.

Instrumenting LLM Calls

The actual instrumentation depends on your language and SDK. I'll show Python since that's what most agent code runs on.

First, install the packages:

pip install opentelemetry-api opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-grpc \
  opentelemetry-instrumentation-requests

Then set up a tracer and wrap your LLM calls:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Initialize
provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint="http://your-collector:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("llm-agent")

def call_llm(prompt, model="claude-sonnet-4-20250514"):
    with tracer.start_as_current_span("llm.call") as span:
        span.set_attribute("gen_ai.system", "anthropic")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.max_tokens", 1024)

        response = your_llm_client.complete(prompt=prompt, model=model)

        span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
        span.set_attribute("gen_ai.response.model", response.model)

        return response.content

The key is using the gen_ai.* semantic conventions consistently. This means your Grafana dashboards, alerts, and queries work the same regardless of which model or provider you're hitting.

Tracing Multi-Step Agent Workflows

Where this gets really useful is tracing a full agent workflow. Each tool call, retrieval step, and LLM invocation becomes a child span:

def run_agent(task):
    with tracer.start_as_current_span("agent.run") as parent:
        parent.set_attribute("agent.task", task)

        # Step 1: retrieve context
        with tracer.start_as_current_span("retrieval.vector_search"):
            context = search_vector_store(task)

        # Step 2: call LLM with context
        result = call_llm(f"Context: {context}\nTask: {task}")

        # Step 3: maybe call a tool
        if needs_tool_call(result):
            with tracer.start_as_current_span("tool.execute") as tool_span:
                tool_span.set_attribute("tool.name", "web_search")
                tool_result = execute_tool(result)
                result = call_llm(f"Tool result: {tool_result}\nOriginal task: {task}")

        return result

When you view this in Grafana via Tempo, you get a waterfall trace showing exactly where time was spent — was it the vector search? The first LLM call? The tool execution? This is the kind of visibility that makes debugging agent behavior tractable instead of guesswork.

What You Actually See in the Dashboard

Once everything is wired up, your self-hosted observability dashboard shows you:

Latency breakdown per agent step — which spans are slow, and whether it's network or model inference
Token usage over time — catch runaway prompts before they drain your API budget
Error rates by model/provider — spot degraded model endpoints early
Trace search — find the exact trace where an agent went off the rails

For a solo operator running a few agents, this level of visibility is the difference between confidently shipping agent workflows and crossing your fingers every deploy.

Rough Edges and Honest Takes

A few things that are still annoying in 2026:

Auto-instrumentation for LLM SDKs is patchy. The OpenAI Python SDK has decent OTel support now, but Anthropic's is still experimental. You'll likely write some manual spans.

Trace volume can surprise you. Agents that loop — retries, multi-turn conversations — generate a lot of spans. Set up sampling early. A simple tail-based sampler that keeps error traces and samples 10% of success traces works well.

Grafana dashboards take time to build. The gen_ai semantic conventions are new enough that there aren't many pre-built dashboards. Budget an afternoon to set up your panels.

Wrapping Up

OpenTelemetry for LLM observability isn't a silver bullet, but it's the most practical foundation I've found for self-hosted setups. The semantic conventions are mature enough to use in production, the Collector is rock-solid, and the cost of running your own Tempo + Grafana stack is a fraction of what you'd pay for a managed platform.

If you're running a handful of agents and want to actually understand what they're doing, this stack is worth the setup time.

[Guide] How to Debug AI Agents in Production

Tijo Gaucher — Fri, 17 Apr 2026 08:42:31 +0000

I run a small outfit — a few AI agents handling tasks like lead qualification, document processing, and customer support triage. Nothing at massive scale. But even with just a handful of agents in production, debugging them has been one of the hardest parts of the job.

Traditional software bugs are predictable. An agent bug? It might only surface when a specific combination of user input, API latency, and model temperature aligns just right. Here's what I've learned about debugging AI agents in the real world.

The Problem with Agent Debugging

When a regular API endpoint fails, you get a status code and a stack trace. When an agent fails, you might get... a confidently wrong answer. Or a tool call loop. Or a response that technically works but costs $4.50 because it made 47 unnecessary API calls.

The core challenge is that agents are non-deterministic systems making autonomous decisions. You can't just write a unit test that covers every scenario. You need a different approach entirely.

Scenario 1: The Silent Wrong Answer

This is the scariest failure mode. Your agent completes its task, returns a result, and everyone moves on — except the result is wrong.

I had a document processing agent that was supposed to extract invoice amounts. It worked great for months until a client started sending invoices with a slightly different format. The agent still extracted numbers confidently, but they were line item totals instead of invoice totals. No error, no warning.

What helped: Adding assertion checks on agent outputs. Not just "did it return something" but "does this value fall within expected ranges." I also started logging the full reasoning chain so I could audit decisions after the fact. Having solid agent observability in place made it possible to catch these kinds of drift issues before they compounded.

Scenario 2: The Runaway Tool Call Loop

Agents that can call tools will sometimes get stuck in loops. Call tool A, get a result, decide it needs to call tool A again with slightly different parameters, repeat forever.

This usually happens when the agent's prompt doesn't clearly define exit conditions, or when a tool returns ambiguous results that the agent keeps trying to "fix."

What helped: Implementing hard limits on tool call counts per session. I cap mine at 15 calls per task — if an agent hits that limit, it stops and flags for human review. I also started using tracing to visualize the full sequence of tool calls. Being able to trace agent tool calls in a timeline view made it immediately obvious when an agent was spinning its wheels.

Scenario 3: Cascading Failures Across Agents

When you have multiple agents that depend on each other, a failure in one can cascade in unexpected ways. Agent A summarizes a document, Agent B uses that summary to make a decision, Agent C acts on that decision. If Agent A's summary is subtly off, you get a game of telephone that ends badly.

What helped: Treating agent handoffs like API contracts. Each agent validates its inputs before proceeding. I also added trace IDs that follow a request across all agents, so when something goes wrong at the end of a chain, I can trace it back to the originating agent.

Practical Log Analysis Patterns

Here are the patterns I actually use day-to-day:

1. Structured logging with context. Every agent action gets logged with: the task ID, the agent name, the tool being called, input parameters, output summary, latency, and token count. JSON-structured logs make it possible to query across all these dimensions later.

2. Diff logging for retries. When an agent retries a tool call, log what changed between attempts. This is usually where bugs hide — the agent is trying to correct something but its correction strategy is wrong.

3. Cost tracking per task. This might sound like a finance concern, not a debugging one, but unexpected cost spikes are one of the best early warning signals. If a task that normally costs $0.03 suddenly costs $0.30, something changed in the agent's behavior. I use a simple calculator to estimate debugging overhead costs and set alerts when any task exceeds 3x its rolling average.

4. Output sampling. Randomly sample 5-10% of agent outputs for human review. This catches the silent wrong answers that no automated check will find.

Handling Production Incidents

When something breaks in production with an agent, here's my playbook:

First, check the trace for that specific request. Look at every tool call, every decision point. Usually the problem is obvious once you can see the full sequence.

Second, check if the failure is reproducible. With agents, sometimes it is and sometimes it isn't — the same input might produce different behavior on the next run. If it's not reproducible, you need to look at what external state might have contributed (API responses, database state, etc.).

Third, check for upstream changes. Did an API you depend on change its response format? Did someone update the system prompt? Did the model provider do a quiet update? These are the most common root causes in my experience.

Tools and Setup That Actually Help

You don't need an elaborate observability stack. Here's what I actually run:

Structured JSON logs shipped to a searchable store
Trace IDs that propagate across agent boundaries
Hard limits on tool calls, tokens, and cost per task
Automated output validation with sensible thresholds
A weekly sample review of agent outputs

The key insight is that agent debugging is more like debugging a distributed system than debugging a single program. You need traces, not just logs. You need to see the full picture of what an agent decided, why, and what happened next.

Wrapping Up

Debugging AI agents in production is genuinely hard, and I don't think anyone has it fully figured out yet. But the basics — good logging, tracing, output validation, and cost monitoring — go a long way. Start with those, and add complexity only when you hit a problem that the basics can't solve.

If you're running agents in production too, I'd love to hear what patterns have worked for you. Drop a comment or find me on Twitter.

Self-Hosting AI Agents vs Managed: Honest Trade-offs From the Trenches

Tijo Gaucher — Tue, 14 Apr 2026 10:55:13 +0000

[Self-Hosting AI Agents vs Managed: Honest Trade-offs]
A few months in, I keep coming back to the same conversation with people building on agents: should you self-host, or just pay someone to run them for you? It sounds like a procurement question. In practice it's a question about how much weirdness you're willing to live with, and how much of the weirdness you want to be yours.

I run a small AI agent service called RapidClaw. My brother Brandon is the tech lead and we cap the number of concurrent agents we run at five — not as a marketing line, as an honest constraint. Five is the number where I can still look at every trace, name every memory key, and tell you what each agent did yesterday. Past that, I start lying to myself about what I actually understand. So I'd rather be small and clear than big and fuzzy.

That bias colors everything below. I'm not trying to talk anyone out of using a managed platform. I'm trying to write down the trade-offs the way I actually experienced them, in case it saves someone a weekend.

The honest pitch for managed

If you've never run an agent in production, start managed. I mean it. The boring stuff — retries, queueing, evals harness, secret rotation, log shipping, a UI someone other than you can use — is six to eight weeks of work that doesn't move your product forward. You're paying a managed provider to skip that, and skipping it is correct when the agent isn't yet the thing your customers love.

The catch is that "managed" is doing a lot of work in that sentence. There's managed-as-in-hosted (your prompts, their runtime), and there's managed-as-in-opinionated (their prompts, their runtime, their memory model). The second kind feels great in week one and starts to chafe in week six, when you realize you can't see why an agent decided what it decided, and your only recourse is a support ticket.

The questions I'd push on before signing anything:

Can I export every trace, every tool call, every memory write — as JSON, on demand, without a CSV button hidden three menus deep?
When a run fails, do I get the actual model response, or a sanitized "something went wrong"?
If I want to swap the underlying model next quarter, is that a config change or a rewrite?
What does the bill look like at 10x my current usage? At 100x? Is it linear, or is there a cliff at the "enterprise" tier?

If the answers are clean, managed is a fine home for a long time.

Why I ended up self-hosting anyway

For RapidClaw the deciding factor wasn't cost. It was the loop time on debugging. We were chasing a memory bug where an agent kept hallucinating a customer's preferred timezone. On a managed runtime I could see the final output and the tool calls, but not the actual sequence of memory reads the agent did before responding. Two days of poking later, Brandon stood up a small local runtime and we found it in twenty minutes — the agent was reading a stale snapshot because the memory write from the prior turn hadn't been flushed before the next read.

That's the kind of bug you can only catch when you can stop the world and look at it. Managed platforms are getting better at this, but "better" is not the same as "I can drop a print statement wherever I want."

The other thing that pushed us was the customer mix. Most of our customers want their agents running in their own VPC, on their own keys, talking to their own internal data. "Send your data to our SaaS" is a non-starter for them. So a self-hosted setup wasn't a nice-to-have — it was the product.

What self-hosting actually costs (the parts nobody warns you about)

Compute is the easy line item. Even the embarrassingly inefficient version of running five agents on a single mid-tier box costs less per month than one good lunch. That's not where the bill is.

Where the bill is:

Observability. You will reinvent some version of structured tracing for agent steps. Tool call in, model response out, memory delta, retry attempts, token counts. You can lean on OpenTelemetry, but the agent-shaped semantics are still yours to define. Budget two weeks the first time and another week every quarter to keep it honest.

Eval harness. Without a managed eval surface, you need to build the small, ugly version yourself. A folder of scenarios, a runner that hits each one, a diff viewer for outputs. It can be a hundred lines of Python. It cannot be zero lines of Python.

On-call. The first time an agent loops forever at 3am, you find out whether you have an on-call rotation. We didn't. Now we do. It's two people taking turns and a Pagerduty free tier, but it exists.

Memory state, specifically. This is the one I underestimated most. Agents that hold any state across turns — which is most useful agents — turn small bugs in your memory layer into very weird behavior in the model layer. I now spend more time thinking about how memory is read, written, snapshotted, and pruned than I spend thinking about prompts. If I were starting again, I'd build the memory inspector before I built the second agent.

The middle path most people end up at

Almost no team I've talked to runs a pure managed or pure self-hosted setup for long. The shape that keeps emerging is: managed for the orchestration and the model gateway, self-hosted for the memory and the tool layer. You give up a little observability on the orchestration side, you keep all the observability on the parts where bugs actually live.

That hybrid is what we ended up shipping for our own customers. The agent dashboard runs as a managed service so people don't have to host a UI, but the agents themselves run wherever the customer wants — their cluster, their keys, their VPC. It's not the cleanest architecture story, and it took us longer than I'd like to admit to stop apologizing for it.

What I'd tell past-me

Three things, none of them clever:

Pick the option that makes your debugging loop shorter, not the one that makes your slide deck better. If you can't see why an agent did what it did, you don't have an agent — you have a wishing well.
Cap the number of concurrent agents at a number you can mentally model. For us that's five. For a bigger team it might be twenty. It is almost certainly not "as many as the platform supports."
Write the boring runbooks early. The retry policy, the memory snapshot policy, the rollback procedure. They feel like overkill until the first real outage, after which they feel like the only adult thing in the room.

If any of this is useful, or if you want to compare notes on what you've broken, I'd love to hear about it — the RapidClaw team is small enough that you'll get an actual human, probably me or Brandon.

We're still figuring this out. I just wanted to write down what we've found so far, while it still feels true.

Why We Built a Managed Platform for OpenClaw Agents (And What We Learned)

Tijo Gaucher — Mon, 13 Apr 2026 02:41:43 +0000

We spent six months wrestling with deploying AI agents before we decided to just build the thing ourselves. This is that story — the ugly parts included.

The Problem Nobody Talks About

Everyone's building AI agents right now. The demos look incredible. You wire up some tools, connect an LLM, and suddenly you've got an agent that can research, plan, and execute tasks autonomously.

Then you try to put it in production.

Suddenly you're dealing with container orchestration, secret management, scaling workers up and down, monitoring token spend, handling failures gracefully, and figuring out why your agent decided to retry the same API call 47 times at 3am.

We were building on OpenClaw — an open-source agent framework that we really liked because it didn't try to do too much. It gave you the primitives and got out of the way. But "getting out of the way" also meant we were on our own for everything else.

What Running Agents in Production Actually Looks Like

Here's a simplified version of what our deploy pipeline looked like before RapidClaw existed:

# Our old "deploy an agent" workflow (simplified, but not by much)
steps:
  - name: Build agent container
    run: docker build -t agent-${{ agent.name }} .

  - name: Push to registry
    run: docker push $REGISTRY/agent-${{ agent.name }}

  - name: Update k8s deployment
    run: |
      kubectl set image deployment/$AGENT_NAME \
        agent=$REGISTRY/agent-${{ agent.name }}:$SHA

  - name: Configure secrets
    run: |
      kubectl create secret generic agent-secrets \
        --from-literal=OPENAI_KEY=${{ secrets.OPENAI }} \
        --from-literal=ANTHROPIC_KEY=${{ secrets.ANTHROPIC }} \
        # ... 12 more provider keys

  - name: Set up monitoring
    run: |
      # Prometheus config, Grafana dashboards, 
      # alerting rules, log aggregation...
      # This alone was 200+ lines of YAML

That's the happy path. We're not even talking about rollback strategies, canary deployments, or what happens when your agent starts hallucinating and burning through your API budget at 2x the normal rate.

We had an incident early on where an agent got stuck in a loop generating images. By the time we noticed, it had burned through about $400 in API calls in under an hour. That was our wake-up call.

Why OpenClaw

We evaluated a bunch of agent frameworks. Most of them wanted to own your entire stack — your prompts, your tool definitions, your execution model, everything.

OpenClaw was different. It's more like a protocol than a framework. You define your agent's capabilities, wire up your tools, and it handles the execution loop. But it's deliberately minimal about infrastructure opinions.

That minimalism is what attracted us, and also what made us realize there was a gap. OpenClaw gives you a great way to build agents. It doesn't give you a great way to run them.

What RapidClaw Does Differently

RapidClaw is basically the managed infrastructure layer that sits underneath your OpenClaw agents. Think of it as the platform that handles all the boring-but-critical stuff:

Deploy flow (what it looks like now):

┌─────────────┐     ┌──────────────┐     ┌─────────────────┐
│  Your Agent  │────▶│  RapidClaw   │────▶│   Production    │
│  (OpenClaw)  │     │   Platform   │     │   Environment   │
└─────────────┘     └──────────────┘     └─────────────────┘
       │                    │                      │
       │              ┌─────┴─────┐          ┌─────┴─────┐
       │              │ Secrets   │          │ Auto-scale │
       │              │ Mgmt      │          │ Monitor    │
       │              │ Isolation  │          │ Cost caps  │
       │              │ Versioning │          │ Rollback   │
       │              └───────────┘          └───────────┘
       │
  rapidclaw deploy my-agent --env production
  # That's it. One command.

The whole point is that you focus on your agent logic — what tools it has, how it reasons, what it's good at — and we handle the infrastructure. Secrets get injected securely, scaling happens automatically, and if your agent starts going off the rails, cost caps kick in before your cloud bill becomes a horror story.

You can dig into the security model if you want the details on how we handle isolation and secret management. It was one of the hardest parts to get right.

What We Learned (The Honest Version)

1. Agents fail in weird ways.

Traditional software fails predictably. API returns 500, you handle it. Database times out, you retry. Agents fail creatively. They'll find edge cases in your tools you never imagined. They'll interpret instructions in ways that are technically correct but completely wrong. Building good guardrails is less about error handling and more about understanding the problem space deeply enough to anticipate creative failures.

2. Cost management is a first-class concern.

This isn't like running a web server where your costs are roughly proportional to traffic. Agent costs can spike 10x in minutes if the agent decides it needs to "think harder" about something. We built per-agent budgets, per-session caps, and anomaly detection into the platform from day one. Should have done it from day negative-one.

3. Observability for agents is fundamentally different.

You can't just look at request/response logs. You need to see the agent's reasoning chain, understand why it chose one tool over another, and track how its behavior drifts over time. We built a trace viewer that shows the full execution tree — every tool call, every LLM interaction, every decision point. It's the feature our users care about most, and it was an afterthought in our original design. Embarrassing.

4. The open-source community taught us more than we expected.

We initially built RapidClaw as a purely internal tool. OpenClaw contributors kept asking us how we were running agents in production, and their questions shaped about 60% of our roadmap. Turns out the problems we were solving weren't unique to us — they were universal. That community feedback loop was the single most valuable thing in our development process.

5. You will underestimate state management.

Agents that run for minutes or hours need persistent state. They need checkpointing. They need the ability to resume after failures. And they need all of that without you having to think about it as an agent developer. Getting this right took us three complete rewrites. Three. We're still not 100% happy with it.

Where We Are Now

RapidClaw is running in production for a handful of teams. It's not perfect — our documentation needs work, our onboarding could be smoother, and there are definitely edge cases we haven't hit yet.

But the core loop works: write your OpenClaw agent, push it to RapidClaw, and it runs reliably in production with monitoring, scaling, and cost management built in. No more 200-line YAML files. No more 3am incidents because an agent went rogue.

If you're running OpenClaw agents (or thinking about it), I'd genuinely love to hear how you're handling the infrastructure side. We're at rapidclaw.dev/try if you want to kick the tires.

What's the gnarliest production issue you've hit with AI agents? I'll bet we've either seen it too or it'll end up on our roadmap. Drop it in the comments — I read every single one.

How I Cut Our AI Agent Token Costs by 73% Without Sacrificing Quality

Tijo Gaucher — Mon, 13 Apr 2026 02:16:50 +0000

Every month I'd open our cloud billing dashboard and wince. Running AI agents in production at RapidClaw meant our token costs were climbing faster than our revenue. Sound familiar?

After three months of aggressive optimization, we cut our monthly token spend by 73% while actually improving agent response quality. Here's exactly how we did it — no vague advice, just the specific techniques that moved the needle.

The Problem: Death by a Thousand Tokens

When you're running AI agents that handle real workloads — deployment automation, infrastructure monitoring, code review — every unnecessary token adds up. Our agents were processing ~2M tokens per day across various tasks. At GPT-4-class pricing, that's not pocket change.

The root causes were predictable once we actually measured:

Bloated system prompts copied-and-pasted across agents (avg 2,400 tokens each)
No caching layer — identical queries hitting the LLM every time
Redundant context stuffed into every request "just in case"
Wrong model for the job — using frontier models for classification tasks

Strategy 1: Prompt Compression (Saved ~30%)

The biggest win was the simplest. We audited every system prompt and applied aggressive compression.

# BEFORE: 847 tokens
SYSTEM_PROMPT_BEFORE = """
You are a helpful deployment assistant for our cloud infrastructure.
You should help users deploy their applications to our Kubernetes cluster.
You have access to kubectl commands and can help troubleshoot issues.
When a user asks you to deploy something, you should first check if 
the namespace exists, then validate the manifest, then apply it.
You should always be polite and professional in your responses.
You should explain what you're doing at each step.
If something goes wrong, provide clear error messages and suggestions.
Always confirm before making destructive changes.
Remember to check resource limits and quotas before deploying.
"""

# AFTER: 196 tokens
SYSTEM_PROMPT_AFTER = """
Role: K8s deployment agent.
Tools: kubectl
Flow: check namespace → validate manifest → apply
Rules: confirm destructive ops, check resource quotas, explain steps
"""

Same behavior, 77% fewer tokens. The key insight: LLMs don't need the verbose instructions we think they do. They need structured, precise constraints.

We built a simple compression pipeline:

import tiktoken

def audit_prompt(prompt: str, model: str = "gpt-4") -> dict:
    enc = tiktoken.encoding_for_model(model)
    tokens = enc.encode(prompt)

    # Flag prompts over 500 tokens for review
    return {
        "token_count": len(tokens),
        "needs_review": len(tokens) > 500,
        "estimated_daily_cost": len(tokens) * CALLS_PER_DAY * COST_PER_TOKEN
    }

# Run this on every agent prompt quarterly
for agent in get_all_agents():
    report = audit_prompt(agent.system_prompt)
    if report["needs_review"]:
        print(f"⚠️  {agent.name}: {report['token_count']} tokens "
              f"(${report['estimated_daily_cost']:.2f}/day)")

Strategy 2: Semantic Caching (Saved ~25%)

This was the highest-ROI engineering investment. We added a semantic similarity cache in front of our LLM calls.

import hashlib
import numpy as np
from redis import Redis

class SemanticCache:
    def __init__(self, redis_url: str, similarity_threshold: float = 0.95):
        self.redis = Redis.from_url(redis_url)
        self.threshold = similarity_threshold

    def get_embedding(self, text: str) -> np.ndarray:
        """Use a cheap embedding model — not the expensive LLM."""
        # text-embedding-3-small costs ~$0.02/1M tokens
        return embed_model.encode(text)

    def lookup(self, query: str) -> str | None:
        query_emb = self.get_embedding(query)

        # Check against recent cached queries
        for key in self.redis.scan_iter("cache:emb:*"):
            cached_emb = np.frombuffer(self.redis.get(key))
            similarity = np.dot(query_emb, cached_emb) / (
                np.linalg.norm(query_emb) * np.linalg.norm(cached_emb)
            )
            if similarity >= self.threshold:
                response_key = key.decode().replace("emb:", "resp:")
                return self.redis.get(response_key).decode()
        return None

    def store(self, query: str, response: str, ttl: int = 3600):
        key_hash = hashlib.sha256(query.encode()).hexdigest()[:16]
        emb = self.get_embedding(query)
        self.redis.setex(f"cache:emb:{key_hash}", ttl, emb.tobytes())
        self.redis.setex(f"cache:resp:{key_hash}", ttl, response)

The 0.95 similarity threshold was critical. Too low and you get stale/wrong cached responses. Too high and your cache hit rate tanks. We tuned this per agent type — deployment agents got 0.97 (precision matters), monitoring summarizers got 0.92 (more tolerance for variation).

Cache hit rates after one week:

Infrastructure status queries: 67% hit rate
Deployment validation: 41% hit rate
Code review suggestions: 12% hit rate (too unique, as expected)

Strategy 3: Model Routing (Saved ~18%)

Not every task needs a frontier model. We built a lightweight router that directs requests to the cheapest capable model:

MODEL_TIERS = {
    "classification": "gpt-4o-mini",     # $0.15/1M input
    "extraction": "gpt-4o-mini",          # Simple structured output
    "summarization": "gpt-4o",            # Needs nuance
    "reasoning": "gpt-4o",               # Complex decisions
    "code_generation": "claude-sonnet-4-6", # Best for code
}

def route_request(task_type: str, complexity_score: float) -> str:
    """Route to cheapest capable model based on task type and complexity."""
    base_model = MODEL_TIERS.get(task_type, "gpt-4o")

    # Override: bump up if complexity is high
    if complexity_score > 0.8 and base_model.endswith("mini"):
        return base_model.replace("-mini", "")

    return base_model

We score complexity using a fast heuristic — input length, number of distinct entities, presence of code blocks, and whether the request involves multi-step reasoning. The heuristic itself runs on the cheapest model as a pre-filter.

Strategy 4: Context Window Management

This one's underrated. Instead of dumping the entire conversation history into every request, we implemented a sliding window with smart summarization:

def prepare_context(messages: list, max_tokens: int = 2000) -> list:
    """Keep recent messages verbatim, summarize older ones."""
    recent = messages[-4:]  # Last 2 exchanges verbatim
    older = messages[:-4]

    if not older:
        return recent

    # Summarize older context with a cheap model
    summary = summarize(older, model="gpt-4o-mini")

    return [{"role": "system", "content": f"Prior context: {summary}"}] + recent

This alone saved 15-20% on our longer agent conversations without any measurable quality drop.

Measuring What Matters

None of this works without observability. We track three metrics for every agent:

Cost per successful task — not just cost per request
Quality score — automated eval comparing optimized vs. unoptimized outputs
Latency — cache hits are 50-100x faster than LLM calls

We built a simple dashboard that shows these per agent, per day. When cost-per-task creeps up, we investigate. When quality drops below threshold, we roll back.

At RapidClaw, we've baked these patterns into our agent deployment pipeline so every new agent starts with sane defaults — compressed prompts, caching enabled, model routing configured. It's not glamorous work, but it's the difference between an AI agent project that's a cost center and one that actually scales.

The Bottom Line

After implementing all four strategies:

Metric	Before	After	Change
Daily token spend	~2M	~540K	-73%
Monthly cost	$1,840	$497	-73%
Avg response latency	2.3s	0.8s	-65%
Task success rate	91%	94%	+3%

The latency improvement was an unexpected bonus — cache hits are basically free and instant.

If you're deploying AI agents and haven't optimized token costs yet, start with prompt compression. It's the fastest win with zero infrastructure changes. Then add caching. Then model routing. Each layer compounds on the last.

We're building more of these optimization primitives into the RapidClaw platform — if you're running agents in production and want to stop bleeding money on tokens, check it out.

I'm Tijo, founder of RapidClaw. I write about the unglamorous but critical parts of running AI in production. Follow me for more posts on agent ops, infra, and building startups with AI.