Your Rust Orchestrator Is Oxidizing From The Inside

Here's what's breaking.

You've got oxidizedGraph running as your DAG orchestrator. Rust, tokio, axum, kube-rs. It's been fine for six months. Then last Tuesday it starts leaking goroutines-equivalent. Tasks pile up. kube-rs watchers reconnect in a loop. Your AKS node pool hits 90% memory and stays there.

Nobody changed anything. That's the fun part.

Why It Happens

Three things rot a long-running Rust orchestrator in k8s:

1. tokio::spawn without a JoinHandle. Fire-and-forget tasks. When the parent scope drops, the child keeps running. Do this in a retry loop and you get thousands of orphaned tasks holding connections.

// ❌ Leaks. Nobody owns this task.
tokio::spawn(async move {
    reconcile_pod(pod_name).await;
});

2. kube-rs watcher streams that never get a fresh Client. The default client holds a connection pool. Under RBAC churn or API server restarts, those connections go stale. Watchers reconnect, but the old ones don't always drop cleanly.

3. Crossplane composite resources reconciling in a tight loop. If your Composition has a status field that the orchestrator writes back, and the orchestrator also watches that status, you get a feedback loop. CPU goes brrr.

Sound familiar?

Manual Fix (Do This Today)

Bound your task lifetime. Use JoinSet:

use tokio::task::JoinSet;

let mut set = JoinSet::new();

for pod in pods {
    set.spawn(async move { reconcile_pod(pod).await });
}

// Drain with a ceiling. Don't wait forever.
while let Some(res) = set.join_next().await {
    if let Err(e) = res {
        tracing::error!(?e, "reconcile task failed");
    }
}

For the watcher, force a client refresh on every Restart event:

use kube::Client;
use futures::StreamExt;

let mut backoff = 1u64;

loop {
    let client = Client::try_default().await?;
    let stream = watcher(api.clone(), watcher::Config::default());

    tokio::pin!(stream);

    while let Some(event) = stream.next().await {
        match event {
            Ok(_) => { backoff = 1; }
            Err(e) => {
                tracing::warn!(?e, backoff, "watcher error, reconnecting");
                tokio::time::sleep(Duration::from_secs(backoff)).await;
                backoff = (backoff * 2).min(60);
                break; // rebuild client + stream
            }
        }
    }
}

For the Crossplane loop, add a generation check. Only reconcile when metadata.generation changes:

if resource.metadata.generation == resource.status.as_ref()
    .and_then(|s| s.observed_generation) {
    return Ok(Action::requeue(Duration::from_secs(300)));
}

This stops the spin. It doesn't tell you why generation was bumping in the first place.

The Part That Actually Hurts

You fix the leak. You deploy. Two days later a different agent — the A2A one talking to dockworker.ai — starts returning malformed task graphs. The orchestrator dutifully schedules them. Nothing errors. Nothing logs. The DAG just quietly does the wrong thing for six hours.

Now you're grepping tracing output across three clusters trying to reconstruct what the agent meant to send.

This is the part where traditional observability gives up. You've got metrics. You've got spans. You don't have the actual task graph the agent produced at 14:23:07 UTC, before your parser normalized it.

With TracePilot

You wrap the agent boundary once:

import { TracePilot } from 'tracepilot-sdk';

const tp = new TracePilot(process.env.TRACEPILOT_API_KEY!);

export async function planGraph(input: string) {
  await tp.startTrace('oxidizedgraph-planner');

  const { result, spanId } = await tp.wrapOpenAI(
    () => openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [{ role: 'user', content: input }],
    }),
    [{ role: 'user', content: input }]
  );

  return { graph: result.choices[0].message.content, spanId };
}

Then when a malformed graph slips through, you open the dashboard, find the span, hit Fork & Rerun, edit the prompt, and see what the agent produces now — against the same input that broke it. No redeploy. No "can you reproduce it?"

You get the raw agent output before normalization. That's the thing you've been missing.

One More Thing

Your dockworker.ai integration is probably fine. Your Crossplane compositions are probably fine. The bug is almost always at the seam between them — where one system's output becomes another's input, and nobody's watching that boundary.

Instrument the seams first. Everything else is noise.

---

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord