DEV Community: pranav-afk

Traced a Multi Agent Failure Back 6 Steps. Here's the Instrumentation That Made It Possible.

pranav-afk — Mon, 17 Aug 2026 15:56:41 +0000

Last week an agent in a production pipeline I was debugging did something that looked, on the surface, completely irrational. Correct tool call, correct parameters, valid output, but the decision to make that call at all was wrong. It took six steps of tracing backward to find out why.
Here's the actual root cause, and why most observability setups wouldn't have caught it.
The setup: a multi agent pipeline where Agent A researches a topic, writes findings to shared memory, and Agent D (four hops later, unrelated task) reads from that same memory scope because the key happened to overlap. Agent A's findings were accurate when written. By the time Agent D read them, the underlying data had changed. Agent D reasoned perfectly, off information that was stale by the time it mattered.
Why this is hard to catch:
The failing agent's logs look completely normal, valid input, valid reasoning, valid output
Standard tracing shows what happened at each step, not when a piece of context was written versus when it was consumed
Nobody flagged it because no individual step was wrong, the failure only exists in the relationship between two steps that happened at different times
What actually solved it:
Timestamp every memory write and read separately, and diff the gap when a failure investigation starts, a large gap between write and read is often the first real clue
Scope memory access explicitly rather than relying on implicit key matching, Agent D shouldn't have been able to see Agent A's write in the first place if the scopes were correctly separated
Make replay possible from any single node, so you can re-run just the suspect step with the memory state as it existed at read time, not the current state
This kind of cross temporal bug is becoming more common as pipelines get longer and memory gets shared across more agents. Most tracing tools are built for single agent debugging and don't surface this class of issue well.
We ended up building this kind of instrumentation into Cartha, our own agent governance layer, after running into this exact problem across a few different customer pipelines. Curious if others have hit similar cross temporal failures, or if this is more of an edge case in typical setups.

Debugging Multi-Agent Systems: Why "It Works" Isn't Enough Anymore

pranav-afk — Mon, 17 Aug 2026 02:51:42 +0000

As agent systems move from single-chain prototypes to multi-agent production pipelines, a new failure mode has become common: the agent technically did what it was told, but the outcome is still wrong.
Debugging a single LLM call is straightforward. You look at the prompt, the response, done. Debugging a multi-agent system is a different problem entirely — a request that gets handed off through 4-5 agents can fail because:
The wrong tool got selected (but the tool call itself succeeded)
Retrieved context was accurate but stale
One agent wrote to a memory scope that a different agent read from later, for an unrelated reason
The final output looks fine, but the reasoning path that produced it wasn't
The core issue: most tracing tools log what happened, not why.
A trace showing "Agent B called Tool X with these params" tells you the call happened. It doesn't tell you why Agent B decided to call Tool X, what context it believed was true at that moment, or whether that context was still valid.
What actually helps:
Tag memory events with intent, not just content — log why a write happened, not just what was written
Make every step replayable in isolation — you should be able to fork a run from any node and re-execute just that segment
Separate "tool call succeeded" from "tool call was the right call" — these get conflated in most logging setups, and it's the second one that actually matters for debugging
We ran into this building Cartha, a governance and observability layer for AI agent fleets, and it changed how we think about tracing — treating memory scope and decision rationale as first-class, queryable data rather than something buried in a log line.
Curious how others are handling this — are you building custom instrumentation, or leaning on LangSmith/Langfuse and accepting the gap?

How to Add Traces, Hard Budgets, and Scoped Memory to AI Agents in Python

pranav-afk — Sun, 02 Aug 2026 04:36:34 +0000

If you ship multi-step AI agents with tools and LLMs, you eventually need more than print statements: traces, cost ceilings, tool permissions, and memory that does not leak across users.

This guide shows a practical Python integration pattern for AI agent observability and governance using the open Cartha platform and the cartha-sdk package.

Official product and dashboard: https://cartha.in
Docs: https://cartha.in/documentation
Pricing / trial: https://cartha.in/pricing
Login / API keys: https://cartha.in/login

What is an AI agent control plane?

An AI agent control plane sits beside your agents (CrewAI, LangGraph, OpenAI, custom code). You keep building agents as usual. The control plane:

Records execution traces (tools, LLM calls, errors)
Enforces hard budget breakers (stop runaway spend)
Applies tool allow-lists (block unauthorized tools before they run)
Stores scoped memory (user / agent / team / org) with server-side isolation
Exposes an ops dashboard for agents, traces, memory, and costs

Cartha is built as that control plane for production agent systems—not a replacement for your agent framework.

Why Python AI agents need observability and budgets

Search traffic and real teams ask the same questions:

Problem	What breaks without a control plane
Debugging	“Which tool failed?” needs step-level traces
Cost	Agent loops can burn API spend overnight
Safety	Support bots must not call wire-transfer tools
Multi-tenant apps	Customer A memory must never appear for Customer B
Audit	Compliance needs “what did the agent know?”

Logging alone is not enough. You want server-enforced budgets, policies, and scopes—visible at cartha.in.

Prerequisites

Python 3.10+
Free workspace + API key from Cartha login
Optional: OpenAI key if you use wrap_openai()

Step 1 — Install Cartha SDK

pip install cartha-sdk
# optional
pip install openai

export CARTHA_API_KEY="cartha_..."           # from https://cartha.in → Settings / Keys
export CARTHA_API_BASE="https://cartha.in"   # same host as the product

Full install notes and examples live in the Cartha documentation (https://cartha.in/documentation).

───

Step 2 — Minimal Python integration (copy-paste)

This is the core AI agent instrumentation pattern: init → tool → trace → optional memory + OpenAI wrap.

```import os
import cartha

Connect to your Cartha workspace (https://cartha.in)

cartha.init(
api_key=os.environ["CARTHA_API_KEY"],
api_base=os.environ.get("CARTHA_API_BASE", "https://cartha.in"),
)

Auto LLM + cost steps for OpenAI chat completions

client = cartha.wrap_openai()

@cartha.tool()
def crm_lookup(user_id: str) -> dict:
"""Recorded as a tool step; blocked if not on allowed_tools."""
return {"user_id": user_id, "plan": "pro"}

@cartha.trace(
id="support_agent",
team="support",
budget_usd=0.50, # hard agent-run budget
allowed_tools=["crm_lookup"], # tool allow-list (authority)
)
async def handle_ticket(user_id: str, ticket: str) -> str:
# Scoped memory — user scope is isolated per customer
await cartha.remember(
user_id=user_id,
content=f"Ticket: {ticket}",
scope="user",
confidence=0.9,
)
hits = await cartha.recall(
user_id=user_id,
context=ticket,
scope=["user", "team"],
top_k=5,
)

data = crm_lookup(user_id)

r = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{
        "role": "user",
        "content": f"{ticket}\nCRM: {data}\nMemory: {hits}",
    }],
)
return r.choices[0].message.content or ""

Run the function once, then open the Cartha dashboard (https://cartha.in):

• Agents — registration / heartbeats
• Traces — step-by-step run replay
• Memory — scoped store/recall
• Costs — spend feeding budget breakers

───



### Step 3 — Hard agent budgets (stop runaway spend)



```from cartha import BudgetExceeded

@cartha.trace(id="support_agent", team="support", budget_usd=0.01)
async def risky_run(user_id: str) -> str:
    # ... LLM / cost events ...
    return "ok"

try:
    await risky_run("user_123")
except BudgetExceeded as e:
    print("Cartha stopped the run:", e)

This is an agent-run budget (per @cartha.trace), not “hope the model stops.” Details: cartha.in/documentation (https://cartha.in/documentation).

───

Step 4 — Tool allow-lists (authority before execution)

@cartha.trace(
    id="support_agent",
    team="support",
    allowed_tools=["crm_lookup"],  # wire tools NOT listed
)
async def support_only(user_id: str):
    return crm_lookup(user_id)
    # calling a non-listed @cartha.tool → ToolNotAuthorized (before body runs)

Support can help customers; it cannot “accidentally” call privileged tools if those tools are outside the allow-list. That’s AI agent governance, not a prompt suggestion.

───

Step 5 — Scoped memory (multi-tenant isolation)

┌───────┬───────────────────────────────┐
│ Scope │ Who can use it │
├───────┼───────────────────────────────┤
│ user │ Memories for that user_id │
├───────┼───────────────────────────────┤
│ agent │ Bound to agent identity │
├───────┼───────────────────────────────┤
│ team │ Shared within a team │
├───────┼───────────────────────────────┤
│ org │ Org-wide (when policy allows) │
└───────┴───────────────────────────────┘

Isolation is enforced on the Cartha API, not only in client code. Product overview: https://cartha.in (https://cartha.in).

Empty Memory Isn’t Missing Data — It’s a Permission Hole

pranav-afk — Sun, 26 Jul 2026 13:48:08 +0000

Definition: In multi-agent AI systems, an empty memory recall often means this agent is not allowed to see that fact — not the fact does not exist. If your stack turns both cases into [], the model will invent over a permission hole.

That problem is one reason we built Cartha — a governance-first agent operations platform (scoped memory, traces, hard budgets, cost attribution). Full setup guide: How to Use Cartha.

The multi-agent failure mode

Single-agent demos treat memory like a vector DB:

embed → top-k → stuff into the prompt → hope.

That breaks as soon as you have:

more than one agent
more than one user
more than one team

Example

Support agent
Finance agent
Shared user_id

Support asks: “What’s this customer’s plan and last payment?”

If finance memory is out of scope for support, a naive stack returns nothing.

The support agent then:

assumes nothing is known
invents a plan
or calls tools it shouldn’t

You only notice when the customer is wronged — or when someone asks:
“What did the agent know when it said that?”

Silent empty vs explicit withhold

Mode	Agent sees	Risk
silent	Empty hits	Hallucination over a permission hole
denied_hint	Explicit “matched but withheld” (no content)	Agent can refuse to invent

For fleets, explicit withhold is the better default.

Not perfect privacy (existence can leak).
But silent empty trains agents to fake authority.

A good denial sounds like:

A finance-scoped memory matched, but this agent cannot read that scope. Do not invent a substitute — treat this as withheld context.

That’s a permission event, not missing data.

On Cartha, org default denial mode is denied_hint (silent is opt-in in Settings). Details: cartha.in/how-to-use.

Scopes that actually mean something

Scope	Who can recall
`user`	Agents serving that user
`agent`	Only this agent
`team`	Agents sharing a team id
`org`	Whole organization

The enum is not the product.

Enforcement is. If the agent can query a raw index with broader credentials, “scopes” are cosplay.

Cartha enforces scopes server-side on store/recall — not only in a system prompt.

What to log (for humans and for ops)

When recall is denied, log:

agent id
requested scopes
denial reason
that content was withheld (not the secret content)

Otherwise you only get “the model made something up” with no path to why.

This pairs with multi-agent debugging: wrongness often starts at a boundary (handoff or memory boundary), not at the final sentence.

Minimal pattern (any framework)


python
# Store with an explicit scope
await remember(user_id=uid, content="Prefers email", scope="user")

# Recall — still server-clamped
hits = await recall(
    user_id=uid,
    context="contact preference",
    scope=["user", "team"],
)

# Prefer structured denials, not only empty lists
# so the agent branches: "no access" vs "no data"

Empty memory isn’t missing data. It’s a permission hole.

pranav-afk — Sat, 25 Jul 2026 15:28:39 +0000

Definition: In multi-agent systems, an empty memory recall often means this agent is not allowed to see that fact — not the fact does not exist. Collapsing both into [] is how agents invent over permission holes.

This is one of the core problems we design for at Cartha — an ops/governance layer for AI agent fleets (scoped memory, traces, costs, hard budgets). Docs: How to Use Cartha.

The failure mode

Most agent stacks treat memory like a vector DB with vibes:

retrieve top-k → stuff into the prompt → hope the model behaves.

That works for a single demo agent.

It fails when you have more than one agent, more than one user, or more than one team sharing infrastructure.

Classic setup:

Support agent
Finance / billing agent
Shared user id or “company memory”

Support asks: “What’s this user’s plan and last payment?”

If finance-scoped memory is out of scope for support, a naive stack returns nothing.

The support agent then:

assumes nothing is known
invents a plan
or calls tools it shouldn’t

You notice when a customer is wronged — or when someone asks: what did the agent know when it said that?

Silent empty vs explicit withhold

Two denial modes:

Mode	What the agent sees	Risk
silent	Empty hits	Agent invents over a permission hole
denied_hint	Explicit “matched but withheld” (no content)	Agent can refuse to invent; may leak existence

For agent fleets, explicit withhold is the better default.

Not perfect privacy (existence can leak).
But silent empty trains models to hallucinate authority they don’t have.

A good denial sounds like:

A finance-scoped memory matched, but this agent cannot read that scope. Do not invent a substitute — treat this as withheld context.

That’s a permission event, not missing data.

Scopes that mean something

Scope	Who can recall
`user`	Agents serving that user
`agent`	Only this agent (private)
`team`	Agents sharing a team id
`org`	Whole organization

The critical part isn’t the enum names.

It’s enforcement on the server — not “please only request allowed scopes” in a system prompt.

If the agent can query a raw index with broader credentials, your “scopes” are cosplay.

At Cartha, memory scopes are enforced server-side; org default denial mode is denied_hint (silent is opt-in). See the full walkthrough: cartha.in/how-to-use.

Minimal shape (any stack)


python
# Store with an explicit scope
await remember(user_id=uid, content="Prefers email", scope="user")

# Recall — requested scopes still get server-clamped
hits = await recall(
    user_id=uid,
    context="contact preference",
    scope=["user", "team"],
)

# Prefer structured denials, not only empty lists
# so the agent can branch: "no access" vs "no data"

The bug was at step 2. You noticed it at step 5.

pranav-afk — Fri, 24 Jul 2026 15:21:56 +0000

Single-agent debugging is mostly a solved shape: see the prompt, see the tool call, see the output.

Multi-agent graphs are different. The pattern that keeps biting teams:

Agent A hands off to agent B
B calls a tool
Result goes back to A
A decides something based on it
Three steps later the final answer is clearly wrong

The failure is usually not at step 5. The wrongness happened at the handoff (step 2). By the time you stare at the final output, you’re reconstructing the chain from logs by hand.

That is the multi-agent debugging tax.

Why handoffs break silently

In a single chain, one component owns the state story.

In a multi-agent run, state is often implicit:

A logs: “I passed the right thing.”
B logs: “I received something reasonable.”
Both look “correct” in isolation.
The gap between what A intended and what B actually operated on is where silent failures live.

If you only log each agent’s own outputs, that gap never shows up.

Three things that help (before perfect tooling)

1. Treat the handoff as a first-class event

Log more than “agent A finished” / “agent B started”:

decision to hand off (why)
payload on the wire
what the receiver was told
what the receiver could access (tools, memory scopes, permissions)

That last line matters. Payload can match while authority still diverges (B can’t read the memory A assumed it could).

2. Prefer step inspection over full re-run

Re-running the whole graph to debug is slow and dangerous when tools have side effects (email sent, ticket updated, money moved).

What you want first is: jump to the handoff step and inspect state there.

3. Detect divergence early (cheap)

A useful pattern from practice: snapshot at the boundary.

Sender snapshot: what A put on the wire
Receiver snapshot: what B bound as working state before its first tool/LLM call
Diff them

If they diverge, you catch it at step 2, not step 5. Cheaper than full replay: compare snapshots, don’t re-execute the graph.

Fork-from-node (test a different tool/policy from that state without replaying upstream) is the recovery half. Detection is the half most teams never build.

What this means for “agent observability”

Generic “show me LLM spans” helps single chains.

Multi-agent systems need boundary-aware ops:

nested / linked runs across agents
what memory was visible at a decision
what tools were authorized
cost and retries rolled up per logical task

That’s the direction we’re building toward with Cartha (SDK-first: traces, scoped memory, budgets, multi-agent nest). We’re still early on full fork-from-node and automatic handoff divergence probes — and we’re honest about that.

If you only take one idea away: log the handoff, not just the agents.

Questions for you

If you run multi-agent graphs (LangGraph or otherwise):

Do you log handoffs as first-class events, or only per-agent I/O?
When you “replay,” is it full re-run or step-level inspect?
Have you built any check that sender payload == receiver interpreted state?

Curious what’s worked (or failed) for you in production.

Docs / try: cartha.in/how-to-use

Soft Alerts Don’t Stop Agent Spend. Hard Budgets Do.

pranav-afk — Wed, 22 Jul 2026 13:23:32 +0000

Soft alerts are comforting.
They are also how you wake up to a surprise OpenAI bill from a retry loop at 3am.

If you run multi-step agents (tools + LLM calls), you need two systems:

Observability — what happened (traces, tokens, errors)
Enforcement — what is allowed to happen next (budget, tools, memory scope)

Most stacks are strong at (1) and weak at (2).

Soft alerts vs hard budgets

Approach	What it does	Failure mode
Alert at 80% of monthly spend	Email / Slack	Agent keeps running
Dashboard “cost this week”	Human looks later	Overnight burn
Hard budget on the run	Reject next spend when ceiling hits	Run stops; you debug

Hard budgets only work if LLM/tool cost is recorded on the hot path (or immediately after each call). A run with only “start/finish” spans cannot enforce real dollars.

Minimal mental model

Agent run starts with budget_usd = 0.50
→ tool call
→ LLM call (+ tokens → $)
→ if spent >= budget → fail closed (raise / stop)
→ else continue

Enforcement should not be “we'll audit later.”
If the gate is only after the wire transfer (or the refund email), it's already too late.

Python sketch (pattern)

You can implement this in app code. The important parts:

Attach a budget to the run (not only org monthly total)
Record every LLM call’s estimated cost
On exceed: raise (or return a controlled error) — don’t log-and-continue


python
class BudgetExceeded(Exception):
    pass

class RunBudget:
    def __init__(self, limit_usd: float):
        self.limit = limit_usd
        self.spent = 0.0

    def charge(self, usd: float) -> None:
        if self.spent + usd > self.limit:
            raise BudgetExceeded(
                f"spent={self.spent:.4f} + {usd:.4f} > limit={self.limit}"
            )
        self.spent += usd

Wire that into your OpenAI wrapper after each completion (use usage tokens × your price table).

Hello DEV, I build ops for AI agent fleets

pranav-afk — Mon, 20 Jul 2026 09:37:26 +0000

Hi, DEV 👋

I'm part of the team building Cartha — an ops layer for AI agents that actually run in production.

This is a short intro: the problem we kept hitting, what we shipped, and what I'll write about here.

The problem

We were debugging an agent that did something inexplicable in production. Logs and traces answered what ran. They didn't answer the question that mattered:

What did the agent know when it made that call — and was it allowed to know that?

If you run multi-agent systems, you eventually need more than a pretty timeline:

Need	Why it matters
Scoped memory	User vs agent vs team vs org — without leaks
Hard budgets	Soft alerts don't stop a retry loop at 3am
Tool limits	Child agents shouldn't inherit the keys to the kingdom
HITL	Some actions need a human before they execute

Observability tells you what happened. Governance decides what is allowed to happen next.

What Cartha is

Not a chatbot builder. You keep your own agents (OpenAI, tools, business logic).

Yes a control plane for fleets:

Run timelines (tools + LLM steps)
Server-enforced memory scopes (user / agent / team / org)
Fail-closed spend budgets on a run
Nested agents + attenuated delegation
Policies + escalations
Dashboard for agents, traces, cost, memory

Minimal Python path


python
import cartha

cartha.init()  # CARTHA_API_KEY + CARTHA_API_BASE=https://cartha.in
client = cartha.wrap_openai()  # auto LLM steps + cost

@cartha.tool()
def crm_lookup(user_id: str) -> dict:
    return {"plan": "pro"}

@cartha.trace(id="support", team="support", budget_usd=0.5)
def handle(user_id: str, ticket: str) -> str:
    data = crm_lookup(user_id)
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"{ticket}\n{data}"}],
    )
    return r.choices[0].message.content or ""

Docs: How to Use (https://cartha.in/how-to-use)
Product: cartha.in (https://cartha.in)