Untitled

Here's the thing that's been eating at me about LLM inference. We've hit a wall where raw parameter count isn't the bottleneck anymore — it's the token budget. You've got a model that's technically capable of complex reasoning, but it burns its entire context window on intermediate steps before it gets to the actual answer.

Here's what's breaking: models like SR²AM at 8B parameters are showing effective intelligence in the 120–355B range. That's not a typo. The trick is simulative planning — the model runs internal simulations, explores branches, prunes dead ends. Sounds great. Except it's eating 95% of your token budget just on the planning phase.

Sound familiar? You've seen it. The agent that spends 47 tool calls "thinking" before it actually does anything useful. The response that's 4,000 tokens of reasoning and 200 tokens of answer. The bill that makes your finance team twitch.

Why does this happen? Because simulative planning is computationally expensive by design. The model isn't just generating text — it's maintaining multiple hypothetical states, comparing outcomes, backtracking. Each branch costs tokens. Each rollback costs more. The actual "intelligence" — the pattern matching, the synthesis, the final output — only happens after all that simulation overhead.

Traditional debugging won't catch this. Your logs show the model "worked." Tokens were consumed. The response was generated. But you can't see where the budget went. You can't replay the branching decisions. You can't identify the step where the model went down a rabbit hole for 8,000 tokens.

Here's the manual fix. If you're using Vercel AI SDK, you can instrument your calls to track token consumption per step:

import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";

const result = streamText({
  model: openai("gpt-4o"),
  messages,
  onStepFinish: async (step) => {
    const tokens = step.tokens;
    const stepType = step.type;

    if (tokens > 5000 && stepType === "reasoning") {
      console.warn(`⚠️ Token bloat at step ${step.id}: ${tokens} tokens`);
      // Log the full step context for later analysis
      await logStepContext(step);
    }
  },
});

This works. You'll see the problem. But you still have to manually trace through the execution tree, figure out which branch was the problem, and re-run with different prompts. That's hours of work per incident.

Guess what happens next? You get a production incident at 2 AM. The agent looped on a planning branch for 14 minutes, burned $45 in tokens, and returned "I couldn't complete the task." You need to see the exact execution state, fork it at the failing step, and test a fix — without redeploying.

This is where TracePilot changes the game. One line change:

import { tracepilot } from "@/lib/tracepilot";

const result = streamText({
  model: openai("gpt-4o"),
  messages,
  ...tracepilot.instrument(),  // ← captures execution tree, token usage, and branching
});

Now every planning branch, every rollback, every token is captured as a structured trace. When the agent fails, you open the dashboard, find the exact span where the budget exploded, and hit Fork & Rerun. Edit the prompt at that specific step. See the new output instantly. No reproduction steps. No redeployment.

The token gap isn't going away. Simulative planning is the future — but the observability layer has to catch up. Your agents are making thousands of decisions per run. You need to see each one, rewind to the bad ones, and fix them surgically.

Stop guessing which step broke. Start replaying the exact execution. That's the difference between debugging agents and just watching them burn tokens.

---

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord