Cost Ceilings in Node: Killing an Agent Loop Before It Kills the Budget

#typescript #agents #ai #node

Book: AI That Ships
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

The published agent-cost incidents share a shape. An agent enters a
state where it keeps calling a tool, the tool keeps returning
something it finds unsatisfying, and the loop keeps going. Nothing
crashes. The invoice arrives later.

Most teams respond by adding a provider spending cap. That is worth
having and it is the wrong instrument for this: it is global, it is
delayed, and when it trips it takes down everything at once.

The control you want is per run, checked before each call, in your
own code.

A budget is state the loop carries

export class Budget {
  private spentUsd = 0;
  private calls = 0;

  constructor(
    readonly limitUsd: number,
    readonly maxCalls: number,
  ) {}

  get spent() { return this.spentUsd; }
  get remaining() { return this.limitUsd - this.spentUsd; }

  record(model: string, usage: Usage) {
    this.spentUsd += costOf(model, usage);
    this.calls += 1;
  }

  assertCanSpend(estimateUsd: number) {
    if (this.calls >= this.maxCalls) {
      throw new BudgetExceeded("calls", this.calls, this.maxCalls);
    }
    if (this.spentUsd + estimateUsd > this.limitUsd) {
      throw new BudgetExceeded("usd", this.spentUsd, this.limitUsd);
    }
  }
}

Both limits, because they fail differently. A call cap catches a tight
loop of cheap calls. A dollar cap catches one enormous call with a
context window someone let grow. Either alone leaves a gap.

The estimate before the call is what makes this a ceiling rather than
a post-mortem. Checking after you have spent tells you what happened;
checking before decides whether it happens.

function estimateCost(model: string, messages: MessageParam[]): number {
  const inTok = estimateTokens(messages);
  const r = RATES[model];
  return (inTok * r.input + 2048 * r.output) / 1_000_000;
}

Rough is fine. It only needs to be right enough to stop the last call
that would blow the limit, and assuming a full max_tokens output
errs in the safe direction.

A typed error, not a string

export class BudgetExceeded extends Error {
  constructor(
    readonly kind: "usd" | "calls",
    readonly spent: number,
    readonly limit: number,
  ) {
    super(`Budget exceeded: ${kind} ${spent} of ${limit}`);
    this.name = "BudgetExceeded";
  }
}

The fields matter more than the message. Callers need to distinguish
"stopped because of policy" from "crashed", and a string comparison
is not a mechanism.

Threading it through the loop

export async function runAgent(task: string, ctx: Ctx, budget: Budget) {
  const messages: MessageParam[] = [{ role: "user", content: task }];
  const done: string[] = [];

  while (true) {
    try {
      budget.assertCanSpend(estimateCost(MODEL, messages));
    } catch (err) {
      if (err instanceof BudgetExceeded) {
        return partial(messages, done, budget, err);
      }
      throw err;
    }

    const res = await client.messages.create({
      model: MODEL, max_tokens: 2048, tools: toolDefs, messages,
    });
    budget.record(MODEL, res.usage);
    messages.push({ role: "assistant", content: res.content });

    if (res.stop_reason !== "tool_use") {
      return { status: "complete" as const, text: textOf(res.content),
               costUsd: budget.spent };
    }

    const results = await runTools(res.content, ctx);
    done.push(...results.map((r) => r.name));
    messages.push({ role: "user", content: results.map((r) => r.block) });
  }
}

The check is at the top of the loop, before the spend. That ordering
is the whole feature.

Degrade, do not throw

The part most implementations miss: hitting the ceiling is not an
error condition for the user. Work has been done. Return it.

async function partial(
  messages: MessageParam[],
  done: string[],
  budget: Budget,
  cause: BudgetExceeded,
) {
  const summary = await summariseProgress(messages, budget);
  return {
    status: "partial" as const,
    text: summary,
    completed: done,
    costUsd: budget.spent,
    reason: cause.kind,
  };
}

An agent that researched four of six items and stopped should say so
and hand over the four. A 500 throws away work the user already paid
for and tells them nothing.

The summarisation call itself needs headroom, so reserve a slice of
the budget for it rather than discovering you cannot afford to
explain why you stopped:

const budget = new Budget(limitUsd * 0.9, maxCalls - 1);

Warn the model before you cut it off

A budget the model cannot see produces a run that stops mid-thought.
Telling it changes the behaviour:

if (budget.remaining < budget.limitUsd * 0.25 && !warned) {
  messages.push({
    role: "user",
    content:
      "Budget is nearly exhausted. Stop investigating and give your " +
      "best answer from what you already have.",
  });
  warned = true;
}

This is the highest-value five lines here. Models respond to it — the
run converges instead of being truncated, and the user gets a
conclusion rather than a fragment.

Per-tenant ceilings

A per-run budget stops one runaway. It does not stop one customer
issuing four hundred runs.

export async function reserve(
  tenantId: string,
  amountUsd: number,
  redis: Redis,
): Promise<boolean> {
  const key = `budget:${tenantId}:${new Date().toISOString().slice(0, 10)}`;
  const spent = await redis.incrbyfloat(key, amountUsd);
  if (spent === amountUsd) await redis.expire(key, 60 * 60 * 48);

  const limit = await limitFor(tenantId);
  if (spent > limit) {
    await redis.incrbyfloat(key, -amountUsd);
    return false;
  }
  return true;
}

Reserve before, release on failure. INCRBYFLOAT is atomic, so
concurrent requests cannot both squeeze past the limit — which a
read-then-write would allow.

Daily rather than monthly. A monthly cap is discovered on the 3rd,
after the damage, and it fails everyone simultaneously. A daily cap
per tenant contains a bad actor or a bad integration to one day and
one tenant.

Why the granularity matters

A single global monthly limit has three properties you do not want.

It is discovered late — by definition, at the point where
everything stops.

It is indiscriminate — the customer whose integration is looping
and the customer with a normal workload both stop.

It is unattributable — nothing about the limit tells you who
consumed it.

Per run, per tenant, per day gives you a ceiling that fails small and
tells you where.

Test the ceiling

it("returns partial work instead of throwing", async () => {
  const budget = new Budget(0.001, 100);   // exhausted immediately
  const out = await runAgent("do something", ctx, budget);

  expect(out.status).toBe("partial");
  expect(out.costUsd).toBeLessThanOrEqual(0.001);
});

it("never exceeds the call cap", async () => {
  const budget = new Budget(100, 3);
  const spy = vi.spyOn(client.messages, "create");
  await runAgent("loop forever", ctx, budget);

  expect(spy).toHaveBeenCalledTimes(3);
});

The second one is the test that matters. Point it at a mock that
always returns a tool call, and assert the loop stops. That is the
runaway scenario, reproduced in a unit test, in half a second.

The one line

If you take one thing: check the budget before the call, not
after. Every other detail here is refinement on that ordering.

If this was useful

AI That Ships covers cost
control properly — per-run budgets, tenant ceilings, graceful
degradation, and the accounting that makes any of it possible.