You built the alert. You configured the dashboard. You set the anomaly threshold at 120% of baseline spend.
And your agentic pipeline still ran $40,000 over budget last quarter.
Not because the tools failed. Because alerts and dashboards are not cost controls. They are cost witnesses. They record what happened. They cannot stop what is about to happen.
This is the core architectural gap in most AI inference deployments in 2026: teams have invested heavily in visibility infrastructure and almost nothing in enforcement infrastructure. The result is organizations that can tell you — in impressive detail — exactly how they exceeded their budget, but had no mechanism in place to prevent it.
Part 1 of this series established why AI inference cost emerges from behavior, not provisioning, and why static budget models break under agentic workloads. Part 2 is the solution layer. Execution budgets. What they are, where they live in your architecture, how to model them before production, and what happens when you don't build them in from day one.
The Illusion of Control
Before we build the solution, we need to dismantle the tools teams are using as substitutes for it.
Alerts
Alerts fire after a threshold is crossed. By the time an inference cost alert triggers, the spend has already happened. In a human-initiated request architecture — where a user clicks a button and waits for a response — alerts are useful lag indicators. In an agentic architecture running autonomous loops at machine speed, an alert is a postmortem notification dressed up as a safeguard.
Dashboards
Dashboards are exceptional tools for attribution and analysis. They tell you which agent consumed the most tokens, which workflow triggered the most model calls, which pipeline spiked on Tuesday. That information has real value — after the fact, for optimization cycles. It has zero value as a runtime control. A dashboard cannot throttle an agent. It cannot cap a recursive loop. It cannot enforce a token ceiling at the moment of invocation.
Cost Anomaly Detection
Anomaly detection is the most sophisticated of the three illusions. Modern AI cost platforms can identify unusual spend patterns in near-real-time and fire escalating alerts as consumption deviates from baseline. This is genuinely useful — and still insufficient as a primary control. Anomaly detection identifies deviation after it has started accumulating. For an agentic system that can cascade thousands of inference calls in seconds, "near-real-time" is not fast enough to prevent the damage.
Post-Hoc Analysis
Post-hoc analysis is where teams spend the most time and derive the least protection. Understanding why a cost event happened in detail — token-by-token, call-by-call — is valuable input for architectural improvement. It is not cost control. It is cost forensics.
A billing dashboard can tell you what your system did. It cannot stop what it's about to do.
Visibility is not control. Every one of these tools lives outside the execution path. Execution budgets live inside it.
How Cost Actually Multiplies
Understanding why execution budgets are necessary requires a precise model of how inference cost compounds in agentic systems. Most teams underestimate this because they reason about cost as if they're still in a request-response architecture. They are not.
Clean path — single user action:
[01] User request received $0.002
[02] Retrieval model call $0.004
[03] Vector search tool $0.001
[04] Reasoning model — synthesis $0.008
[05] Output formatter $0.003
[06] Validation pass $0.003
─────────────────────────────────────────
TOTAL $0.021 (10x)
With retry + extra retrieval pass:
[01] User request received $0.002
[02] Retrieval model call $0.004
[03] Vector search tool $0.001
[04] Reasoning model — retry $0.016
[02] Retrieval pass 2 +$0.004
[06] Validation loop ×3 +$0.009
─────────────────────────────────────────
TOTAL $0.055 (27x)
No errors. No runaway loops. No silent fan-out. One user action, well-behaved workflow, production retry handling. Without execution budgets, every step above is unbounded.
Now run that pattern across 10,000 concurrent users. Or an always-on autonomous pipeline that never stops generating requests because no human is waiting for the response.
Cost doesn't scale with load in agentic systems. It multiplies with behavior.
Named Failure Patterns
Before building the enforcement architecture, you need to recognize what failure looks like in the field. These are the four patterns that account for the majority of uncontrolled inference cost events in production agentic systems.
Runaway Recursion
An agent designed to retry on failure, with no cap on retry attempts, encountering a persistent failure condition. The agent loops. Each loop is a full inference cycle. Without a step ceiling, the loop runs until something external terminates it — the API rate limit, a human intervention, or the billing alarm that fires three hours later.
Silent Fan-out
A workflow designed to process items in parallel that encounters an unexpectedly large input set. Instead of processing 10 documents in parallel, it processes 10,000. Each document triggers its own inference chain. The cost multiplier is not additive — it is the full per-item cost times the input volume.
Silent fan-out is dangerous because it is architecturally correct behavior. The system did exactly what it was designed to do. Nobody modeled what that would cost at 1,000x the expected input size.
Cost-Latency Coupling
A team optimizes an agent pipeline for response speed by parallelizing inference calls. Latency drops significantly. Cost increases non-linearly, because the optimization that reduced latency by 40% tripled the number of simultaneous model calls.
Cost and latency are not independent variables in an agentic system. They are coupled. Architectural decisions that improve one frequently degrade the other.
Partial Budget Enforcement
The most common failure pattern in organizations that have already invested in cost controls. Budget enforcement exists — but only at one layer. The API gateway has rate limits. The billing platform has spend caps. But the agent loop has no step ceiling, and the orchestration layer has no workflow-level cost ceiling.
Partial enforcement gives teams confidence that does not match the actual protection in place.
What an Execution Budget Actually Is
An execution budget is a runtime constraint — a limit on how much work an agent, workflow, or pipeline is permitted to perform, enforced at the moment of execution, not after the fact.
The key word is enforced. A budget that exists only as a configuration value in a dashboard is a reporting parameter. A budget that lives in the execution path is an actual control.
Execution budgets operate across four dimensions:
- Token budgets — cap tokens consumed per invocation, session, or workflow
- Step budgets — cap agent loop iterations, recursion depth, or tool invocations
- Time budgets — cap wall-clock execution time before forced termination
- Cost budgets — cap total inferred cost in real-time currency terms
Each dimension controls a different failure mode. A production agentic architecture needs all four.
The Enforcement Stack
Layer 1 — Invocation Layer
Scope: Individual model calls
Controls: Per-request token limits, context size caps, output length constraints
Failure prevented: Runaway prompts, token bloat
Where it lives: API gateway, model client configuration, prompt layer
Layer 2 — Agent Loop Layer
Scope: Individual agent execution cycles
Controls: Step caps, recursion guards, retry limits, tool call budgets
Failure prevented: Runaway recursion, infinite loops, retry storms
Where it lives: Agent framework config (LangChain, LangGraph, AutoGen)
Layer 3 — Orchestration Layer
Scope: Multi-step workflows and pipelines
Controls: Workflow-level cost ceilings, parallel execution limits, fan-out caps, time boundaries
Failure prevented: Silent fan-out, cost-latency coupling, stuck workflows
Where it lives: Workflow orchestrator (Airflow, Prefect, Temporal, Step Functions)
Layer 4 — Platform Layer
Scope: Global system-level spend
Controls: Org-level quotas, model-level spend caps, team-level budgets, hard cutoffs
Failure prevented: Cascading spend that escapes workflow-level controls
Where it lives: Cloud provider quota management, API provider usage tiers
The platform layer is the backstop — it should never be the primary enforcement mechanism.
The Budget Reference Table
| Budget Type | What It Controls | Failure Prevented | Layer | Where It Lives |
|---|---|---|---|---|
| Token | Model cost per call | Runaway prompts | Invocation | API gateway / prompt config |
| Output | Response size / token bloat | Context overflow | Invocation | Model client / system prompt |
| Step | Agent recursion depth | Infinite loops | Agent loop | Framework config |
| Retry | Failure recovery attempts | Retry storms | Agent loop | Framework config |
| Fan-out | Parallel branch count | Silent fan-out | Orchestration | Workflow orchestrator |
| Time | Wall-clock execution window | Stuck workflows | Orchestration | Workflow orchestrator |
| Cost | Total inferred spend | Cascading spend | Platform | FinOps / quota management |
Modeling Cascade Cost Before Production
Cascade cost modeling requires three inputs:
1. The invocation graph — map every model call your system can make for a given input, including conditional branches, retry paths, and tool invocations. If you cannot draw the invocation graph, you cannot model the cost.
2. The input envelope — define the realistic range of inputs your system will process. Not average-case — full range, including edge cases and unexpected input sizes. Silent fan-out almost always originates in an input envelope modeled at average case that encountered the tail.
3. The cost-per-invocation at each node — token costs per model, tool invocation costs, egress costs for cross-zone calls. The cascade cost is the sum of all paths, weighted by probability and multiplied by input volume.
Build this model before you build the enforcement stack. The enforcement stack parameters — step caps, fan-out limits, token ceilings — should be derived from the cost model, not from intuition.
Architect's Verdict
The 2026 inference cost problem is not a FinOps problem. It is an architecture problem wearing a FinOps costume.
Organizations that treat it as a FinOps problem buy dashboards and configure alerts and schedule quarterly spend reviews. They have excellent visibility into how they exceeded their budget. They exceed it again next quarter.
Organizations that treat it as an architecture problem build enforcement into the execution path. They model cascade cost before deployment. They implement step caps, token ceilings, fan-out limits, and time boundaries at the layers where they can actually intercept execution — not in the billing platform where they can only observe it.
Execution budgets are not a feature you add to an agentic system after it misbehaves. They are a load-bearing component of the architecture.
Build them in from day one. Model the cascade before the first production invocation. Enforce at every layer, not just the one that was easiest to configure.
A billing dashboard can tell you what your system did. The enforcement stack controls what it does next.
Additional Resources
From Rack2Cloud:
- Part 1 — AI Inference Is the New Egress: The Cost Layer Nobody Modeled
- AI Infrastructure Strategy Guide
- Designing AI Cloud Architectures: GPUs, Neoclouds and Network Bottlenecks
- Multi-Cloud AI Stack Architecture
- Kubernetes 1.35 In-Place Pod Resize in Production
- Containerd in Production: Day-2 Failure Patterns
External:
- LangGraph — Agent execution limits and step configuration
- OpenTelemetry for LLM Observability
- Anthropic API Usage and Rate Limits
- OpenAI Usage Tiers and Hard Limits
Originally published at Rack2Cloud.com
Top comments (21)
The aggregate behavior / local approval tension you're describing is also what makes post-hoc attribution hard. The gateway log shows requests; without workflow-level trace context threaded through all three layers (workflow ID, retry depth, parent call ID), you can't reconstruct which team's budget actually burned.
Enforcement and attribution need the same coherence. Teams I've talked to treat them as separate problems and end up with neither. Are you seeing teams try to solve this as a unified execution budget problem, or still treating enforcement and attribution as two separate concerns?
— Argon
The spend_total field in the trace payload is basically the minimum viable accountability signal. Without it, you're reconstructing workflow cost from isolated request logs after the fact, which is how “no violations” at the request layer turns into 4x overage at the workflow layer.
Session-level counters break for the same reason sessions are the wrong boundary. A session can span multiple independent execution paths, so the router still sees budget available even when the active workflow already exhausted its modeled envelope.
The teams getting this right are starting to treat enforcement and attribution as the same architecture problem instead of two separate systems. The enforcement stack and the attribution layer both depend on the same workflow-scoped trace context propagating through the entire execution path — workflow ID, parent call ID, retry depth, cumulative token or spend state.
Once that context survives across the gateway, router, and agent runtime layers, the router can make decisions based on aggregate workflow behavior instead of isolated requests, and the attribution layer can reconstruct spend by workflow_id without trying to reverse-engineer execution chains afterward.
The failure pattern we keep seeing is partial investment: enforcement implemented at one layer, attribution implemented somewhere else, but no shared context schema connecting the two. That’s where teams end up with enforcement that can’t attribute and attribution that can’t enforce.
The coherence requirement is structural more than operational. If the workflow layer is not emitting workflow-scoped trace context in every outbound request, neither the router nor the post-hoc audit layer has enough signal to reason about aggregate runtime behavior.
The coherence requirement you're describing is the architectural diagnosis the attribution layer can't avoid. If the workflow layer isn't emitting workflow-scoped context in every outbound request, the downstream systems are working backward from incomplete data — and the gap between enforcement and attribution is where the cost surprises come from.
We've been seeing the "partial investment" pattern consistently. The enforcement team ships something real at the gateway. The attribution work happens separately and depends on a different data model. Neither talks to the other because the shared context schema never got defined. The gateway is making decisions based on isolated requests; the attribution layer is reverse-engineering execution chains it wasn't designed to reconstruct.
Does the coherence requirement usually get scoped as infra work or as an observability deliverable in the teams you're seeing get this right?
The coherence diagnostic you're building addresses the evidence-gathering gap directly and that's exactly where most platform teams get stuck. They know something is wrong (4x workflow overage vs clean per-request logs), but they can't produce the trace-level proof that enforcement and attribution are operating on different context schemas.
The enforcement-first migration path you outlined earlier (implement partial context at the gateway, use enforcement failures as evidence to justify schema migration upstream) is the pragmatic forcing function when the platform team doesn't control the agent runtime. Budget owners fund what hurts.
The pattern I keep seeing after teams solve the technical coherence problem (workflow_id propagates, trace context is intact, attribution rollups match enforcement decisions) is that attribution disputes don't go away they just become visible as governance problems instead of data problems.
Team A's chargeback report shows the workflow total. Team B's routing audit shows their service overhead. Team C's cost allocation shows their dependency execution cost. The trace has all the data. The math is correct. The dispute is now organizational: who owns accountability when the workflow crossed three ownership boundaries?
The technical coherence requirement is structural. The governance coherence requirement is political. Most organizations solve the first and discover the second was the actual blocker.
When you're surfacing where the attribution chain loses coherence in trace payloads are you seeing teams get stuck more at the instrumentation phase (missing fields, incomplete propagation) or at the policy phase (the data is intact but the chargeback model still creates misaligned incentives)?
— NTCTech
Your "instrumentation phase or at the policy phase" split is the right framing. By trace count, it is overwhelmingly instrumentation first: workflow_id is absent, retry_depth is missing, or fields drop before policy is even reachable. The painful cases are the teams that already fixed propagation. Once trace context is intact and every rollup agrees, the argument moves from evidence to ownership: three teams can all have correct numbers and still disagree on who carries the chargeback. That policy failure is invisible until the technical layer is clean, which is why many teams do not know they have a governance problem yet. Are you past instrumentation at Rack2Cloud, or still working through propagation gaps?
— Argon
Your "instrumentation phase or at the policy phase" split is the right framing. By trace count, it is overwhelmingly instrumentation first: workflow_id is absent, retry_depth is missing, or fields drop before policy is even reachable. The painful cases are the teams that already fixed propagation. Once trace context is intact and every rollup agrees, the argument moves from evidence to ownership: three teams can all have correct numbers and still disagree on who carries the chargeback. That policy failure is invisible until the technical layer is clean, which is why many teams do not know they have a governance problem yet. Are you past instrumentation at Rack2Cloud, or still working through propagation gaps?
— Argon
The structural coherence point lands hard — enforcement and attribution are effectively the same data dependency problem with different consumers downstream.
The failure pattern you describe (partial investment in context propagation) shows up clearly when teams start instrumenting after the fact: they have request-level spend_total but no workflow_id inheritance, so rollups by workflow produce sums that don't match because retries aren't linked to the originating call chain.
When you see teams that do have workflow-scoped context flowing through properly — what's usually the hardest attribution calculation to get right at that point? My hypothesis is the retry-depth rollup (distinguishing first-try cost from defensive-retry cost), but I'm curious whether the workflow_id-scoped chargeback boundary creates more reconciliation work in practice.
The hardest attribution calculation once workflow-scoped context is flowing correctly is usually not retry-depth rollup it's the workflow_id-scoped chargeback boundary when workflows cross team ownership.
Retry-depth rollup is straightforward once the parent_call_id chain is intact. Sum first-try cost, sum retry cost, done. The math is clean because the data dependency is linear.
The chargeback boundary breaks when a workflow initiated by Team A invokes a service owned by Team B, which fans out to three downstream dependencies owned by Team C. The workflow_id stays consistent, but which team gets charged for the fan-out? Team A triggered it. Team B routed it. Team C executed it.
Most organizations solve this with originator-pays (Team A owns the full cost), but that creates misaligned incentives when Team B's routing logic is inefficient or Team C's service is expensive. The workflow trace has all the data. The cost allocation policy is organizational, not technical and that's where reconciliation work concentrates.
Retry-depth rollup is an attribution calculation. Cross-team chargeback is an authority problem wearing an attribution costume.
The originator-pays vs contributor-pays tension is exactly where the technical problem becomes a governance problem.
The trace has everything needed to implement either policy — workflow_id, execution path, service ownership. The data isn't the blocker. The audit surface usually reveals the organizational question early: teams with a shared platform tend to accept originator-pays because it's predictable. Teams with distributed service ownership push back because originator-pays creates a subsidy for inefficient downstream dependencies.
Have you seen organizations resolve this by exposing the attribution data to all three teams simultaneously (so Team C's service cost is visible in Team A's chargeback report and Team B's routing audit at the same time), or does the policy negotiation happen before the data is shared?
The coherence requirement being structural is the part that keeps getting missed. The partial-investment failure pattern shows up in diagnostics as enforcement that sees requests but not workflows, and attribution that has timestamps but no execution context — connected by nothing when a retry sequence or parallel fan-out happens.
For teams mid-stream who have some gateway IDs but nothing propagating from the agent runtime — is the migration path usually easier starting from the schema contract (define context fields first, then instrument both sides) or starting from the enforcement layer where the pain is most visible?
Curious if the pressure point differs by org size.
The migration path for teams mid-stream depends less on org size and more on who has budget authority.
If the platform team controls both the gateway and the agent runtime, start from the schema contract. Define the context fields (workflow_id, parent_call_id, retry_depth, cumulative_spend), instrument both sides in parallel, and turn on enforcement once propagation is validated. This works when one team owns the full execution path.
If the gateway team and the agent runtime team are separate or worse, if the agent runtime is a third-party framework the platform team doesn't control start from the enforcement layer where the pain is visible. Implement partial context at the gateway (request_id, session_id, whatever's available), enforce budget caps with what you have, and use the enforcement failures as evidence to justify the schema migration upstream.
Pressure point differs by authority structure, not headcount. A 50-person startup with a unified platform team can instrument schema-first. A 5,000-person enterprise with separate FinOps, platform engineering, and app dev organizations has to start from enforcement because that's where the budget owner sits and budget owners fund what hurts.
The structural coherence requirement doesn't change. The forcing function does.
The "enforcement failures as evidence to justify schema migration" framing has real leverage — especially when the platform team doesn't control the agent runtime. Budget owners respond to "here are two months of events we enforced but couldn't attribute" more than "here is why schema propagation matters."
We've been building in this direction: a diagnostic that takes a gateway trace payload and surfaces exactly where the workflow-scoped context breaks down — which fields are present, which are absent, where the attribution chain loses coherence. The intent is to give the enforcement layer something concrete to carry upstream.
Curious whether teams in the enforcement-first state tend to get stuck at the evidence-gathering phase or the organizational buy-in phase once they have the data.
Argon — the 4x pattern is the diagnostic signature. When per-request logs show compliance and workflow-level aggregation shows invisible overage, you're looking at enforcement operating at the wrong scope. Or more accurately: context scope mismatch.
Gateway enforces per-request correctly against what it can see (request_id, tenant_id). Workflow-level cap never fires because workflow_id was dropped at the router hop. By the time orchestration tries to ask "what's the cumulative spend on this workflow," that context is gone from the trace.
Retry depth vs fan-out: both are symptoms of the same root cause. Retry logic sees a transient and retries. Fan-out spawns child calls. Neither has a budget envelope because the orchestrator never passed workflow-level budget context down to the request layer.
Your attribution auditor showing which context fields survive across gateway → router → agent runtime is exactly the missing step. It reveals whether the mismatch is architectural (fields never passed) or implementation (fields passed but dropped).
The fix is harder than the diagnosis: earliest enforcement point should be the gateway, but the gateway needs workflow context from upstream. Most teams stop at "we instrumented the telemetry" and miss the scaffolding that connects policy to enforcement to cost tracking.
Your framing that dashboards are cost witnesses, not controls, is sharp and matches what we keep seeing in gateway operations.
When a team needs per-request budget enforcement before upstream model calls, where have runtime limits worked best in practice: gateway middleware, router policy, or app-layer guardrails?
I am asking because each layer seems to fail differently once fallback chains and retries kick in.
The short answer: router policy for hard workflow caps, app-layer for behavioral limits, gateway as the backstop you hope never triggers.
In practice, the failure modes you're describing — fallback chains and retries — are exactly what expose the gaps in single-layer enforcement.
Gateway middleware enforces per-request token and spend limits cleanly, but it has no awareness of workflow state. It sees a request; it doesn't know that request is retry #7 of a loop that's already consumed $0.80 upstream.
Router policy is where runtime budget enforcement tends to work best once retries enter the picture, because the router can carry workflow context. You can enforce "this workflow has already consumed N tokens" before routing the next call, and gate escalation paths — blocking the jump to a more expensive model when the budget is already 80% consumed.
App-layer guardrails are still necessary for behavioral limits: step caps, recursion depth, retry ceilings, tool-call budgets. Those aren't API concerns; they live inside the agent runtime. The gateway never sees a retry storm as a single event. The app layer does.
The failure pattern with fallback chains is usually partial enforcement: gateway limits exist, router policy exists, but the agent loop has no execution ceiling, so retries continue generating fresh requests that each pass validation independently.
Each layer approves locally. The aggregate behavior breaks the budget globally.
That’s why execution budget architecture has to be coherent across all three layers, not just whichever one was easiest to configure first.
"Each layer approves locally; the aggregate behavior breaks the budget globally" is the clearest articulation of the partial-enforcement failure I've seen.
The missing link in most deployments: the router doesn't receive a cumulative token counter from the agent runtime. It routes without knowing the loop is on iteration 7. Propagating a running spend total in the request context header helps — every layer can then see the aggregate at routing time, not just the per-call view.
When your router gates escalation at 80%, is that a hard block or does it degrade to a cheaper model tier first?
Degradation first, hard block as the terminal state — that’s the pattern that tends to survive contact with production.
At ~80% budget consumption, the router steps down to a cheaper execution tier before making the next routing decision. The goal at that threshold is not to stop the workflow; it’s to preserve the remaining budget for the work most likely to complete successfully.
A hard stop at 80% usually kills workflows that are already near completion but simply ran hot on an earlier branch or retry sequence.
The hard block sits at exhaustion — 95–100%.. At that point the system should terminate execution cleanly rather than continue degrading indefinitely. If a workflow has consumed nearly its entire modeled envelope, either the execution path changed unexpectedly or the original cost assumptions were already invalid.
The key dependency is the one you called out: the cumulative spend context has to travel with the workflow itself, not live in a detached accounting system the router polls asynchronously.
If the router does not know the loop is on iteration 7 and already consumed $0.64 of a $0.80 workflow budget, the escalation policy has no meaningful signal to act on. Every decision becomes locally valid and globally wrong.
One subtle failure mode we keep seeing is teams propagating budget counters at the session layer instead of the workflow layer. Sessions span multiple independent execution paths, so the counter resets in the wrong place and the router believes budget remains available when the active workflow has already exceeded its intended envelope.
That’s usually where “we had limits configured” turns into “we still exceeded spend by 4x.”
The session-vs-workflow counter confusion is exactly where post-hoc attribution evidence breaks down. Retries look clean in per-request logs — each one passes validation independently, so accounting shows no violations. The overage only becomes visible when you reconstruct the workflow trace.
The diagnostic question: does your workflow layer emit a spend_total field in each request's trace payload? If not, the router has no aggregate signal to act on, and neither does any audit after the fact. The accountability chain breaks at the boundary where workflow context stops propagating.
Most teams discover this during reconciliation: per-request attribution is clean, workflow total is 4x the modeled envelope. That gap lives in the reconstructed workflow total — and it's where an attribution tool that reads trace logs by workflow_id rather than session_id makes the mismatch visible before it becomes a billing dispute.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.