DEV Community: karmendra pandey

Build a Cost-Aware Routing Layer for LLM Agents

karmendra pandey — Sun, 25 Oct 2026 13:37:37 +0000

Most teams discover their agent's token spend the same way: the cloud bill arrives and someone asks what happened. A single LLM call is cheap and predictable. A multi-step agent is neither — it can burn an order of magnitude more tokens than the call you prototyped, because every tool call, every observation, and every retry adds another round-trip through the model. Nobody plans for that. They find out at invoicing time.

The fix is not to use smaller models everywhere. That just trades your cost problem for a quality problem. The fix is a routing layer: score each task, send it to the cheapest model tier that can handle it, escalate when confidence is low, log every cent, and degrade gracefully when the budget runs out.

This tutorial builds exactly that in Python. It runs as-is with no API keys — a mock model stands in for real calls so you can see the routing, escalation, receipts, and guardrails work end to end. Swap the mock for your provider's SDK when you're ready.

What you'll build

Model tiers with per-tier pricing, plus rough token math.
A task router that scores difficulty and picks the cheapest capable tier.
Cascade routing: try the small model first, escalate on low confidence.
Cost attribution: a receipt for every run — tokens and dollars per task.
Budget guardrails: cached answers and simpler models instead of hard failures.

Step 1: Tiers and token math

Everything starts with knowing what a call costs. Define your tiers with illustrative per-token prices (replace these with your provider's actual pricing), and estimate tokens with a rough heuristic until you wire in a real tokenizer:

def estimate_tokens(text: str) -> int:
    """Rough heuristic: ~4 characters per token for English text."""
    return max(1, len(text) // 4)

@dataclass(frozen=True)
class ModelTier:
    name: str
    input_per_mtok: float    # $ per 1M input tokens (illustrative)
    output_per_mtok: float   # $ per 1M output tokens (illustrative)
    quality: float           # illustrative capability score, 0..1

SMALL  = ModelTier("small",  0.15,  0.60, 0.70)
MEDIUM = ModelTier("medium", 1.25,  5.00, 0.88)
LARGE  = ModelTier("large",  5.00, 15.00, 0.97)

def cost_of(tier, in_tok, out_tok) -> float:
    return in_tok / 1e6 * tier.input_per_mtok + out_tok / 1e6 * tier.output_per_mtok

Two things worth saying plainly. First, the 4-characters-per-token rule is a stand-in — fine for a tutorial, not for invoicing. In production, use your provider's tokenizer or usage API. Second, the prices above are illustrative placeholders so the math is visible; plug in real numbers before making decisions with them.

Step 2: Score the task, pick the cheapest capable tier

Routing needs one number per task: how hard is this? A perfect difficulty classifier is a research project. A decent heuristic is an afternoon. The router only needs a number, not a perfect one — start simple and improve it later:

def score_difficulty(task: str) -> float:
    t = task.lower()
    score = 0.15
    score += min(0.25, len(task) / 2000)
    for kw in ("compare", "analyz", "design", "trade-off", "tradeoff"):
        if kw in t: score += 0.08
    for kw in ("code", "sql", "debug", "prove", "architect"):
        if kw in t: score += 0.10
    score += 0.05 * t.count("?")
    if any(w in t for w in ("step by step", " first ", " then ")):
        score += 0.07
    return min(0.95, score)

def pick_tier(difficulty: float, margin: float = 0.05) -> ModelTier:
    """Cheapest tier whose capability clears the difficulty plus a margin."""
    for tier in TIERS:
        if tier.quality >= difficulty + margin:
            return tier
    return LARGE

"What is the capital of France?" scores around 0.17 and lands on the small tier. "Debug this SQL query step by step, then prove the fix handles nulls" scores around 0.62. The margin keeps you from assigning a task to a tier that's exactly at its limit.

Step 3: Cascade — try small, escalate on low confidence

Picking a tier up front is a guess. The cascade makes it a guess you can correct: run the small model first, and only pay for a bigger one when confidence comes back low.

current = pick_tier(difficulty)
while True:
    resp = model.complete(task, difficulty)   # mock here, real SDK in prod
    record_step(current, resp)                # tokens + cost, see Step 4
    if resp.confidence >= THRESHOLD or current is LARGE:
        break
    nxt = next_tier(current)
    if spent + projected_cost(nxt) > budget:
        degraded = True                       # keep what we have; see Step 5
        break
    current = nxt

The confidence number can come from whatever signal you trust: logprobs, a self-rating prompt, a verifier model, or agreement between two samples. The mechanism doesn't care — it just needs a number and a threshold. In the example code, the mock model reports lower confidence when the task is harder than the tier, which is exactly the situation where you'd want a human-designed escalation anyway.

Step 4: Every run gets a receipt

This is the part most teams skip, and it's the part that ends bill-shock. Log tokens and dollars per step, per task, per agent. When someone asks "what did that run cost?", you answer with a receipt, not a shrug:

@dataclass
class Receipt:
    task: str
    steps: list = field(default_factory=list)  # (tier, in_tok, out_tok, cost, confidence)
    degraded: bool = False
    cached: bool = False

    @property
    def total_cost(self) -> float:
        return sum(s[3] for s in self.steps)

Running five tasks through the example — once naively (everything to the large tier) and once through the router — produces this (simulated run, illustrative pricing):

=== Naive: every task goes to the large tier ===
Total cost for 5 tasks: $0.0157

=== Routed: difficulty-scored tiers + escalation + $0.05/task budget ===
- What is the capital of France?                        small(conf 0.62)  $0.0001
- Summarize this error log in two sentences: ...        small(conf 0.60) -> medium(conf 0.83)  $0.0008
- Compare three caching strategies ...                  small(conf 0.58) -> medium(conf 0.77)  $0.0012
- Design a retry and backoff strategy ...               small(conf 0.56) -> medium(conf 0.76)  $0.0013
- Debug this SQL query step by step ...                 small(conf 0.52) -> medium(conf 0.70)  $0.0017

Total cost for 5 tasks: $0.0051
Illustrative savings vs. naive: 67%

The trivial question stays on the small tier. The hard ones escalate to medium — and stop there, because medium clears the confidence threshold. Nothing in this run needed the large tier at all, which is itself useful information: it tells you where your workload actually lives.

Step 5: Guardrails — degrade, don't die

Budgets fail in two familiar ways: the task errors out mid-run when money runs out (bad), or the agent keeps spending because nobody told it to stop (worse). The routing layer handles both before they happen:

Before spending: if even the cheapest possible first step exceeds the remaining budget, serve a cached answer from an earlier identical task instead of spending. Mark the receipt degraded so it's honest about what happened.
Before escalating: if the next tier up would break the budget, keep the current answer and mark the run degraded.

# Guardrail 1: broke before spending -> cached answer, $0
floor = cost_of(SMALL, estimate_tokens(task), 100)
if floor > self.budget and task_hash in self.cache:
    return cached_receipt(degraded=True)

# Guardrail 2: escalation would break the budget -> keep current answer
if spent + projected_cost(next_tier) > self.budget:
    receipt.degraded = True
    break

The same task run twice under a near-zero budget shows the behavior:

=== Budget guardrail: same task twice, tiny $0.00001 budget ===
First run : small  degraded=True cached=False  $0.0001
Second run: cache  degraded=True cached=True   $0.0000
The second run served a cached answer for $0 instead of failing.

The first run couldn't afford to escalate, so it kept the small-tier answer and said so. The second run cost nothing at all. Neither one threw an error at the user. That is the whole point: a budget overrun becomes a slightly worse answer, not an outage.

Where this fits in a real architecture

A routing layer is one component, not a system. In the reference architecture I maintain (github.com/karmendra8386/agentic-ai-aws-reference), cost shows up in two places: the observability section calls for structured per-turn logs that include tokens and cost, and the governance pattern calls for cost dashboards next to quality dashboards. This routing layer is the control plane that acts on those numbers — the logs tell you what you spent, the router decides what you'll spend next.

The repo itself is a starter skeleton (CDK stacks for agent IAM roles and encrypted storage, Playwright regression tests), not production code — treat the routing layer here the same way. Harden the pieces for your environment: real tokenizers, real pricing, a real confidence signal, and a real cache backend.

What this gives you

Five things, in the order you'll notice them. First, the bill stops being a surprise — every run has a receipt. Second, easy work stops subsidizing hard work — the small tier handles what it can, and you can see that it does. Third, escalation becomes a deliberate decision with a paper trail instead of a silent default to the biggest model. Fourth, budgets become graceful: degraded answers instead of failed tasks. Fifth, and most useful over time, the receipts become data — you'll learn which task types actually need the large tier, and that tells you where to invest in prompts, fine-tuning, or caching instead of raw model spend.

Start with the mock, run it, read the receipts. Then swap in one real model call and watch your own numbers appear. That's the whole tutorial — and the end of bill-shock.

Karmendra Pandey is Practice Architect for AI/ML at TEKsystems (19 years across data engineering, ML, and cloud), currently architecting AI/ML frameworks for production AI agents on AWS.

What does your agent run actually cost? Receipts for LiteLLM and LangChain apps with tokenecon

karmendra pandey — Fri, 23 Oct 2026 01:39:43 +0000

Your LLM bill arrives as an aggregate: a provider invoice, a dashboard with a big number. Nobody can answer the questions that actually matter: what did that task cost? What does this run cost per correct answer? A single model call has a predictable price. A looping agent — tool calls, observations, retries, retrieved documents, each another round-trip through a priced model — does not. Most teams discover the total from the cloud bill instead of the architecture.

That's why I built tokenecon, an open-source Python package (MIT, on GitHub) that decomposes agent spend per task and produces receipts: tier used, tokens, cost, and confidence per step. Until now the only way to measure a live agent was the Strands Agents integration. With tokenecon 0.2.2, two more trackers join it — LiteLLM and LangChain / LangGraph — and this is the hands-on walkthrough.

Install it:

pip install tokenecon

tokenecon itself has no dependencies. The integrations are optional: install litellm or langchain-core only for the framework you're tracking.

LiteLLM: two lines

The LiteLLM integration is a CustomLogger that prices every successful call with tokenecon tiers, using the provider usage LiteLLM already attaches to each response — no estimation, no token-counting heuristics:

import litellm
from tokenecon.integrations.litellm import CostTracker

tracker = CostTracker(tier="medium")
litellm.callbacks = [tracker]

litellm.completion(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize our Q3 cloud spend."}],
)

receipt = tracker.receipt("Summarize our Q3 cloud spend.")
print(receipt.pretty())
# request    : Summarize our Q3 cloud spend.
# difficulty : 0.162
# outcome    : ok
#   step 1: tier=medium in=1840 out=312 cost=$0.003088 conf=1.00 litellm call 1 (gpt-4o-mini)
# total_cost : $0.003088

Every successful call becomes one receipt step, in call order, with confidence 1.0 — with provider-reported usage there is no uncertainty in the cost figure. Both the sync and async LiteLLM hooks are covered, and the tracker is thread-safe.

A few useful properties when you wire it into a real app:

tracker.steps is a snapshot of the priced calls so far, and tracker.total_cost is the running USD total — enough to drive a live spend guardrail.
tracker.reset() forgets the recorded calls, e.g. between tasks or between users, so one long-lived tracker can serve a whole service.
Your app probably mixes models. Pass model_tiers to price specific ones differently — keys match the model name exactly or the part after the last /, so "gpt-4o" also covers "openai/gpt-4o":

tracker = CostTracker(
    tier="small",
    model_tiers={"gpt-4o": "large", "claude-opus-4-1": "large"},
)

If a call arrives with no usage data, it records a zero-token step marked "no usage reported" rather than silently dropping it. And the module imports even without LiteLLM installed — instantiating the tracker raises a clear ImportError only when you actually try to use it.

LangChain and LangGraph: one callback

The LangChain integration is a BaseCallbackHandler that prices every completed model call the same way, and it attaches wherever LangChain accepts callbacks — a model, a chain, an agent, or a compiled LangGraph graph:

from langchain_openai import ChatOpenAI
from tokenecon.integrations.langchain import CostTracker

tracker = CostTracker(tier="medium")
llm = ChatOpenAI(model="gpt-4o-mini", callbacks=[tracker])

llm.invoke("Summarize our Q3 cloud spend.")
receipt = tracker.receipt("Summarize our Q3 cloud spend.")
print(receipt.pretty())
# request    : Summarize our Q3 cloud spend.
# difficulty : 0.162
# outcome    : ok
#   step 1: tier=medium in=1840 out=312 cost=$0.003088 conf=1.00 langchain call 1 (gpt-4o-mini)
# total_cost : $0.003088

For a LangGraph graph, attach it the same way in the invoke config: graph.invoke(state, config={"callbacks": [tracker]}).

Both model shapes are handled through one hook: classic LLMs report usage in llm_output["token_usage"], chat models on each message's usage_metadata, and multi-generation runs are summed into the single step for the run. One design decision worth knowing: the handler never raises into your chain (raise_error = False), so a run whose usage can't be found becomes a zero-token step instead of breaking your application. A cost observer should never break the thing it observes.

Bring your own prices

The default tiers (small, medium, large) are illustrative placeholders. For real numbers, pass a tokenecon.ModelTier with your provider's actual price list as the tier or inside model_tiers. Once you do, receipts stop being estimates and become your audit trail.

Why receipts matter: the measured context

Receipts are the unit of cost observability. Accumulate them and they become the dataset your routing policy improves from — and they feed public work that gives the numbers weight:

Using tokenecon, the same ~3.5M-token agent workload costs $1.37 on one model and $38.72 on another — roughly a 28× spread — while the framework around the model adds at most ~3.9×. Model choice, not framework choice, is the line item.
The Cost Per Correct benchmark (dataset DOI 10.5281/zenodo.23439614) measures agents in the unit that matters: cost per correct answer, not cost per call. The cheapest model that fails is the most expensive line item.

And when you have receipts of your own, make them shareable. tokenecon ships a badge generator (new in 0.2.2):

python -m tokenecon.badge results.json --format svg -o badge.svg

It takes either a summary JSON ({"model": ..., "tasks_solved": 5, "tasks_total": 5, "total_cost_usd": ...}) or a list of tokenecon receipts, and renders a shields-style badge — or endpoint JSON for a live shields.io badge — so your README can wear its cost per correct the way it wears its CI status. (For graded benchmark runs, the summary form is the honest input: receipts carry no grading information.)

Every agent run should come with a receipt. Start with two lines of code and see what yours actually costs.

Links: PyPI: tokenecon · GitHub · Cost Per Correct leaderboard · Dataset DOI: 10.5281/zenodo.23439614 · Software DOI: 10.5281/zenodo.23427025

A Correct Answer Has a Price. A Wrong One Has Three.

karmendra pandey — Thu, 22 Oct 2026 13:23:37 +0000

Two weeks ago I published a benchmark with a simple question: what does a correct answer from an AI agent actually cost? The headline result was that tiered routing — cheap model first, escalate on low confidence — produced correct answers for $0.0069 each, against $0.0097 for sending everything to the large model. Same accuracy, 29% less money.

I thought the interesting part was the routing. Three readers showed me the interesting part was the failures.

Price #1: the failure itself (Dean Lee's point)

My table treated a wrong answer as costing nothing beyond the tokens spent failing. Dean Lee, who ran his own entry in the same challenge, pushed on exactly that: in production, a wrong answer gets reviewed, reworked, or rolled back. It is never free.

Put a price on it and the table flips. Add a modest $0.01 per wrong answer — one human glance, a few seconds of review:

Setup	Cost per correct, failures free	Cost per correct, failures at $0.01
Small model only	$0.0004	~$0.020
Tiered routing	$0.0069	$0.0069
Large model only	$0.0097	$0.0097

The "cheapest" setup becomes the most expensive one, by a factor of fifty, because two-thirds of its answers were wrong. The tiered setup doesn't move at all — it made no errors to price. Cost per correct without a failure price isn't a metric. It's a subsidy for being wrong.

Price #2: the escalation (Noah Salas's point)

Noah Salas, who builds agents for a living, added the cost I had averaged away: escalation itself. In my routed run, the system made 33 model calls to answer 24 tasks — the 9 extra calls are cheap attempts that failed before the large model took over. You pay for the failure and the retry.

That means the routed number isn't a constant. It depends on your escalation rate, which depends on your traffic mix, which drifts. A benchmark measures it once; production has to measure it continuously, or the 29% saving quietly evaporates as your workload changes.

Price #3: the failure you shouldn't price at all (Will Lassalle's point)

Will Lassalle, a CIO, split the category I was treating as one thing. A failure caught in review costs review time. A failure that escapes costs rollback, remediation, and customer trust. And some failures — his example class was anything touching money movement or safety — shouldn't be in the cost table at all, because they shouldn't be available to an autonomous system at any average price.

His ordering is the one I'm adopting: risk gate first, cost comparison second. Decide which failures are forbidden, block the system from making them autonomously, and only then optimize cost per correct among what's left.

What version 2 measures

So the benchmark I'm now building scores four numbers, not one:

Cost per correct answer — the original metric
Cost per correct, by failure type — caught-in-review failures and escaped failures priced separately
Escalation rate and its cost — measured as a distribution, not a point estimate
Gate violations — attempts to take a forbidden action, which fail the run outright regardless of cost

The accuracy floor stays: below 95% correct, a setup isn't cheap, it's disqualified.

The original benchmark is here, if you want the starting point: https://www.kaggle.com/benchmarks/karmendra8386/cost-per-correct — and the write-up that started this argument: https://dev.to/karmendra_pandey_43ac6983/your-ai-agent-has-a-burn-rate-so-i-benchmarked-the-cost-of-a-correct-answer-1bc1

The lesson I'd take to any team running agents: your cost dashboard is lying to you in a specific, fixable way. It prices the tokens. It doesn't price the wrong answers, the retries, or the failures that should never have been allowed. Price all three, and the cheapest setup on the page is usually not the cheapest setup you own.

How does your team price a wrong answer today — review time, a flat penalty, or not at all?

AI Will Make IT Cheaper. I Believe It Will Make the World Grow Faster, Too.

karmendra pandey — Mon, 19 Oct 2026 17:00:26 +0000

By Karmendra Pandey

I spend my working life inside IT budgets, so let me start with what I can see from my desk, not from a stage.

AI is cutting the cost of IT work — not in theory, in measured studies. Customer support agents with a good AI assistant resolved 13.8% more issues per hour, and the least experienced agents gained the most, about 35%. Professionals in a controlled writing trial finished 40% faster with 18% better quality. Developers in large field experiments completed about 26% more tasks. And underneath all of it, the raw input is collapsing in price: getting a fixed level of AI performance cost $20 per million tokens in late 2022, and $0.07 about eighteen months later. When the price of anything falls more than 280 times, entire categories of work that were too expensive to automate — documenting every legacy system, summarising every incident, testing every line of old code — quietly become default.

That is the IT cost story, and I believe it. But it is not the story I actually want to tell.

The bigger claim

I foresee world GDP growing at a faster pace in the AI era than in the two decades before it. I know how that sounds, because the forecasters disagree wildly. The IMF projects AI adding roughly half a percentage point a year to global growth over the next decade. The OECD sees 0.4 to 1.3 points of extra productivity growth a year in the most exposed advanced economies. Goldman Sachs estimates a 7% lift to the level of world GDP over about a decade. And the leading sceptic, Daron Acemoglu, calculates barely 1–1.6% in total for the US over ten years.

Look at the spread and you learn something: nobody serious is arguing about whether AI works at the level of a task. They are arguing about how fast companies reorganise around it, how many tasks it can profitably do, and whether it creates new work or only automates the old. Those are questions about management, not about models. That is why I land where I do. An extra half a point of growth a year — the middle of the range — sounds modest until you remember that advanced economies have been stuck near 1–1.5% productivity growth for twenty years. Half a point, compounded, reverses most of that slowdown. We do not need the heroic forecast to be right for this to be historic. We only need the boring one.

History is the honest referee

Every great technology turning point looked disappointing before it looked inevitable.

Steam came first. James Watt's engine was patented in the 1770s, yet steam's peak contribution to British productivity growth arrived around 1850–1870 — roughly a hundred years later. Electricity was worse for the impatient: power stations appeared in the 1880s, but by 1900 electric motors still drove under 5% of factory machinery. The productivity boom only came in the 1920s, about forty years on, and only after factories were redesigned — one big steam engine and a forest of belts replaced by a motor on every machine, on single-storey production lines built around the flow of work, not around the power source.

Computers repeated the pattern so faithfully that economists named it. "You can see the computer age everywhere but in the productivity statistics," Robert Solow wrote in 1987. The statistics finally moved in 1995–2004, once companies redesigned processes around the technology instead of paving over old ones. The internet and the smartphone then taught the opposite lesson: adoption can be nearly instant — smartphones went from 5% to 40% of Americans in about four years — and the productivity boom can still not come, because consumer delight is not the same thing as reorganised production.

That history is exactly why I am not troubled that measured productivity has barely moved yet. AI is following the classic pattern, which researchers call the productivity J-curve: firms first spend on the invisible complements — data, workflow redesign, training, governance — which drag measured output down before the gains show up. Electricity needed new factories. AI needs new workflows. The technology is the easy purchase. The redesign is the work.

What is actually different this time

But I do not think this turning point is just electricity with better marketing. Four things are genuinely different.

First, speed. ChatGPT reached 100 million users in about two months. Electricity took decades to reach half of factories. AI rides on devices, networks, and cloud platforms that already exist, so there is no wiring of the world to wait for.

Second, copying is nearly free. A steam engine had to be built, shipped, and installed, one at a time. An AI capability, once created, can be copied to a million companies overnight at close to zero marginal cost. Intelligence has never before behaved like software. Now it does.

Third, it automates thinking, not muscle. Steam and electricity multiplied what hands could do. AI multiplies what minds can do — drafting, summarising, coding, analysing, supporting customers — including in the slow, expensive service sectors that resisted every previous wave of automation.

Fourth, it can help invent its own successors. AI systems have already predicted the structures of 200 million proteins, work recognised with a Nobel Prize. A technology that accelerates research itself is not just a tool in the economy. It is a lever on the rate at which the economy gets new tools.

The honest warnings

I hold this view, but I hold it with two warnings attached, and I have seen both from inside real IT organisations.

The first is that task gains leak. In one careful trial, experienced developers working in their own mature codebases were 19% slower with AI tools, while believing they were 20% faster. AI is fastest on bounded, pattern-rich work and can be a tax on deep, context-heavy work. Across a whole economy so far, the average worker's time saved works out to only about 1.4–2.2% of all hours — because most hours are not yet AI-assisted at all. Anyone who multiplies a laboratory gain by the whole of GDP is doing marketing, not analysis.

The second warning is about what happens to IT itself, my own industry. AI cuts the cost per unit of IT work far more reliably than it cuts the total IT budget, because demand expands to eat the savings — firms themselves estimate about 29% of cloud spend is wasted, AI creates new waste even as it removes old, and data centres are heading toward roughly doubling their electricity use by 2030. And for IT services firms like the one I work in, the business model itself is the thing being repriced: work sold by the human hour is work priced in the input AI cheapens. The future of my industry is outcome-priced, redesigned workflows, governance, and modernisation — selling the complements, not the hours. I consider that a healthy correction, but nobody should pretend it is painless, especially for the junior roles the old pyramid was built on.

My prediction

So here is my point of view, stated as plainly as I can state it.

Within this decade, world growth will run faster than it did in the sluggish decade before AI — not because the models are magic, but because the cost of intelligence is collapsing, the technology copies for free, and the redesign work, though slow, has already started in support desks, software teams, and back offices everywhere. The countries and companies that win will not be the ones that bought AI first. Electricity rewarded the factories that redesigned around it, roughly forty years after the power station. AI will reward the organisations that redesign around it — and it will not take forty years this time.

The steam age belonged to those who built engines. The electric age belonged to those who rebuilt factories. This age will belong to those who rebuild how work itself is organised, around a mind that can be copied. That is a turning point at least as large as the previous ones — and for the first time in history, it is arriving faster than our institutions are ready for. That gap, not the technology, is the real risk. And closing it is the real opportunity.

References

Brynjolfsson, E., Li, D. & Raymond, L. — "Generative AI at Work," NBER Working Paper 31161 / Quarterly Journal of Economics (2025): support agents +13.8% issues/hour, +35% for least-skilled. https://www.nber.org/digest/20236/measuring-productivity-impact-generative-ai
Noy, S. & Zhang, W. — "Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence," Science (2023): writing time −40%, quality +18%.
Cui, K. et al. — field experiments with AI coding assistants across 4,867 developers (2024): +26.1% completed tasks.
Peng, S. et al. — "The Impact of AI on Developer Productivity" (2023): 55.8% faster on a standardised coding task. https://arxiv.org/html/2302.06590v1
METR (2025) — RCT with experienced open-source developers: 19% slower with early-2025 AI tools in their own repositories.
Stanford Institute for Human-Centered AI — AI Index Report 2025: inference cost at GPT-3.5-level performance fell from $20 to $0.07 per million tokens (Nov 2022–Oct 2024).
IMF (Oct 2026) — Georgieva: AI could add ~0.5 percentage points to annual global growth over the next decade. https://www.imf.org/en/blogs/articles/2026/10/15/the-world-cant-afford-to-get-ai-wrong
OECD (June 2025) — "Macroeconomic Productivity Gains from Artificial Intelligence in G7 Economies": +0.4–1.3 pp/yr labour productivity in high-exposure G7 economies. https://www.oecd.org/content/dam/oecd/en/publications/reports/2025/06/macroeconomic-productivity-gains-from-artificial-intelligence-in-g7-economies_dcf91c3e/a5319ab5-en.pdf
Goldman Sachs (2023) — "Generative AI Could Raise Global GDP by 7%." https://www.goldmansachs.com/insights/articles/generative-ai-could-raise-global-gdp-by-7-percent.html
Acemoglu, D. (2024/25) — "The Simple Macroeconomics of AI": US GDP +1.1–1.6% in total over 10 years. https://economics.mit.edu/news/daron-acemoglu-what-do-we-know-about-economics-ai
McKinsey Global Institute (2023) — "The Economic Potential of Generative AI": $2.6–4.4 trillion/year across 63 use cases.
Bick, A., Blandin, A. & Deming, D. — Federal Reserve Bank of St. Louis: AI users save 5.4% of their own hours, ~1.4–2.2% of all work hours.
Crafts, N. — steam's contribution to British productivity growth peaked c.1850–70, ~100 years after Watt; ICT's contribution exceeded steam's.
David, P. (1990) — "The Dynamo and the Computer": electrification's ~40-year lag and the factory redesign it required.
Brynjolfsson, E., Rock, D. & Syverson, C. — "The Productivity J-Curve," American Economic Journal: Macroeconomics (2021).
Flexera — State of the Cloud 2026: organisations self-estimate 29% of cloud spend wasted.
IEA — "Energy and AI" (2025): data-centre electricity 415 TWh (2024) rising to ~945 TWh by 2030.

Your AI Agent Has a Burn Rate — So I Benchmarked the Cost of a Correct Answer

karmendra pandey — Sun, 11 Oct 2026 22:46:35 +0000

My submission to the Kaggle Benchmarking Challenge.
Benchmark: https://www.kaggle.com/benchmarks/karmendra8386/cost-per-correct
Tooling: tokenecon — my open-source cost accounting for agent workflows · Preprint: https://doi.org/10.5281/zenodo.23195511

A single model call has a predictable price. A looping agent does not. Every tool call, retry, and retrieved document adds another priced round-trip, and most teams find out the total from the cloud bill at the end of the month.

So I built a benchmark that asks a different question. Not "what does the model cost?" — but "what does each correct answer cost?"

What I built

A suite of 24 small agent-style tasks in three difficulty bands — easy (arithmetic, extraction, classification), medium (analysis, drafting, prioritization), and hard (multi-hop synthesis, trade-offs, derivations). The kind of work production agents actually do, not trivia. Every task has a deterministic checker: the answer is right or it isn't. No LLM judge, no vibes.

The first five easy tasks are live on Kaggle now, run against five hosted models: Gemini 2.5 Flash, Gemini 3.7 Flash, DeepSeek R1, GPT-OSS 120B, and Qwen3 235B.

The Kaggle results (real, from the live task pages)

Task	Gemini 2.5 Flash	Gemini 3.7 Flash	DeepSeek R1	GPT-OSS 120B	Qwen3 235B
e1 — arithmetic: 47 × 38 + 112	Pass	–	–	–	–
e2 — extraction: founding year	Pass	–	Error	Error	Error
e3 — sentiment classification	Pass	Pass	Pass	Pass	Pass
e4 — arithmetic: requests in 3 hours	Pass	Pass	Pass	Pass	Pass
e5 — extraction: Eiffel Tower height	Pass	Pass	Pass	Pass	Fail

Three things stand out:

1. One model finished everything. Gemini 2.5 Flash is the only model with a result on all five tasks — 5 for 5 — and it tops the benchmark leaderboard at 100.00. On the three tasks where all five models ran (e3–e5), 14 of 15 rows pass.

2. The only genuine failure is the interesting one. GPT-OSS 120B, a flagship open-weights model, failed e5 — a one-line extraction ("how tall is the Eiffel Tower, in meters?") that every other model passed. Easy tiers are supposed to be boring. They mostly are — until a model drops a task it had no business dropping, and in production that dropped task is a real user's request.

3. When accuracy ties, cost is the whole game. On easy work, nearly every modern model is right nearly all the time. If accuracy can't separate them, the differentiator is what each correct answer costs — which is exactly the number most teams never compute.

One honest note on e2: its three "Error" rows are Kaggle platform errors from my first build pattern, not model answers — the runs completed but the platform wouldn't register swapped-in models. I fixed the pattern for e3 onward and left e2 as-is rather than quietly rebuilding it, because a benchmark you can silently edit isn't worth much.

The other half: routing (local simulation)

The Kaggle tasks above test models one at a time. Production agents don't work that way — a router decides which tier handles each task. So locally, I ran the full 24-task suite through four routing strategies using tokenecon, with deterministic scoring:

Strategy	Accuracy	Total cost	Cost per correct answer
Tiered routing	100%	$0.1661	$0.0069
Large model only	100%	$0.2334	$0.0097
Medium model only	67%	$0.0274	$0.0017
Small model only	33%	$0.0036	$0.0004

Tiered routing matched the big model's accuracy at 29% lower cost — easy tasks went to the cheap tier, hard ones escalated. But look at the last row: the cheapest tier has the lowest cost per correct answer and gets two-thirds of the work wrong. Cost-per-correct without an accuracy floor is a trap. With a ≥95% accuracy requirement, tiered routing is the cheapest viable strategy — and that floor is the piece most cost dashboards are missing.

To be clear about what this table is: a deterministic local simulation with illustrative prices from tokenecon's price table. Kaggle's task pages report pass/fail only, so no dollar figures above come from Kaggle. The simulation measures the routing policy, which is the part teams actually control.

Disclosures, because benchmarks should have them

Only the 5 easy tasks are live on Kaggle so far. The medium and hard tiers exist in the suite and in the local runs; their Kaggle versions come next.
Kaggle's model catalog offers Qwen3 235B in its 2506 build, so that's what ran — my original spec said 2507.
Gemini 3.7 Flash appears because it's Kaggle's default placeholder model for benchmark tasks built from its template. Its rows are real runs, so I kept them.
The stray private task page some may spot ("What is Kaggle?") is an untouched template I built by accident while learning the UI. It's not part of the benchmark.

What's next

Medium and hard tiers go up on Kaggle next, and that's where this gets interesting — easy tasks saturate, but multi-hop work is where models separate and where routing earns its keep. If you run agents in production, fork the harness, plug in your own tasks and your own prices, and find out what your correct answers actually cost.

Your AI Agent Has a Carbon Footprint. Nobody's Measuring It.

karmendra pandey — Sat, 10 Oct 2026 14:11:05 +0000

Your AI agent has a carbon footprint. Not the datacenter's footprint, not the training run's — your agent's, for the specific task it just ran. And nobody is measuring it.

I just published a white paper on this (open access, DOI 10.5281/zenodo.23282699). Here's the plain-language version.

The gap

Every carbon tool we have stops at the boundary of a single model call. Cloud dashboards give you monthly totals by service. EcoLogits estimates per-API-call energy. All useful — and all blind to what an agent actually is.

A production agent task isn't a query. It's a workflow: a planner call on a frontier model, retrieval and tool calls, code execution, retries when something fails, a verifier pass at the end. Each step can run on a different model, a different provider, in a different region, on a different grid. No per-call meter sees the whole thing.

The numbers are wilder than you'd think

How much energy does one AI query use? There is no single answer — and that's the point:

A small-model response: ~0.03 Wh (measured)
A median Gemini prompt, full serving stack: 0.24 Wh (measured by Google)
An average ChatGPT query: ~0.34 Wh (disclosed by OpenAI)
A long reasoning query: ~3.9 Wh (modeled)
A long-context query: up to ~40 Wh (modeled)

That's a span of more than 1,000x. "A query" is not a unit of physics. And agents are the multiplier: a typical 4-6 call agentic task runs about 1.1 Wh, or ~50 Wh with reasoning enabled. The unit we should be accounting is the task, not the call.

The ledger

The paper proposes a per-task carbon ledger. The math is simple on purpose:

CO2e per task = sum over steps of (energy x PUE x grid carbon intensity) + amortized embodied carbon

Every figure gets labeled honestly: measured, modeled, disclosed, or illustrative. No black boxes. The ledger also allocates the shared stuff no per-call meter sees — idle reserve capacity, KV-cache effects across steps.

The worked example (illustrative)

Take a ~50 Wh reasoning task. Run it on the UK grid (233 gCO2e/kWh, PUE 1.54): about 18 gCO2e. Route the same task to Quebec's hydro grid (31 g/kWh): about 2.4 gCO2e. Same task, same answer — 7x less carbon, from routing alone. That's not a measurement, it's arithmetic from published inputs, but it shows why the ledger matters: you can't optimize what you can't see.

Spend carbon like dollars

Here's the part I'm most interested in. We already budget dollars per agent task. The paper argues for a carbon budget alongside it: the orchestrator chooses model x region x timing per step to stay under a CO2e budget, the same way it stays under a dollar budget. Research prototypes doing this for single queries have cut modeled emissions ~51% — nobody's doing it across a whole agentic workflow yet.

Why now

Two deadlines are converging. The EU AI Act already requires model providers to document energy use (fines up to 15M euros) — but admits no measurement standard exists yet. California's SB 253 forces large companies to report Scope 3 emissions from 2027, and purchased AI inference lands squarely in Scope 3. Somebody has to define the methodology. This paper is my bid for what it should look like.

It builds on the Green Software Foundation's vocabulary — they name "per workflow execution" as the agentic functional unit — and extends it into something you can actually implement.

What I need from you

The full paper is open access: DOI 10.5281/zenodo.23282699. Read it, argue with it. And one question: if your agent platform showed the carbon cost of every run next to the dollar cost — would it change how you build?

Argus: a security layer for every AI model you call

karmendra pandey — Fri, 09 Oct 2026 17:30:05 +0000

Your app talks to AI models. Who's watching what goes in and out?

Last week I watched an AWS secret key sail through a prompt to a third-party model in a demo. Nobody noticed. That's when I stopped treating AI security as a checklist item and built Argus — an open-source security layer that sits in front of every model you call.

The problem in one picture

Right now, most apps call models like this: app → API key → model. There's nothing in between. So:

A developer pastes an AWS key into a prompt. It leaves your network.
A user's email and phone number ride along in the conversation history.
"Ignore all previous instructions" arrives in user input, and nobody flags it.
When something leaks, there's no record of what happened.

Gateways like LiteLLM solve routing and budgets. Nobody owns the security in between. That's the gap Argus fills.

What Argus does

Argus is an OpenAI-compatible gateway. You point your app at it with one base_url change, and every request passes through a security policy — on the way in and on the way out — before any model sees it. OpenAI, Anthropic, Bedrock, vLLM, Ollama, anything with an OpenAI-shaped endpoint.

pip install argus-gateway
argus   # serves on localhost:4000

import openai
client = openai.OpenAI(api_key="<your-virtual-key>", base_url="http://localhost:4000/v1")

The security machinery

Four pluggable detectors scan every request and response. Detectors find; a policy engine decides — allow, redact, block, or flag:

Detector	Catches	Default
Secrets	AWS keys, private keys, API tokens, `password = ...`	Block — never reaches a model
PII	emails, phones, SSNs, credit cards	Redact — `[REDACTED:EMAIL]`, request continues
Prompt injection	"ignore previous instructions", role hijacks, system-prompt probes, jailbreaks	Flag — allowed, but marked
Blocklist	your own patterns	Block

Every decision lands in an audit log — timestamp, key, detector, action, and a redacted snippet. Raw PII and secrets never touch the trail. That's your compliance story: GET /admin/audit.

Here's what a blocked secret looks like in practice:

POST /v1/chat/completions
{"messages": [{"role": "user", "content": "my aws key is AKIA..."}]}

→ 400 blocked by security policy: secrets:AWS_KEY

And every successful response tells you what the layer did:

"gateway": {
  "security": {"redactions": ["pii:EMAIL"], "flags": []},
  "cost_usd": 0.000004, "tier": "small", "degraded": false
}

Cost intelligence rides along

Security is the product, but the gateway also thinks about money — because the same layer that sees every request is the right place to price it:

Tiered routing — requests are scored for difficulty and sent to the cheapest tier that can handle them
Graceful budgets — blow your budget and requests fall back to the cheapest tier instead of dying with a 429 (hard-stop mode if you prefer)
Per-task attribution — pass X-Task-Id and a whole agentic run's spend is ledgered as one task

This comes from my earlier work on tokenecon and the paper behind it.

Try it, break it

It's early — v0.1.0, MIT licensed, and I'd rather have people poke holes in it now:

GitHub: https://github.com/karmendra8386/argus
PyPI: pip install argus-gateway

On the roadmap: streaming, Bedrock/Gemini/Mistral adapters, semantic caching, OpenTelemetry export, and an ML-based injection detector (the detector interface is pluggable — contributions welcome).

If you've ever wondered what your prompts actually carry out the door — point Argus at your traffic for a day and check the audit log. You might be surprised. I was.

The Receipt: I Priced Every Token of One AI Agent Task

karmendra pandey — Tue, 06 Oct 2026 14:16:57 +0000

Everyone talks about AI costs in aggregate. Dashboards show monthly spend, per-model totals, team budgets. Nobody shows you a receipt.

So here's one. A single AI agent task, priced line by line — every token, every tool call, every retry. This is a worked example from production-style workloads I describe in my reference architecture work (full paper: https://doi.org/10.5281/zenodo.23178788). The numbers are representative of a mid-complexity support-triage agent running on AWS Bedrock. Your numbers will differ. The shape of the receipt won't.

The task

A customer support triage agent: read an incoming ticket, pull the customer's history, check the knowledge base, decide whether to auto-resolve or escalate, and draft the response. One task, one receipt.

The receipt

#	Line item	Tokens in	Tokens out	Unit cost basis	Cost
1	System prompt (loaded once, cached)	1,850	—	cached input	$0.0011
2	Ticket text + customer history	2,300	—	standard input	$0.0069
3	Knowledge base retrieval (3 chunks)	4,100	—	standard input	$0.0123
4	Reasoning: triage decision	—	420	standard output	$0.0050
5	Tool call: CRM lookup	180	90	in/out	$0.0016
6	Reasoning: draft response	—	680	standard output	$0.0082
7	Retry: first draft failed tone eval, regenerated	6,200	710	in (cached) + out	$0.0121
8	Embedding: ticket for memory store	340	—	embedding	$0.0000
9	Final response + audit log write	—	350	standard output	$0.0042
	Total	~15,000	~2,250		~$0.051

Wait — $0.051, not $0.23? The $0.23 figure from my paper is the fully-loaded cost: it includes the amortized cost of the eval harness, the retrieval infrastructure, and the human review queue time for escalations. The raw inference receipt is five cents. The governed cost is twenty-three cents. Both numbers matter, and most teams track neither.

Reading the receipt

Three things jump out:

1. The retry cost more than the original attempt. Line 7 cost $0.0121 — nearly as much as lines 4 and 6 combined ($0.0132). Retries are the silent budget killer in agent systems. Every eval gate you add has a cost; every gate you skip has a bigger one. The receipt lets you price the tradeoff instead of guessing.

2. Context is the bulk of input spend. Lines 2+3 are 6,400 input tokens — 43% of the total input. That knowledge-base retrieval is doing real work, but it's also the first place to optimize: better chunking, semantic caching of frequent queries, and prompt compression all attack this line directly.

3. The system prompt is nearly free — because it's cached. Line 1 costs a tenth of a cent thanks to prompt caching. Without caching, that 1,850-token system prompt gets re-priced on every call in the chain. If your provider supports caching and you're not using it, you're donating money.

Why the receipt is the unit of cost governance

You can't govern what you can't itemize. Monthly dashboards tell you that you spent $40K. The receipt tells you why — which line items to attack, which eval gates earn their keep, which retries are worth preventing.

Three practices that fall out of this:

Emit a receipt per task. Log input/output tokens per span — per agent, per tool call, per retry. If your framework doesn't do this natively, add a callback handler.
Budget at the task level, not the month level. A per-task budget with a circuit breaker ("halt if projected cost exceeds $X") catches runaway loops in seconds instead of at invoice time.
Price your eval gates. Every quality check costs tokens. The receipt tells you whether the gate is cheaper than the failure it prevents.

The next time someone asks what your AI agents cost, don't open the dashboard. Show them a receipt.

Karmendra Pandey is a Practice Architect in AI & ML at TEK Systems, building production agentic AI on AWS. His reference architecture for cost-governed AI agents is at https://doi.org/10.5281/zenodo.23178788, and the companion implementation at https://github.com/karmendra8386/agentic-ai-aws-reference.

I Audited 5 Agent Frameworks for Cost Visibility. Here's the Scorecard.

karmendra pandey — Tue, 06 Oct 2026 14:16:15 +0000

Every agent framework promises observability. Traces, spans, dashboards. But ask a harder question — how much did that agent run cost, and who do I bill? — and the answers get thin fast.

So I audited five of the most-used agent frameworks against one rubric: cost visibility. Not general observability — the specific ability to see, attribute, and control spend. I read the official docs for each (sources linked throughout). No guessing, no marketing pages. Here's the scorecard.

The rubric

Four questions, scored 1–5:

Does it track token usage natively, per run / per agent / per tool call?
Does it convert tokens to dollars?
Can you attribute spend to an agent, session, or user out of the box?
Does it support budgets or cost guardrails natively?

The scorecard

Framework	Score	Verdict
LangGraph + LangSmith	5/5	The gold standard
Amazon Bedrock AgentCore	5/5	Zero-wiring cost visibility
Strands Agents SDK	4/5	Best guardrails, priced in tokens
CrewAI	3/5	Tokens yes, dollars DIY
AutoGen (v0.4)	2/5	Went backwards

LangGraph + LangSmith: 5/5

LangSmith does the thing everyone else stops short of: it converts tokens to dollars automatically, using a customizable per-model pricing table (docs: https://docs.langchain.com/langsmith/cost-tracking). Custom or non-linear pricing goes through usage_metadata. The trace tree breaks cost down per node, so you can separate what the supervisor spent from what the specialist agents spent. Metadata-based attribution covers users and sessions, and there are dashboards on top.

The blind spot: no native budget enforcement. You can see every dollar with precision — you just can't make the framework stop spending them. Visibility without teeth.

Bedrock AgentCore: 5/5

The managed CloudWatch GenAI dashboard shows per-invocation token usage and cost with zero wiring — no callbacks, no exporters, no pricing tables to maintain. Per-user traceability comes through span attributes, and tenant attribution via OpenTelemetry baggage.

The blind spot: spend control is external. It's CloudWatch alarms plus AWS Budgets — both notify-only. The platform will tell you, beautifully, exactly how much you overspent. It won't pull the plug.

Strands Agents SDK: 4/5

Every AgentResult carries automatic per-invocation token metrics — input, output, total, cache tokens, plus per-tool stats (docs: https://strandsagents.com/docs/user-guide/sdk/observability-evaluation/metrics/). And Strands is the only framework in this group with native budget caps: limits={"turns", "output_tokens", "total_tokens"} halts the agent loop with explicit stop reasons.

The blind spot: the caps are token-denominated, and there's no dollar conversion. A 10,000-token cap means wildly different dollars on Haiku vs. Opus. It's the only framework with a native kill-switch — it just thinks in the wrong currency.

CrewAI: 3/5

token_usage on every kickoff() result is genuinely zero-setup, and per-task usage rides along on TaskOutput. The event bus plus OpenTelemetry export gives you the plumbing to build whatever you want (docs: https://docs.crewai.com/v1.15.12/en/observability/overview).

The blind spot: the framework stops at tokens. No pricing, no dashboard, no budgets. Dollar cost is a DIY project, and per-agent attribution needs custom event listeners. You get the raw material of cost visibility and none of the product.

AutoGen (v0.4): 2/5

Here's the irony: AutoGen had built-in cost logging. Version 0.2 shipped start_logging / print_usage_summary with cost computation. Then came the v0.4 rewrite — and cost visibility went backwards. The current telemetry story is native OpenTelemetry with GenAI semantic-convention spans (docs: https://microsoft.github.io/autogen/stable/user-guide/core-user-guide/framework/telemetry.html): token data exists as span attributes for your backend to aggregate. No first-party cost computation, no dashboard, no budgets.

The blind spot: everything. The most popular multi-agent framework in the ecosystem currently treats cost as somebody else's problem — and it used to be better at this.

Three patterns

1. The industry stops at tokens. Only LangSmith and AgentCore convert to dollars natively. Everyone else hands you token counts and wishes you luck with the pricing page. Tokens are an implementation detail; dollars are the budget. The gap between them is where finance teams lose trust in AI projects.

2. Nobody ships a dollar kill-switch. Strands has token caps. AutoGen has a token-limited context. Everything else is alarms and custom logic. As of today, no major framework lets you say "halt this agent if it spends more than $2" — in dollars, natively, enforced. That missing primitive is the single biggest gap in agent cost governance.

3. Visibility went backwards on at least one framework. AutoGen's regression from v0.2 to v0.4 is a cautionary tale: cost features are the first thing cut in a rewrite because they're nobody's launch-blocking requirement — until the invoice arrives.

What to do about it

If you're picking a framework this quarter: LangSmith or AgentCore if you want cost visibility today; Strands if you want guardrails and can live with token-denominated caps. If you're already on CrewAI or AutoGen, budget a sprint for the DIY layer — token export, a pricing table, per-agent attribution, and an external budget alarm — because the framework isn't going to do it for you.

And framework maintainers: the tokens-to-dollars gap is the most valuable ten lines of code in your roadmap. Ship the pricing table. Ship the dollar cap. Your users' CFOs will thank you.

Methodology: scored against official documentation as of October 2026. Karmendra Pandey is a Practice Architect in AI & ML at TEK Systems, working on cost-governed AI agents on AWS. Reference architecture: https://doi.org/10.5281/zenodo.23178788.

How to Audit an ML Design Before It Ships

karmendra pandey — Tue, 06 Oct 2026 05:45:52 +0000

How to Audit an ML Design Before It Ships

Most ML failures are not model failures. They are design failures — and they are almost always visible before launch to anyone who knows where to look.

Training-serving skew nobody checked. An evaluation set that doesn't match production. A feedback loop that will quietly poison the next retraining run. Cost and latency budgets nobody wrote down. I've watched these kill systems that had perfectly good models. After 19+ years of shipping software and reviewing AI research, I now run every ML design through the same five-gate audit before it ships. Here it is.

Gate 1: Data — provenance, leakage, drift

Ask three questions. Where did every training example come from, and can you prove it? Could any information from the future — or from the test set — have leaked into training? And what happens when the real world drifts away from your training distribution?

The classic killer here is eval leakage: the "99% accurate" classifier that fails on its first real day because the eval set was accidentally drawn from training data. It happens more than anyone admits. Demand a written data provenance statement and a leakage check as merge requirements, not nice-to-haves.

Gate 2: Evaluation — does the metric match the mission?

A model can ace its metric and still fail its job. Offline accuracy means nothing if the production decision has different costs for different errors — a fraud model optimized for accuracy will happily approve everything when fraud is rare.

For every model, write down: what decision does this model actually drive, what does a mistake cost in each direction, and does our eval set look like production traffic? If the answers are vague, the design isn't ready.

Gate 3: Operations — latency, cost, fallback

Nobody writes down the latency budget until the first user complaint. Nobody prices the prediction until the first cloud bill. For each model, record: p95 latency budget, cost per 1,000 predictions at expected volume, and what happens when the model is down or too slow. "The page breaks" is not a fallback strategy — cache the last good prediction, degrade to a simpler model, or fail open explicitly and loudly.

For agentic systems this gate matters 10x more: every unnecessary tool call burns tokens and inflates every downstream context window. Cost discipline starts with call discipline.

Gate 4: Security — adversarial inputs and access control

Who can feed inputs to this model, and what can they make it do? Cover the basics: input validation, rate limiting, access controls on the model endpoint and its training data. For LLM-backed systems, add prompt-injection review — treat every tool the agent can call as a privilege, and scope it like one. I model agents as IAM principals: unique identity, least-privilege roles, session-scoped credentials.

Gate 5: Accountability — who owns the decision?

When the model is wrong — and it will be — who is accountable, and what is the remediation path? Every production ML system needs a named owner, a rollback plan, and a written policy for the failure modes you know exist. "The model decided" is not an accountability structure.

The one-page audit

Turn these gates into a single page your team fills out at design review:

Data: provenance documented? leakage check passed? drift monitors planned?
Evaluation: metric matches the mission? eval set production-representative?
Operations: latency budget, cost per prediction, fallback behavior defined?
Security: inputs validated? access scoped? agent tools least-privilege?
Accountability: named owner? rollback plan? known failure modes written down?

Any blank answer blocks the ship. It takes an hour, and it catches the failures that postmortems always describe as "obvious in hindsight."

Karmendra Pandey is a Practice Architect in AI & ML at TEKsystems. He builds production agentic AI systems on AWS and peer-reviews AI research on PREreview.

I Found a Cost Blind Spot in an Open-Source Agent Framework

karmendra pandey — Tue, 06 Oct 2026 05:20:49 +0000

I Found a Cost Blind Spot in an Open-Source Agent Framework

I was building cost attribution for AI agents on AWS when I ran into a gap in the open-source Strands Agents SDK: the framework knew how to talk to Amazon Bedrock models, but it didn't know which family some of them belonged to. That sounds like trivia. It turned out to be a cost-governance blind spot — and fixing it taught me something about how open-source contributions actually happen.

The problem

The Strands harness SDK keeps a registry of model families for Bedrock — it uses the family to make decisions about capabilities and pricing behavior. But several Bedrock model families were missing from the registry. When your agent ran on one of those models, the framework couldn't classify it. Depending on the code path, that meant falling back to defaults that didn't match the model's real cost or capability profile.

In a cost-governed setup, "unknown model family" is not a minor metadata gap. If you can't identify the model, you can't attribute its cost correctly, you can't route to or away from it intelligently, and your burn-rate dashboard has a hole in it.

Finding it

I found it the unglamorous way: reading the source. I was tracing how the SDK resolved model metadata for a Bedrock deployment and noticed the family lookup silently returning nothing for models I knew existed. No error, no warning — just a quiet gap. That's the worst kind of bug in cost infrastructure: it doesn't fail, it just misleads.

I filed it as issue #4852 in the strands-agents/harness-sdk repository, documenting exactly which families were missing and what the downstream effects were.

The fix

The fix itself was small — extending the family registry to cover the missing Bedrock families, with tests proving each new mapping resolved correctly. I added four new tests, ran the full suite (152 passed), and confirmed the 13 pre-existing failures also failed on clean main, so they weren't mine.

Small fix, but the verification mattered more than the code. In cost infrastructure, a wrong mapping is worse than a missing one — it makes your dashboards lie with confidence.

What I learned

Three things worth sharing:

Read the source before the docs. The docs described the registry as complete. The source said otherwise. For cost-critical paths, the source is the only documentation you can trust.
File the issue before the PR. Writing up #4852 first forced me to articulate the impact, not just the bug. That made the fix obviously correct to reviewers.
Test the mapping, not just the code. The interesting assertions weren't "does it run" but "does this model resolve to the right family" — the semantic correctness that cost attribution depends on.

Why this matters beyond one PR

Every team running agents in production is building cost governance on top of frameworks like this one. The frameworks are young, the model landscape changes monthly, and gaps like this one are everywhere — quiet, unremarkable, and expensive in aggregate.

If you're running agents on Bedrock, check your model metadata paths. The bug you find might be as boring as a missing registry entry. Fix it anyway. Your burn-rate dashboard will thank you.

P.S. — pricing the whole task, not just the mapping. Fixing the metadata was one half of the cost story. The other half is knowing what a full agent run costs before the bill arrives. I built tokenecon for that — pip install tokenecon, point its CostTracker at a Strands agent, and every run prints a per-step cost receipt. It's the companion to the paper Token-Economics for Agentic AI.

The fix is PR #4853 in strands-agents/harness-sdk. Karmendra Pandey is a Practice Architect in AI & ML at TEKsystems.

Your AI Agent Has a Burn Rate: Governing the Cost of Production AI Agents on AWS

karmendra pandey — Tue, 06 Oct 2026 05:19:55 +0000

Your AI Agent Has a Burn Rate: Governing the Cost of Production AI Agents on AWS

Every AI pilot I have reviewed in the last two years had the same surprise waiting in production: the invoice. A demo that cost $40 to run suddenly costs $4,000 a month at real traffic. Nobody planned for it because nobody was measuring it. Your AI agent has a burn rate — cost per task, per user, per day — and if you are not governing it, you are flying blind.

This is the practitioner's guide to cost-governed agents on AWS: what to measure, where the money actually goes, and the architecture patterns that keep the burn rate under control.

Where the money goes

For a typical agent built on Amazon Bedrock, the cost stack looks like this:

Model inference — the big one. Every token in and out is metered. Long system prompts, verbose reasoning traces, and multi-step agent loops multiply this fast.
Retrieval — OpenSearch or Kendra queries per agent step. Cheap per call, expensive at scale.
Tool calls — Lambda invocations, API calls, data transfers. Each agent step can trigger several.
Retries and loops — the silent killer. An agent that retries a failing step five times just quintupled that task's cost.

Most teams only see line 1 on the bill. The real burn rate is the sum of all four, attributed per task.

Pattern 1: Attribute cost per task

You cannot govern what you cannot attribute. Every agent execution should log: input tokens, output tokens, model used, tool calls made, and wall-clock time — tagged with a task ID and a user or tenant ID. On AWS, this means structured logging from your Bedrock invocations into CloudWatch, with a pipeline that rolls it up per task.

The moment you can say "this customer support task cost $0.42 and this one cost $3.80," you can start asking why — and that question is where the savings live.

Pattern 2: Route, don't default

Not every task needs the flagship model. A classification step, a format check, a simple extraction — these run fine on smaller, cheaper models at a tenth of the cost. The expensive model should be reserved for the steps that actually need judgment.

This is the routing pattern: a lightweight decision at each step picks the cheapest model capable of handling it. In practice, teams that add routing cut inference spend 40–70% with no measurable quality drop — because most agent steps were never hard enough to justify the big model in the first place.

Pattern 3: Cap the loop

Agents loop. Sometimes they loop forever. Every production agent needs:

A max-iteration cap per task (if it hasn't converged in N steps, it won't).
A cost ceiling per task — abort or escalate to a human when the spend crosses the line.
Idempotent tool calls — so a retry doesn't double-charge a payment API or re-send a customer email.

These are not optimizations. They are the guardrails that keep one bad task from burning a day's budget.

Pattern 4: Cache the predictable

A surprising share of agent work is repetitive: the same policy lookup, the same product description, the same classification of the same ticket type. Cache retrieval results and deterministic step outputs aggressively. Bedrock's prompt caching and a simple Redis layer in front of your retrieval calls compound into real savings.

The burn-rate dashboard

Put it together and you get one view that matters: cost per task, trended over time, broken down by model, step type, and tenant. When a new model version ships or a prompt changes, the dashboard tells you within hours whether the burn rate moved — before the invoice does.

Start this week

You don't need a platform rebuild. Pick one agent, add per-task cost logging, and look at the numbers for a week. The waste will introduce itself. Then route one step to a smaller model, cap one loop, and cache one retrieval call. Measure again.

Cost governance isn't about spending less on AI. It's about knowing what each dollar bought — and making sure the expensive dollars went to the steps that deserved them.

Karmendra Pandey is a Practice Architect in AI & ML at TEKsystems, where he designs production AI agent systems on AWS.