DEV Community: karmendra pandey

The Receipt: I Priced Every Token of One AI Agent Task

karmendra pandey — Tue, 06 Oct 2026 14:16:57 +0000

Everyone talks about AI costs in aggregate. Dashboards show monthly spend, per-model totals, team budgets. Nobody shows you a receipt.

So here's one. A single AI agent task, priced line by line — every token, every tool call, every retry. This is a worked example from production-style workloads I describe in my reference architecture work (full paper: https://doi.org/10.5281/zenodo.23178788). The numbers are representative of a mid-complexity support-triage agent running on AWS Bedrock. Your numbers will differ. The shape of the receipt won't.

The task

A customer support triage agent: read an incoming ticket, pull the customer's history, check the knowledge base, decide whether to auto-resolve or escalate, and draft the response. One task, one receipt.

The receipt

#	Line item	Tokens in	Tokens out	Unit cost basis	Cost
1	System prompt (loaded once, cached)	1,850	—	cached input	$0.0011
2	Ticket text + customer history	2,300	—	standard input	$0.0069
3	Knowledge base retrieval (3 chunks)	4,100	—	standard input	$0.0123
4	Reasoning: triage decision	—	420	standard output	$0.0050
5	Tool call: CRM lookup	180	90	in/out	$0.0016
6	Reasoning: draft response	—	680	standard output	$0.0082
7	Retry: first draft failed tone eval, regenerated	6,200	710	in (cached) + out	$0.0121
8	Embedding: ticket for memory store	340	—	embedding	$0.0000
9	Final response + audit log write	—	350	standard output	$0.0042
	Total	~15,000	~2,250		~$0.051

Wait — $0.051, not $0.23? The $0.23 figure from my paper is the fully-loaded cost: it includes the amortized cost of the eval harness, the retrieval infrastructure, and the human review queue time for escalations. The raw inference receipt is five cents. The governed cost is twenty-three cents. Both numbers matter, and most teams track neither.

Reading the receipt

Three things jump out:

1. The retry cost more than the original attempt. Line 7 — a failed tone evaluation forced a regeneration — cost $0.0121, more than lines 4+6 combined. Retries are the silent budget killer in agent systems. Every eval gate you add has a cost; every gate you skip has a bigger one. The receipt lets you price the tradeoff instead of guessing.

2. Context is the bulk of input spend. Lines 2+3 are 6,400 input tokens — 43% of the total input. That knowledge-base retrieval is doing real work, but it's also the first place to optimize: better chunking, semantic caching of frequent queries, and prompt compression all attack this line directly.

3. The system prompt is nearly free — because it's cached. Line 1 costs a tenth of a cent thanks to prompt caching. Without caching, that 1,850-token system prompt gets re-priced on every call in the chain. If your provider supports caching and you're not using it, you're donating money.

Why the receipt is the unit of cost governance

You can't govern what you can't itemize. Monthly dashboards tell you that you spent $40K. The receipt tells you why — which line items to attack, which eval gates earn their keep, which retries are worth preventing.

Three practices that fall out of this:

Emit a receipt per task. Log input/output tokens per span — per agent, per tool call, per retry. If your framework doesn't do this natively, add a callback handler.
Budget at the task level, not the month level. A per-task budget with a circuit breaker ("halt if projected cost exceeds $X") catches runaway loops in seconds instead of at invoice time.
Price your eval gates. Every quality check costs tokens. The receipt tells you whether the gate is cheaper than the failure it prevents.

The next time someone asks what your AI agents cost, don't open the dashboard. Show them a receipt.

Karmendra Pandey is a Practice Architect in AI & ML at TEK Systems, building production agentic AI on AWS. His reference architecture for cost-governed AI agents is at https://doi.org/10.5281/zenodo.23178788, and the companion implementation at https://github.com/karmendra8386/agentic-ai-aws-reference.

I Audited 5 Agent Frameworks for Cost Visibility. Here's the Scorecard.

karmendra pandey — Tue, 06 Oct 2026 14:16:15 +0000

Every agent framework promises observability. Traces, spans, dashboards. But ask a harder question — how much did that agent run cost, and who do I bill? — and the answers get thin fast.

So I audited five of the most-used agent frameworks against one rubric: cost visibility. Not general observability — the specific ability to see, attribute, and control spend. I read the official docs for each (sources linked throughout). No guessing, no marketing pages. Here's the scorecard.

The rubric

Four questions, scored 1–5:

Does it track token usage natively, per run / per agent / per tool call?
Does it convert tokens to dollars?
Can you attribute spend to an agent, session, or user out of the box?
Does it support budgets or cost guardrails natively?

The scorecard

Framework	Score	Verdict
LangGraph + LangSmith	5/5	The gold standard
Amazon Bedrock AgentCore	5/5	Zero-wiring cost visibility
Strands Agents SDK	4/5	Best guardrails, priced in tokens
CrewAI	3/5	Tokens yes, dollars DIY
AutoGen (v0.4)	2/5	Went backwards

LangGraph + LangSmith: 5/5

LangSmith does the thing everyone else stops short of: it converts tokens to dollars automatically, using a customizable per-model pricing table (docs: https://docs.langchain.com/langsmith/cost-tracking). Custom or non-linear pricing goes through usage_metadata. The trace tree breaks cost down per node, so you can separate what the supervisor spent from what the specialist agents spent. Metadata-based attribution covers users and sessions, and there are dashboards on top.

The blind spot: no native budget enforcement. You can see every dollar with precision — you just can't make the framework stop spending them. Visibility without teeth.

Bedrock AgentCore: 5/5

The managed CloudWatch GenAI dashboard shows per-invocation token usage and cost with zero wiring — no callbacks, no exporters, no pricing tables to maintain. Per-user traceability comes through span attributes, and tenant attribution via OpenTelemetry baggage.

The blind spot: spend control is external. It's CloudWatch alarms plus AWS Budgets — both notify-only. The platform will tell you, beautifully, exactly how much you overspent. It won't pull the plug.

Strands Agents SDK: 4/5

Every AgentResult carries automatic per-invocation token metrics — input, output, total, cache tokens, plus per-tool stats (docs: https://strandsagents.com/docs/user-guide/sdk/observability-evaluation/metrics/). And Strands is the only framework in this group with native budget caps: limits={"turns", "output_tokens", "total_tokens"} halts the agent loop with explicit stop reasons.

The blind spot: the caps are token-denominated, and there's no dollar conversion. A 10,000-token cap means wildly different dollars on Haiku vs. Opus. It's the only framework with a native kill-switch — it just thinks in the wrong currency.

CrewAI: 3/5

token_usage on every kickoff() result is genuinely zero-setup, and per-task usage rides along on TaskOutput. The event bus plus OpenTelemetry export gives you the plumbing to build whatever you want (docs: https://docs.crewai.com/v1.15.12/en/observability/overview).

The blind spot: the framework stops at tokens. No pricing, no dashboard, no budgets. Dollar cost is a DIY project, and per-agent attribution needs custom event listeners. You get the raw material of cost visibility and none of the product.

AutoGen (v0.4): 2/5

Here's the irony: AutoGen had built-in cost logging. Version 0.2 shipped start_logging / print_usage_summary with cost computation. Then came the v0.4 rewrite — and cost visibility went backwards. The current telemetry story is native OpenTelemetry with GenAI semantic-convention spans (docs: https://microsoft.github.io/autogen/stable/user-guide/core-user-guide/framework/telemetry.html): token data exists as span attributes for your backend to aggregate. No first-party cost computation, no dashboard, no budgets.

The blind spot: everything. The most popular multi-agent framework in the ecosystem currently treats cost as somebody else's problem — and it used to be better at this.

Three patterns

1. The industry stops at tokens. Only LangSmith and AgentCore convert to dollars natively. Everyone else hands you token counts and wishes you luck with the pricing page. Tokens are an implementation detail; dollars are the budget. The gap between them is where finance teams lose trust in AI projects.

2. Nobody ships a dollar kill-switch. Strands has token caps. AutoGen has a token-limited context. Everything else is alarms and custom logic. As of today, no major framework lets you say "halt this agent if it spends more than $2" — in dollars, natively, enforced. That missing primitive is the single biggest gap in agent cost governance.

3. Visibility went backwards on at least one framework. AutoGen's regression from v0.2 to v0.4 is a cautionary tale: cost features are the first thing cut in a rewrite because they're nobody's launch-blocking requirement — until the invoice arrives.

What to do about it

If you're picking a framework this quarter: LangSmith or AgentCore if you want cost visibility today; Strands if you want guardrails and can live with token-denominated caps. If you're already on CrewAI or AutoGen, budget a sprint for the DIY layer — token export, a pricing table, per-agent attribution, and an external budget alarm — because the framework isn't going to do it for you.

And framework maintainers: the tokens-to-dollars gap is the most valuable ten lines of code in your roadmap. Ship the pricing table. Ship the dollar cap. Your users' CFOs will thank you.

Methodology: scored against official documentation as of October 2026. Karmendra Pandey is a Practice Architect in AI & ML at TEK Systems, working on cost-governed AI agents on AWS. Reference architecture: https://doi.org/10.5281/zenodo.23178788.

How to Audit an ML Design Before It Ships

karmendra pandey — Tue, 06 Oct 2026 05:45:52 +0000

How to Audit an ML Design Before It Ships

Most ML failures are not model failures. They are design failures — and they are almost always visible before launch to anyone who knows where to look.

Training-serving skew nobody checked. An evaluation set that doesn't match production. A feedback loop that will quietly poison the next retraining run. Cost and latency budgets nobody wrote down. I've watched these kill systems that had perfectly good models. After 19+ years of shipping software and reviewing AI research, I now run every ML design through the same five-gate audit before it ships. Here it is.

Gate 1: Data — provenance, leakage, drift

Ask three questions. Where did every training example come from, and can you prove it? Could any information from the future — or from the test set — have leaked into training? And what happens when the real world drifts away from your training distribution?

The classic killer here is eval leakage: the "99% accurate" classifier that fails on its first real day because the eval set was accidentally drawn from training data. It happens more than anyone admits. Demand a written data provenance statement and a leakage check as merge requirements, not nice-to-haves.

Gate 2: Evaluation — does the metric match the mission?

A model can ace its metric and still fail its job. Offline accuracy means nothing if the production decision has different costs for different errors — a fraud model optimized for accuracy will happily approve everything when fraud is rare.

For every model, write down: what decision does this model actually drive, what does a mistake cost in each direction, and does our eval set look like production traffic? If the answers are vague, the design isn't ready.

Gate 3: Operations — latency, cost, fallback

Nobody writes down the latency budget until the first user complaint. Nobody prices the prediction until the first cloud bill. For each model, record: p95 latency budget, cost per 1,000 predictions at expected volume, and what happens when the model is down or too slow. "The page breaks" is not a fallback strategy — cache the last good prediction, degrade to a simpler model, or fail open explicitly and loudly.

For agentic systems this gate matters 10x more: every unnecessary tool call burns tokens and inflates every downstream context window. Cost discipline starts with call discipline.

Gate 4: Security — adversarial inputs and access control

Who can feed inputs to this model, and what can they make it do? Cover the basics: input validation, rate limiting, access controls on the model endpoint and its training data. For LLM-backed systems, add prompt-injection review — treat every tool the agent can call as a privilege, and scope it like one. I model agents as IAM principals: unique identity, least-privilege roles, session-scoped credentials.

Gate 5: Accountability — who owns the decision?

When the model is wrong — and it will be — who is accountable, and what is the remediation path? Every production ML system needs a named owner, a rollback plan, and a written policy for the failure modes you know exist. "The model decided" is not an accountability structure.

The one-page audit

Turn these gates into a single page your team fills out at design review:

Data: provenance documented? leakage check passed? drift monitors planned?
Evaluation: metric matches the mission? eval set production-representative?
Operations: latency budget, cost per prediction, fallback behavior defined?
Security: inputs validated? access scoped? agent tools least-privilege?
Accountability: named owner? rollback plan? known failure modes written down?

Any blank answer blocks the ship. It takes an hour, and it catches the failures that postmortems always describe as "obvious in hindsight."

Karmendra Pandey is a Practice Architect in AI & ML at TEKsystems. He builds production agentic AI systems on AWS and peer-reviews AI research on PREreview.

I Found a Cost Blind Spot in an Open-Source Agent Framework

karmendra pandey — Tue, 06 Oct 2026 05:20:49 +0000

I Found a Cost Blind Spot in an Open-Source Agent Framework

I was building cost attribution for AI agents on AWS when I ran into a gap in the open-source Strands Agents SDK: the framework knew how to talk to Amazon Bedrock models, but it didn't know which family some of them belonged to. That sounds like trivia. It turned out to be a cost-governance blind spot — and fixing it taught me something about how open-source contributions actually happen.

The problem

The Strands harness SDK keeps a registry of model families for Bedrock — it uses the family to make decisions about capabilities and pricing behavior. But several Bedrock model families were missing from the registry. When your agent ran on one of those models, the framework couldn't classify it. Depending on the code path, that meant falling back to defaults that didn't match the model's real cost or capability profile.

In a cost-governed setup, "unknown model family" is not a minor metadata gap. If you can't identify the model, you can't attribute its cost correctly, you can't route to or away from it intelligently, and your burn-rate dashboard has a hole in it.

Finding it

I found it the unglamorous way: reading the source. I was tracing how the SDK resolved model metadata for a Bedrock deployment and noticed the family lookup silently returning nothing for models I knew existed. No error, no warning — just a quiet gap. That's the worst kind of bug in cost infrastructure: it doesn't fail, it just misleads.

I filed it as issue #4852 in the strands-agents/harness-sdk repository, documenting exactly which families were missing and what the downstream effects were.

The fix

The fix itself was small — extending the family registry to cover the missing Bedrock families, with tests proving each new mapping resolved correctly. I added four new tests, ran the full suite (152 passed), and confirmed the 13 pre-existing failures also failed on clean main, so they weren't mine.

Small fix, but the verification mattered more than the code. In cost infrastructure, a wrong mapping is worse than a missing one — it makes your dashboards lie with confidence.

What I learned

Three things worth sharing:

Read the source before the docs. The docs described the registry as complete. The source said otherwise. For cost-critical paths, the source is the only documentation you can trust.
File the issue before the PR. Writing up #4852 first forced me to articulate the impact, not just the bug. That made the fix obviously correct to reviewers.
Test the mapping, not just the code. The interesting assertions weren't "does it run" but "does this model resolve to the right family" — the semantic correctness that cost attribution depends on.

Why this matters beyond one PR

Every team running agents in production is building cost governance on top of frameworks like this one. The frameworks are young, the model landscape changes monthly, and gaps like this one are everywhere — quiet, unremarkable, and expensive in aggregate.

If you're running agents on Bedrock, check your model metadata paths. The bug you find might be as boring as a missing registry entry. Fix it anyway. Your burn-rate dashboard will thank you.

The fix is PR #4853 in strands-agents/harness-sdk. Karmendra Pandey is a Practice Architect in AI & ML at TEKsystems.

Your AI Agent Has a Burn Rate: Governing the Cost of Production AI Agents on AWS

karmendra pandey — Tue, 06 Oct 2026 05:19:55 +0000

Your AI Agent Has a Burn Rate: Governing the Cost of Production AI Agents on AWS

Every AI pilot I have reviewed in the last two years had the same surprise waiting in production: the invoice. A demo that cost $40 to run suddenly costs $4,000 a month at real traffic. Nobody planned for it because nobody was measuring it. Your AI agent has a burn rate — cost per task, per user, per day — and if you are not governing it, you are flying blind.

This is the practitioner's guide to cost-governed agents on AWS: what to measure, where the money actually goes, and the architecture patterns that keep the burn rate under control.

Where the money goes

For a typical agent built on Amazon Bedrock, the cost stack looks like this:

Model inference — the big one. Every token in and out is metered. Long system prompts, verbose reasoning traces, and multi-step agent loops multiply this fast.
Retrieval — OpenSearch or Kendra queries per agent step. Cheap per call, expensive at scale.
Tool calls — Lambda invocations, API calls, data transfers. Each agent step can trigger several.
Retries and loops — the silent killer. An agent that retries a failing step five times just quintupled that task's cost.

Most teams only see line 1 on the bill. The real burn rate is the sum of all four, attributed per task.

Pattern 1: Attribute cost per task

You cannot govern what you cannot attribute. Every agent execution should log: input tokens, output tokens, model used, tool calls made, and wall-clock time — tagged with a task ID and a user or tenant ID. On AWS, this means structured logging from your Bedrock invocations into CloudWatch, with a pipeline that rolls it up per task.

The moment you can say "this customer support task cost $0.42 and this one cost $3.80," you can start asking why — and that question is where the savings live.

Pattern 2: Route, don't default

Not every task needs the flagship model. A classification step, a format check, a simple extraction — these run fine on smaller, cheaper models at a tenth of the cost. The expensive model should be reserved for the steps that actually need judgment.

This is the routing pattern: a lightweight decision at each step picks the cheapest model capable of handling it. In practice, teams that add routing cut inference spend 40–70% with no measurable quality drop — because most agent steps were never hard enough to justify the big model in the first place.

Pattern 3: Cap the loop

Agents loop. Sometimes they loop forever. Every production agent needs:

A max-iteration cap per task (if it hasn't converged in N steps, it won't).
A cost ceiling per task — abort or escalate to a human when the spend crosses the line.
Idempotent tool calls — so a retry doesn't double-charge a payment API or re-send a customer email.

These are not optimizations. They are the guardrails that keep one bad task from burning a day's budget.

Pattern 4: Cache the predictable

A surprising share of agent work is repetitive: the same policy lookup, the same product description, the same classification of the same ticket type. Cache retrieval results and deterministic step outputs aggressively. Bedrock's prompt caching and a simple Redis layer in front of your retrieval calls compound into real savings.

The burn-rate dashboard

Put it together and you get one view that matters: cost per task, trended over time, broken down by model, step type, and tenant. When a new model version ships or a prompt changes, the dashboard tells you within hours whether the burn rate moved — before the invoice does.

Start this week

You don't need a platform rebuild. Pick one agent, add per-task cost logging, and look at the numbers for a week. The waste will introduce itself. Then route one step to a smaller model, cap one loop, and cache one retrieval call. Measure again.

Cost governance isn't about spending less on AI. It's about knowing what each dollar bought — and making sure the expensive dollars went to the steps that deserved them.

Karmendra Pandey is a Practice Architect in AI & ML at TEKsystems, where he designs production AI agent systems on AWS.

How AI is changing data processing and insight

karmendra pandey — Fri, 20 Jun 2025 05:22:33 +0000

🔄 From Static Pipelines to Adaptive Intelligence
Traditional ETL pipelines were rigid: predefined schemas, manual transformations, static schedules. Now, AI introduces:
• Auto-adjusting workflows: Pipelines adapt based on schema drift, error rates, or downstream demand.
• Semantic understanding: LLMs can interpret column meanings, detect sensitive data, or auto-document datasets.
It’s not just about moving data anymore—it’s about understanding it as it moves.

⚙️ Processing Power Meets Prediction
AI accelerates and enriches processing by:
• Pattern recognition: Detecting anomalies in real-time log streams or IoT sensor feeds.
• Entity extraction: Parsing legal contracts, medical records, or video transcripts at scale (think: media analytics + Rekognition).
• Data summarization: Quickly distilling terabytes into consumable formats—crucial for analysts, execs, or even fine-tuned dashboards.

🔍 Insight Generation Becomes Conversational
No more staring at dashboards hoping the insights jump out. AI enables:
• Natural language Q&A over your data (via Bedrock agents or BI copilots).
• Automated storytelling: Reports that highlight why KPIs changed, not just what changed.
• Forecasting & simulation: What-if models baked into insight tools, not buried in notebooks.
It’s the difference between “What are the numbers?” and “What do I do next?”

🔁 Continuous Learning & Feedback
AI-driven systems evolve:
• User behavior fine-tunes what’s shown or flagged.
• Drift detection retriggers retraining or pipeline updates.
• Closed-loop systems (especially in DevOps or media pipelines) correct issues without manual intervention.