DEV Community: Akash Melavanki

Your AI Agent Just Spent $200 on a $2 Task. Here's Why Nobody Warned You

Akash Melavanki — Tue, 02 Jun 2026 14:39:01 +0000

And why the tools we have right now aren't built for this.

I want to talk about something that's quietly becoming a real problem as more people ship autonomous agents into production — and nobody's really naming it clearly.

We're not talking about prompt injection or model hallucinations here. We're talking about something more boring and more expensive: your agent running off the rails and you not knowing until the bill lands.

The Problem Nobody Warned You About

Here's how it usually plays out.

You build an agent. It works perfectly in testing. You ship it. A few days later you open your OpenAI billing dashboard and something is... off. One run cost $47. A $2 research task. You dig in, and somewhere in the logs you find it — the agent hit a bad tool call, got a weird response, and started retrying. 80 times. Nobody stopped it.

That's not a bug in your LLM. That's not a hallucination. That's just what autonomous agents do when there's no guardrail.

The problem is structural. When you write while (!done) and hand control to an LLM, you're trusting a non-deterministic system to know when to stop. Sometimes it doesn't. And unlike a crashed server or a 500 error — a runaway agent keeps working. It looks healthy. It's just spending.

Why "Just Set max_iterations=10" Doesn't Cut It

I've seen this answer come up a lot. It makes sense at first glance — cap the loops, problem solved.

But here's what it doesn't cover:

1. Cost isn't uniform across iterations.
One step might cost $0.003. Another might cost $3.00 depending on the model, the prompt size, and what tools were called. Iteration count tells you nothing about money.

2. Loops don't look like loops.
A stuck agent doesn't always repeat the exact same call. It might slightly permute the prompt each time — same semantic intent, different tokens. max_iterations catches the obvious case. Loop signature detection catches the real ones.

3. Multiple services, multiple agents.
The moment you have more than one agent running — across services, across team members, across environments — where does your max_iterations config live? In a .env file in each repo? Good luck keeping those consistent when you push a hotfix at 2am.

4. Nobody can see what's happening.
Your console.log statements don't survive a process restart. Your finance lead can't query them. Your teammate who shipped a different agent can't see if there's a pattern emerging across runs.

The iteration cap is duct tape. It works for one developer protecting one script in a weekend project. It doesn't work for a team shipping agents to real users.

The Deeper Issue: Agents Are Actors, Not Functions

This is the thing that took me a while to fully internalize.

When you call a normal function, it runs, it returns, it's done. You have deterministic cost. Predictable behavior. Easy to reason about.

An autonomous agent is different. It's not a function call — it's a process that makes decisions. It decides what tools to call, in what order, how many times. It decides when it thinks the task is done. You set the objective. The agent figures out the path.

That's the whole point of agentic AI. That's why it's powerful.

But it also means the cost model is completely different. You're not paying per-call anymore. You're paying for a sequence of decisions you didn't make. And if one of those decisions is wrong — a bad retry strategy, a hallucinated tool call, a reasoning loop — you're paying for that too.

We built observability and governance for the old world — where you call an API, you know the cost upfront, and it's done. We haven't fully built it for this new world yet.

What Actually Needs to Exist

When I started thinking about this seriously, I realized there's a specific set of things that need to be true for agents to be safe in production:

Hard budget ceilings per run. Not soft warnings. Not "we'll alert you". The agent stops when it hits the ceiling. With a partial result. Before more money is spent.

Loop detection at the semantic level. Not just counting iterations — actually detecting when the agent is spinning on the same reasoning pattern.

A kill switch that lives outside your code. If your agent is running in service-A, you shouldn't have to redeploy service-A to stop a runaway run. The kill switch should be callable from a dashboard, an API, a webhook — anything.

Persistent audit logs. Every step, every tool call, every decision — logged somewhere that doesn't disappear when the process exits. Not for debugging. For governance. So you can ask "what did this agent actually do, and why?"

Policy that's centralized. Not per-service config files. One place where you define the rules, and every agent your team ships inherits them.

That last one is the key insight. The difference between "add a budget library to your project" and "have actual governance" is whether the policy lives above the code or inside it.

The Analogy That Made It Click for Me

Think about how Datadog relates to your application metrics.

You could instrument your app with Prometheus locally and write metrics to stdout. That works for one app. But the moment you have multiple services, you need a layer above your code — a place where metrics from all of them converge, where you set alerts, where your whole team can see what's happening.

That's the shape of what agent governance needs to look like. Not a library you add to each project. A control plane that sits above all of them.

Library runs in your process. Stops when your process stops. Lives in one repo.

Control plane runs above your code. Persists across deploys. Spans every agent your team ships.

I Built Something in This Space — Looking for Design Partners

I've been working on exactly this problem. It's called Thskyshield — a runtime governance layer for autonomous agents.

The idea is simple: you wrap your agent loop with three SDK calls — beginRun, beforeStep, afterStep — and the control plane handles the rest. Hard budget ceilings, loop detection, kill switch, step-by-step audit trail. Sub-10ms enforcement (so it doesn't add latency to your actual LLM calls). Works with LangGraph, CrewAI, OpenAI Agents SDK, or whatever you're using.

The policy lives in your dashboard, not in your code. Change a limit without touching a deploy. See what every agent your team shipped actually did.

const run = await shield.beginRun({
  agent: 'research-agent',
  budgetLimitUsd: 2.00,
  iterationLimit: 30,
  loopThreshold: 5,
})

try {
  while (!done) {
    const ok = await run.beforeStep({ tool, args, estimatedTokens })
    if (!ok.allowed) break
    const result = await callLLMAndTool()
    await run.afterStep({ tokens, toolResult })
  }
} catch (e) {
  if (e instanceof ShieldKilledError) {
    console.log(`Stopped: ${e.reason}. Spent: $${e.spent}`)
  }
}

Five lines. Drop into any agent loop.

MVP is live. I'm actively working with a small group of teams to shape the agent SDK — specifically want to talk to people who are shipping real agents and hitting real problems. First five design partners get it free forever.

If you're building in this space and this problem sounds familiar — I'd genuinely love to talk. Drop a comment or reach out at thskyshield.com/contact.

The agent era is real. The tooling to govern it safely is still catching up. That gap is what I'm working on.

If you're building agents in production and haven't thought about this yet — now's the time.

#ai #agents #llm #devops #opensource #typescript #buildinpublic #webdev

Vercel got hacked. Your API keys rotated. You're still not safe.

Akash Melavanki — Tue, 21 Apr 2026 16:44:56 +0000

I host Thskyshield on Vercel.

So when I woke up to the news that Vercel had been breached — internal systems compromised, customer environment variables exposed, data allegedly being sold on BreachForums for $2 million — I didn't panic. I rotated my keys immediately, like everyone else.

And then I sat with a question that I don't think enough developers are asking:

What happens in the window between a key being stolen and you rotating it?

That question is literally why I built what I built.

What actually happened at Vercel

Let me explain the attack chain, because it's more interesting than "Vercel got hacked."

A Context.ai employee got hit with Lumma Stealer malware in February 2026. Lumma is an infostealer — it quietly harvests credentials, OAuth tokens, session cookies. The attacker sat on those credentials for weeks.

Then they used the stolen OAuth token to access Vercel's Google Workspace. That gave them access to Vercel's internal environments. And in those environments sat thousands of customer environment variables — API keys, database credentials, signing tokens — that weren't marked as "sensitive."

One employee. One third-party tool. One overly permissive OAuth grant. And suddenly an attacker has the keys to a significant chunk of the web's developer infrastructure.

This is what a supply chain attack looks like in 2026. It's not brute force. It's patient, precise, and automated.

The standard advice is incomplete

"Rotate your keys immediately."

Yes. Obviously. Do that.

But here's what nobody's talking about: key rotation is reactive. You rotate after you know you're compromised. The attacker who stole your key at 2 AM on a Sunday doesn't wait for Monday morning. They act the moment they have it.

With AI-powered automation, that window between theft and damage is now measured in seconds, not hours. Google's Threat Intelligence team found that the time between initial access and full breach has collapsed from 8 hours in 2022 to 22 seconds in 2025. The attacker doesn't need you to be asleep. They're done before you finish reading the breach notification email.

So yes, rotate your keys. But also ask: if this key gets stolen tonight, what is the maximum damage the attacker can do with it?

What a stolen LLM API key actually enables

Here's where I want to be honest about scope, because I think most people only think about one dimension of this.

A stolen OpenAI, Anthropic, or Gemini API key gives an attacker several options:

1. Denial of Wallet — loop your chatbot endpoint with high-token payloads. Max out your billing. Leave you with a $5,000 invoice by sunrise.

2. Data exfiltration — if your LLM calls include user data, system prompts, or sensitive context, the attacker can extract that by replaying your own endpoints.

3. Content generation at your cost — use your key as a free compute resource. Generate content, run agents, build products — all billed to you.

4. Prompt injection into your users — if the attacker can make calls to your endpoint, they may be able to manipulate the responses your actual users see.

I want to be clear: I'm only solving one of these. Thskyshield is a financial kill-switch. It stops the billing damage. It does not stop data exfiltration. It doesn't block malicious prompt injection. There are other tools for those layers.

What I believe is: every attack that costs money gets stopped at a known ceiling. If your OpenAI key is stolen and the attacker tries a Denial of Wallet attack, they can only drain up to whatever daily limit you set per user. Not $5,000. Not $500. Whatever you decided.

That ceiling exists even if everything else failed. Even if Vercel leaked your key. Even if the attacker has full access. The financial blast radius is bounded.

Why AI makes this more urgent, not less

I've had this instinct for a while, and the data is starting to confirm it.

Attacks are getting faster because the automation is getting smarter. The Lumma Stealer that hit the Context.ai employee — that's not a human manually harvesting credentials. That's a piece of software running autonomously, finding credentials, exfiltrating them, and handing them off to an operator in real time.

The same automation that makes AI products useful makes AI-powered attacks cheap to run at scale. A Denial of Wallet attack used to require someone sitting at a keyboard, writing a loop script, running it manually. Today it's a five-line agentic task: "Loop this endpoint until the budget hits $X or you get a 429."

The attack surface for every AI product is a financial endpoint. Every chatbot, every AI feature, every LLM-powered tool has a cost function. And attackers are starting to understand that better than most developers do.

The thing I keep coming back to

The Vercel breach is not an outlier. LiteLLM in March. Axios in March. Context.ai in February. Vercel in April.

Every one of these is a supply chain attack. Every one of them exposed developer credentials. Every one of them happened through trusted infrastructure — things developers rely on every day without thinking twice.

You can't stop the breaches. You can't guarantee your keys won't be stolen. What you can control is what happens after.

Rotate your keys — yes. But also put a ceiling on what a stolen key can do to your business.

That's the layer most developers don't have yet.

If you're building with LLM APIs and you want a hard limit on what a stolen key can drain: thskyshield.com

Or if you want to watch a simulated Denial of Wallet attack fire in real time: thskyshield.com/simulator

Curious what others are doing about this. Are you relying on provider-side limits? Rolling your own governance? Or just hoping it doesn't happen to you?

Rate limiting your LLM API is useless. Here's what actually protects you.

Akash Melavanki — Tue, 14 Apr 2026 13:32:34 +0000

Last month, the LiteLLM supply chain attack exposed API keys across thousands of developer projects.

The standard advice: rotate your keys immediately.

Here's what nobody tells you after that: a rotated key doesn't protect you from the next attack. Rate limiting doesn't either. I'll show you why — and what actually works.

The problem with rate limiting LLMs

Rate limiting assumes all requests cost roughly the same. For traditional APIs, that's true.

For LLMs, it's completely wrong.

Request 1: "Hi"
→ ~10 tokens → cost: $0.0001

Request 2: "Summarize this 50-page PDF"
→ ~30,000 tokens → cost: $0.45

An attacker doesn't need high volume. They just need expensive requests. 10 requests per minute means nothing when each request costs $0.45.

What you actually need is budget limiting — enforcing a maximum dollar spend per user, per day, in real time.

The race condition nobody talks about

OK, so you decide to track spend in Redis. Simple, right?

Wrong. Here's what happens at scale.

Your app receives 10 concurrent requests from the same user.

Instance A reads budget: $0.05 remaining. Proceeds.
Instance B reads budget: $0.05 remaining. Proceeds.
Instance C reads budget: $0.05 remaining. Proceeds.
...all 10 instances read $0.05 and proceed.

All 10 fire $1.00 LLM requests. Your user's budget was $1.00. You just spent $10.00.

This is the race condition. Standard Redis GET + SET cannot solve it — there's always a gap between reading and writing where another instance sneaks through.

The fix: atomic Lua scripts

The solution is to move the entire check-and-update logic into a single atomic operation inside Redis. Lua scripts on Redis run as one uninterruptible step — no interleaving, no race condition possible.

-- runs atomically inside Redis — no race condition possible
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local cost = tonumber(ARGV[2])
local current = tonumber(redis.call('GET', key) or "0")

if current + cost > limit then
  return 0 -- BLOCK
end

redis.call('INCRBYFLOAT', key, cost)
return 1 -- ALLOW

This runs in ~10ms on edge infrastructure. Instance A and Instance B hitting this at the exact same millisecond? Redis queues them. One passes, one fails. Budget enforced. Mathematically consistent.

The two-phase protocol

There's one more problem: you don't know the exact cost of an LLM call until it finishes.

The solution is a two-phase commit:

Phase 1 — pre-flight (before the LLM call)
Estimate the cost based on max possible tokens. Reserve that amount atomically. If budget exceeded, return 429 immediately — the LLM never even gets called.

Phase 2 — reconciliation (after the LLM call)
OpenAI returns actual token usage. Reconcile: release the estimate, apply the real cost. If the estimate was too high, refund the difference back to the user's budget.

This means your budget enforcement is tight even under worst-case conditions.

Benchmark: controlled Denial of Wallet attack

I ran a simulated DoW attack against a standard GPT-4o endpoint.

Setup: Recursive script, concurrent requests, 800+ token payloads per request.

	Unprotected	With atomic governance
Duration	47 seconds	Stopped at request 3
Total spend	$847.00	$0.08
Intervention	Manual	Automatic

The governance layer fired a 429 at the third request. The attacker's loop never got traction.

Why this matters after supply chain attacks

Here's the thing about the LiteLLM and Axios breaches: rotating your API key is the right move, but it's reactive. The damage happens in the window between the breach and you waking up.

Budget governance is your last line of defense. Even with a stolen key, the attacker can only drain up to the limit you set. No $1,000 surprise at sunrise.

The implementation

I built this into an open-source SDK called Thskyshield. Two lines to wrap any LLM call:

const { allowed, requestId } = await shield.check({
  externalUserId: userId,
  model: 'gpt-4o',
  estimatedTokens: { input: 500, output: 200 }
})

if (!allowed) return Response.json({ error: 'Budget exceeded' }, { status: 429 })

// ...your LLM call here...

await shield.log({ requestId, externalUserId: userId, model: 'gpt-4o',
  tokens: { input: usage.prompt_tokens, output: usage.completion_tokens }
})

The SDK handles the atomic reservation, the two-phase reconciliation, and the 429 response automatically. Works on Vercel Edge, Cloudflare Workers, or any Node.js environment.

→ SDK: npm install @thsky-21/thskyshield
→ Live attack simulation: thskyshield.com/simulator

What are you actually using to cap LLM spend right now? Are you relying on OpenAI's hard limits, or have you built something custom? Would genuinely like to know what's working.

How I built a real-time LLM "Kill-Switch" for Vercel Edge using Atomic Redis

Akash Melavanki — Tue, 07 Apr 2026 14:39:17 +0000

Last week, the Axios supply chain attack compromised over 100 million weekly downloads. A week before that, it was LiteLLM.

In both cases, the goal was simple: Exfiltrate API keys. As developers, we are taught to rotate our keys immediately. But there’s a massive gap in that advice. If an attacker gets your OpenAI key at 2 AM, they don't wait for you to wake up. They loop your endpoints, drain your credits, and leave you with a $1,000+ bill by sunrise.

This is what OWASP calls LLM10:2025 – Unbounded Consumption (or "Denial of Wallet"). I spent the last two weeks building a way to stop it at the Edge.

The Problem: Why Rate Limiting Fails LLMs
Standard rate-limiting (e.g., 10 requests per minute) is useless for LLMs.

Request 1: "Hi" (10 tokens) — Cost: $0.0001

Request 2: "Summarize this 50-page PDF" (30,000 tokens) — Cost: $0.45

An attacker doesn't need a high volume of requests to ruin you; they just need expensive requests. We need Budget Limiting, not Rate Limiting.

The Technical Challenge: The Stateless Race Condition
I’m building this for Next.js on Vercel Edge.

Vercel Edge functions are stateless. If you try to track a user's spend in a local variable, it vanishes. If you use a standard database, the latency kills your UX.

But the real "final boss" is the Race Condition.

Imagine a user fires 10 concurrent requests.

Instance A checks the budget: "Remaining: $0.05. Proceed."

Instance B checks the budget: "Remaining: $0.05. Proceed."

Both fire $1.00 requests.

Result: You are now -$1.95 in the hole.

The Solution: Atomic Lua Scripts on Redis
To solve this, I moved the logic into an Atomic Lua Script on Upstash Redis. Instead of "Check then Update" (two steps), the logic happens in one single, uninterruptible step inside the database memory.

-- The "Kill-Switch" Logic
local key = KEYS[1] -- user_budget_key
local limit = tonumber(ARGV[1]) -- e.g., 1.00 USD
local cost = tonumber(ARGV[2]) -- estimated cost
local current = tonumber(redis.call('GET', key) or "0")

if current + cost > limit then
  return 0 -- BLOCK
end

redis.call('INCRBYFLOAT', key, cost)
return 1 -- ALLOW

This runs in ~10ms. If Instance A and B hit the script at the exact same millisecond, Redis queues them. One passes, the second fails. No race condition. No $1,000 surprises.

The Benchmark: A Controlled Stress Test
To quantify the risk, I ran a simulated Denial of Wallet (DWL) attack against a standard Next.js API route.

The Setup:

Attacker: A simple recursive script firing concurrent requests with high-token payloads (800+ tokens/request).

Target: A GPT-4o endpoint.

The Result (Unprotected): The script ran for 47 seconds. Total simulated cost reached $847.00 before manual intervention.

The Result (Thskyshield): Using the same script, the governance layer triggered a 429 (Too Many Requests) at the 3rd call. Total spend: $0.08.

Watch the Live Simulation →

The "Two-Phase" Protocol
The hardest part was handling the fact that you don't know the exact cost of an LLM call until it's finished. I settled on a two-phase approach:

Phase 1 (Pre-flight): Check the budget based on the max possible tokens. "Lock" that amount.

Phase 2 (Post-flight): Once the LLM returns, reconcile the actual usage and "Refund" the difference to the user's budget.

Conclusion
Supply chain attacks like the Axios one are the "new normal." We can't stop every key from being stolen, but we can stop a stolen key from being a business-ending event.

I’ve open-sourced the SDK for this under Thskyshield. If you're building with Next.js and want to stop worrying about your OpenAI bill, it's free for founders.

SDK: @thsky-21/thskyshield

Website: thskyshield.com

Would love to hear how others are handling "Denial of Wallet" risks. Are you just relying on OpenAI's hard limits, or are you building your own governance layer?