The hidden cost problem in AI agents

#ai #webdev #openai #claude

Why one power user cost 3x more than the plan price — and why rate limiting can't stop it.

An AI research assistant. Flat $49/month. Solid margins on paper — average user cost ~$8 in OpenAI credits.

One user — call him Charlie — cost $147 alone. On a $49 plan.

Charlie wasn't malicious. He was engaged. Every article he read went through the assistant. Every task chained ten LLM calls. Overnight deep-research prompts. The founder found out on July 3rd. Damage done.

Every AI product has Charlies

Typical usage distribution:

Median user: ~$5-10/mo in LLM spend
95th percentile: ~$50-100
99th percentile: $200+

Top 1-5% don't just eat margin. They erase it. And unlike traditional SaaS, AI cost is variable per call — 20 GPT-4o calls with long context can cost $5 in one session.

Flat pricing + unmetered AI = your worst users decide your profitability.

Rate limiting was built for a different problem — traffic protection. It fails at AI cost control because:

Cost is variable per request. GPT-4o can cost $0.001 or $1.50 depending on context.
Per-key quotas can't see your users. OpenAI's dashboard tells you the org spent $5K. Not which user.
Observability is backward-looking. By the time it shows up, you've already been billed.
Models cost 30x apart. GPT-4o vs GPT-4o-mini. Request counts don't differentiate.
One prompt = many calls. Agents chain 5-10 LLM calls per user prompt.

The unit that matters isn't requests per second. It's cost per user.

What actually works

Two things, in order:

Cost metering — capture tokens, model, cost per user, per session, per model. Every call.

Cost gating — check spending limits before the call leaves your server. Warn at 80%. Block at 100% — no tokens consumed, no cost incurred.

Observability tools tell you what happened. Cost gating stops what's about to happen.

Paygent

Paygent is a tool that drops into any app calling OpenAI or Anthropic.
Three lines:

Every LLM call is now attributed to a user, checked against their plan, and blocked if they're over — before it reaches OpenAI. Soft gate at 80% fires a callback (warn, upgrade prompt, model fallback). Hard gate at 100% raises PaygentLimitExceeded.

Works with OpenAI, Anthropic, LangChain/LangGraph (drop-in callback), CrewAI (step callback).

What this looks like with Charlie

July 15th: Charlie crosses $30 on his $49 plan. Soft gate fires. Founder shows him a banner: "Consider upgrading."
July 22nd: Charlie hits $49. Hard gate fires. His next call is blocked. "You've reached your limit."
July invoice: normal amount. No surprise.

Charlie upgrades to $199 or waits until August. Either way, no subsidized power user.