DEV Community: John Medina

Your LLM Budget Alerts Won't Save You. Here's Why.

John Medina — Mon, 07 Sep 2026 14:06:08 +0000

You get the email at 2 AM. Your cloud provider, with a tone of forced helpfulness, informs you you've burned through 80% of your monthly LLM budget. In the first week.

"No problem," you think. "I have budget alerts for this." But the damage is done. The runaway agent, the one you tasked with summarizing customer support tickets, decided to recursively summarize its own summaries. For 12 hours straight.

The truth is, the simple budget alerts we all set up are a trap. They give us a false sense of security in the age of agentic AI.

The Agentic Spend Problem

A single user click can now trigger a cascade of LLM calls. An agent might read a document, then decide to search the web for context, then summarize its findings, then translate them. That's four API calls, not one. Your old-school "dollars per day" alert is blind to this. It only sees the total. It can't tell you that one specific user's request just cost you $50.

This is the new reality of "agentic spend." It's unpredictable, spiky, and hard to trace back to a single cause.

From Monitoring to Attribution

The only way to control these costs is to move from passive monitoring to active attribution. You need to know, in real-time:

Which user is generating the most cost?
Which agent or feature is the most expensive?
Which model is eating up your budget?

Without this level of granularity, you're flying blind. You can't optimize what you can't see.

This is why I built LLMeter. It's an open-source dashboard that gives you per-user, per-model, and per-day cost attribution. It's not just another alert system. It's a tool for understanding where your money is going, so you can stop the bleeding before it starts.

Fwiw, a simple dashboard showing real-time spend per user has been more effective for us than any alert we've ever set up. The game has changed. Your tools need to change too.

3 Costly Mistakes I Made With the OpenAI API (So You Don't Have To)

John Medina — Mon, 31 Aug 2026 14:06:17 +0000

You get the first bill from OpenAI and it's 10x what you expected. We've all been there.

It's not about the cost of a single call. It's about the thousands of calls you didn't know you were making. Here are three traps I fell into.

1. Forgetting about Temperature and Max Tokens

This is the classic. You're testing in the playground, temperature is at 0.7 for creative outputs, and max_tokens is wide open. Works great. Then you push it to production.

Suddenly, a simple summarization task that should take 200 tokens is regularly using 1500 because the model is getting creative with its output length. Every. Single. Time. For high-volume tasks, this adds up faster than you can blink.

The fix: Set max_tokens to a sane, hard limit for your specific use case. And unless you need creativity, turn temperature down to 0.1 or 0.2. Your wallet will thank you.

2. Not Tracking Costs Per User

When you're starting out, all API calls go through a single key. It's fine. Until you have your first power user.

One user starts running complex reports, another is just doing simple lookups. But your bill just shows one big number. You have no idea who is costing you money or who your most valuable (or expensive) users are. You can't bill them, you can't limit them, you can't do anything. You're flying blind.

The fix: From day one, attribute every single API call to a user ID. Don't just log it somewhere—track it. You need to know which users are driving your costs. It's not optional if you plan to build a business on this.

3. Ignoring the Model Version

gpt-4 is not gpt-4-turbo. The costs are wildly different. It's easy to just use the "latest and greatest" model for everything during development. But moving to production with that mindset is a recipe for a financial shock.

I once had a background job running on the full gpt-4 model for a simple text classification task. It worked, sure. But gpt-3.5-turbo would have been 10x cheaper and 99% as effective. The difference was thousands of dollars over a month for a non-critical feature.

The fix: Audit your model usage. Use the most powerful models only where absolutely necessary. For everything else, find the cheapest model that gets the job done. This requires constant monitoring.

I got so frustrated with flying blind on costs that I built a tool to fix it. It's an open-source dashboard called LLMeter. It hooks into OpenAI (and Anthropic, DeepSeek, etc.) and gives me a real-time view of costs per model, per user, per day. It's how I caught that gpt-4 issue.

You can check it out at https://llmeter.org if you're tired of guessing.

3 Ways to Track Anthropic API Costs (And Why the Console Isn't Enough)

John Medina — Mon, 24 Aug 2026 14:06:40 +0000

PLATAFORMA: Dev.to

The Anthropic console gives you one number: your monthly bill. If you're building a multi-tenant app on top of Claude, that's not nearly enough. You need to know which customers are costing you the most, which features are driving that cost, and how to spot a runaway bill before it happens.

Here are three ways to get more granular cost data, from the simple to the more robust.

1. The Usage API: A Good First Step

Anthropic's Usage API is a solid starting point. You can query it to get cost data broken down by user_id (if you're creating a key per user, which I don't recommend), model, and date.

It's better than nothing, but it's still reactive. You're looking at costs that have already been incurred.

2. The SDK Wrapper: Proactive, But More Work

A better approach is to wrap the Anthropic SDK. This lets you intercept every API call, log the token counts from the response along with your own internal customer_id or feature_id, and then store that data in your own database.

This is a more proactive approach. You can build your own real-time dashboards and set up alerts based on your own business logic. The downside is that you have to build and maintain it yourself.

3. Unified, Proxyless Monitoring: The Best of Both Worlds

The ideal solution is a dedicated, open-source tool that handles this for you. That's why I built LLMeter.

LLMeter uses async, out-of-band monitoring to track your costs without adding any latency to your API calls. It gives you a unified dashboard for all your LLM providers (not just Anthropic), with per-user and per-feature cost attribution out of the box.

You can self-host it, so your data never leaves your infrastructure.

The Anthropic console is fine for a hobby project. But if you're building a business on top of their API, you need to own your cost data. Don't wait for the end-of-month surprise.

The 'AI Trust Tax': The Hidden 30% You're Paying on Every LLM Call

John Medina — Mon, 17 Aug 2026 14:08:11 +0000

PLATAFORMA: Dev.to

You've calculated your LLM costs down to the last token. You know the price of GPT-4o vs. Sonnet 3.5. You've optimized your prompts.

So why is your bill still 30% higher than you expected?

You're paying an invisible "AI Trust Tax."

This isn't about the inference cost. This is the tax you pay to make sure the model doesn't go off the rails. It's the cost of guardrail models, safety checks, evaluation calls, and moderation APIs.

It's the cost of:

Pre-flight checks: Running a prompt through a smaller, faster model (like Haiku) to check for PII or policy violations before sending it to the big, expensive model.
Post-flight analysis: Sending the model's output to a moderation API to check for hate speech or other unwanted content.
Red-teaming and evaluation: Constantly running your prompts and outputs through evaluation pipelines to measure quality and detect regressions.

These are all necessary evils. You can't ship a production AI app without them. But they are also silent budget killers.

The problem is that most monitoring tools only show you the cost of the final generation. They don't show you the cost of the entire chain of calls that it took to produce that generation.

This is where a tool that monitors all your API calls becomes essential. When I was building LLMeter, I made sure it could track every single call to every provider. That's how you uncover the hidden "trust tax."

Once you can see the full cost, you can start to optimize it. Maybe you can batch your moderation calls. Maybe you can run your pre-flight checks on a cheaper model.

But you can't optimize what you can't see. Stop paying the invisible tax. Get a monitoring tool that shows you the whole picture.

The Collapsing Stack: How Vendor Lock-in is Coming for Your AI-Powered Applications

John Medina — Mon, 10 Aug 2026 14:02:24 +0000

You chose Bun for its speed, but you might be getting more than you bargained for. What happens when your runtime, your CLI, and your LLM are all owned by the same company?

This is the new reality of the AI-powered stack. The lines are blurring between the tools we use to build our applications and the AI services we consume. And this "collapsing stack" is creating a new and dangerous form of vendor lock-in.

When a single company controls your runtime, they can start to influence your choice of AI provider. They can offer "seamless integrations" that are anything but seamless to disentangle. They can bundle services in a way that makes it impossible to switch one component without switching them all.

Think about it:

What if your runtime starts making metered calls to the vendor's AI service by default?
What if your CLI tool for deploying your application also "helpfully" provisions the vendor's AI services?
What if the terms of service for your runtime are updated to include clauses about your usage of their AI services?

This isn't just a hypothetical scenario. We're already seeing the first signs of this with Bun's rewrite in Rust by the same company that's behind the Claude models.

The only way to protect yourself is to maintain control over your stack. You need to use tools that are open, interoperable, and vendor-agnostic.

This is especially true for monitoring. You can't rely on your vendor's dashboard to give you an unbiased view of your costs. You need an independent tool that can monitor your usage across all your providers.

That's why we built LLMeter (https://llmeter.org). It's an open-source, self-hosted tool that gives you a single pane of glass for all your LLM costs. It's vendor-agnostic by design.

Don't let the collapsing stack lock you in. Choose your tools wisely, and maintain control of your own destiny.

Monitor LLM Costs with Prometheus & Grafana (Without a Proxy)

John Medina — Mon, 03 Aug 2026 14:07:58 +0000

PLATAFORMA: Dev.to

If you're running any observability stack, you've probably got Prometheus and Grafana at the core. You've got your CPU usage, your memory, your network I/O... but what about your single biggest, most volatile line item? Your LLM bill.

Getting that data into your existing dashboards is a pain. Most solutions want you to route all your traffic through their proxy, adding a point of failure and latency you don't need.

It doesn't have to be that complicated.

I'm a big believer in async, out-of-band monitoring. Your app should fire and forget logging events, not wait for a monitoring service to respond. This is why when I built LLMeter, I made sure it could plug into the tools you already use.

Here's how you can pipe your LLM costs into Prometheus without a proxy.

LLMeter exposes a /metrics endpoint that Prometheus can scrape, just like it does for your other services. All you need to do is add a new job to your prometheus.yml:

scrape_configs:
  - job_name: 'llmeter'
    scrape_interval: 15s
    static_configs:
      - targets: ['your-llmeter-instance.local:3000']

That's it. Now you can build Grafana dashboards that show your LLM costs right next to your other metrics. You can see how a spike in traffic to a certain endpoint correlates with a spike in your Anthropic bill. You can finally see the real cost of that new AI feature you just shipped.

This approach gives you:

Zero-latency monitoring: Your API calls go straight to the provider, not through a third-party proxy.
Real-time visibility: See your costs as they happen, not at the end of the month.
Unified dashboards: No more context-switching between your observability stack and your LLM provider's billing page.

Stop flying blind. Your LLM bill is part of your infrastructure cost. It's time to start treating it that way.

The Caching Illusion: Why Semantic Cache Won't Save Your LLM Budget

John Medina — Mon, 27 Jul 2026 14:02:57 +0000

Everyone tells you to just drop a semantic cache in front of OpenAI or Anthropic to cut costs. I tried it. tbh, it's mostly an illusion.

Here is the reality of AI agents: they don't ask the same questions twice. They inject dynamic context, timestamps, user-specific IDs, and multi-step reasoning traces into every single prompt.

I looked at 50,000 requests from a basic RAG setup. The cache hit rate? 4.2%.

A 4% cache hit rate doesn't save a $5k/mo bill. It just adds Redis infrastructure costs and a layer of latency you didn't need. The real problem isn't caching. The real problem is you don't know which user or which agent loop is burning the other 96% of the budget.

If you have a multi-tenant app, your biggest risk isn't missing a cache hit. It's one power user triggering a runaway agent loop that eats your entire API quota while everyone else gets rate-limited.

You need per-user cost attribution, not just a cache.

I built LLMeter (https://llmeter.org?utm_source=devto&utm_medium=article&utm_campaign=devto-caching-illusion-llmeter) exactly for this. It's an open-source (AGPL-3.0) dashboard that tracks costs per model, per user, per day. It supports OpenAI, Anthropic, DeepSeek, and OpenRouter.

Stop trying to cache dynamic agent loops. Start tracking who is actually spending your money.

Don't Trust OpenAI's Billing Alerts Alone

John Medina — Mon, 20 Jul 2026 14:04:57 +0000

PLATAFORMA: Dev.to

You set a hard budget limit in your OpenAI dashboard. You think you're safe. Then at the end of the month, you get a bill for 2x what you expected.

Happens all the time. Saw a few devs complaining about it again this week.

The "hard limit" isn't hard. It's a notification threshold that can be minutes or even hours delayed. By the time it triggers and you get the email, a runaway script or a high-traffic feature could have already burned through a lot more cash.

The issue is that usage data isn't processed in real-time. It's batched. So the dashboard you're looking at is always slightly in the past.

It's not a bug, it's how their billing infra is designed. But it's a problem if you're bootstrapping or giving per-user access to an LLM feature. One user having a field day can blow your whole month's budget.

The only real fix is to monitor costs on your end, request-by-request.

I built a simple open-source tool, LLMeter (llmeter.org), to deal with this. It tracks costs as they happen and has its own alerting system that's much closer to real-time. It also helps attribute costs per-user or per-feature, which is something OpenAI's dash doesn't do.

fwiw, whatever you use, don't rely on the official billing alerts alone for cost protection. You'll get burned eventually.

Everyone's sleeping on the easiest 50% discount for LLM APIs

John Medina — Mon, 13 Jul 2026 14:01:44 +0000

You see all these complex guides on prompt engineering, model routing, caching... all promising to shave a few percentage points off your LLM bill. They're not wrong, but they're missing the forest for the trees.

OpenAI and Anthropic are literally offering a 50% discount on their most powerful models. And it seems like nobody is talking about it.

It's called the Batch API.

It's not new, it's not sexy, and it doesn't involve any "AI magic". It's just a different endpoint. Instead of sending requests one by one, you bundle them up and send them in a single file. 24 hours later, you get a file back with the results.

The catch? It's async. Heavily async. You can't use this for a real-time chatbot. The 24-hour SLA is real.

But for any non-interactive workload? It's a no-brainer.

Summarizing articles for a newsletter.
Generating embeddings for a RAG pipeline.
Classifying user-generated content.
Running batch evaluations on a new prompt.

All these tasks are perfect for batching. You're probably running them on the real-time endpoint right now and paying double for the privilege.

The implementation is a bit different. You upload a JSONL file with your requests, start the batch job, and then poll for completion. It's a bit more work than a simple POST request, but it's not rocket science.

We wrote a more detailed breakdown of the pricing and gotchas here: https://simplifai.tools/blog/llm-batch-api-pricing

Honestly, before you spend a week implementing a caching layer, just check if your workload can be batched. It's the most boring, most effective way to cut your LLM costs. fwiw.

Your LLM Bill is a Ticking Time Bomb. Here's How to Defuse It.

John Medina — Mon, 06 Jul 2026 14:07:17 +0000

Everyone's rushing to bolt LLM features onto their products. A quick look at Upwork shows thousands of jobs for "LLM integration" and "AI automation". It's a gold rush. But very few are talking about the second-order effects. Specifically, the cost.

It's not about the per-token price on a vendor's website. That's just the entry fee. The real cost—the one that blows up budgets—is far more insidious. We're all dangerously skipping the rigor on cost analysis, and it's going to bite us.

The Real TCO of an LLM Feature

I saw a team blow through $20k in a single weekend. The culprit? A runaway agent caught in a retry loop. Their generic cloud budget alerts didn't even notice until Monday morning.

This is the new reality. Traditional tools like AWS Cost Explorer can't see inside an API call. They see you made a request to OpenAI; they have no idea if it was for a $0.01 summary task or a $100 recursive agent run. This is the "FinOps for AI" gap.

The Total Cost of Ownership (TCO) for your shiny AI feature isn't just (prompt_tokens * input_cost) + (completion_tokens * output_cost). It's:

Engineering Time: Hours spent debugging prompts, optimizing models, and firefighting when costs spike.
Scaling Risk: A feature that costs $10/day with 100 users will bankrupt you at 10,000 users if it scales linearly.
Cost of Errors: Bad or inconsistent outputs burn API credits and, more importantly, user trust.

How to Actually Do Cost Control

If you're building with LLMs and don't have a dashboard showing you cost-per-user or cost-per-feature, you're flying blind.

The first step is instrumentation. Every single API call needs to be tagged with metadata. Who is the user? What feature are they using? What's the conversation ID?

// Don't just call the API
// const response = await openai.chat.completions.create({...});

// Add your own context
const response = await trackCost(
  {
    userId: "user_123",
    featureId: "summarize_document",
    tenantId: "acme_corp"
  },
  () => openai.chat.completions.create({...})
);

Without this, you can't attribute costs. You can't tell which customers are profitable, which features are too expensive, or when a single user starts abusing the system.

You need a dedicated tool for this. I've been building LLMeter (llmeter.org) to solve this problem for my own projects. It's an open-source dashboard that monitors costs across providers like OpenAI, Anthropic, and Mistral. It gives you that crucial per-user and per-feature breakdown. Since it's AGPL, you can just self-host it and avoid vendor lock-in.

Whatever you use, the principle is the same: stop guessing. Start measuring. Your budget depends on it.

How to Track Per-User OpenAI Costs in Your Next.js App

John Medina — Tue, 30 Jun 2026 16:01:35 +0000

So you've integrated the OpenAI API into your Next.js app. Your users love it. But then you get the bill, and you have no idea who's costing you what. A single "power user" could be driving 90% of your costs, and you'd be flying blind.

Let's fix that. Tracking per-user costs isn't just a nice-to-have, it's essential for any application that exposes an LLM to end-users.

The Problem: The Un-instrumented API Call

Your code might look something like this. You have an API route in Next.js that calls OpenAI.

// pages/api/generate.ts
import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export default async function handler(req, res) {
  const completion = await openai.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'gpt-4o',
  });

  res.status(200).json({ result: completion.choices[0].message.content });
}

This works, but it's a black box. You know a call was made, but you don't know who made it.

The Solution: Attribute Every Call

To fix this, you need to attribute every single API call to a user. This means passing a unique user identifier with every request.

First, protect your API route to make sure you have a user session. Then, wrap your OpenAI calls in a helper function that logs the cost.

// lib/openai.ts
import OpenAI from 'openai';
import { getCostForModel, saveUsageToDatabase } from './billing'; // Fictional helpers

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function createChatCompletion(userId: string, options: any) {
  const completion = await openai.chat.completions.create(options);

  const promptTokens = completion.usage?.prompt_tokens ?? 0;
  const completionTokens = completion.usage?.completion_tokens ?? 0;

  // Calculate cost based on model (you have to maintain these prices)
  const cost = getCostForModel(options.model, promptTokens, completionTokens);

  // Save to your database
  await saveUsageToDatabase({
    userId,
    model: options.model,
    promptTokens,
    completionTokens,
    cost,
  });

  return completion;
}

You'd need a database table to store this, maybe in Supabase:

CREATE TABLE llm_usage (
  id bigserial primary key,
  created_at timestamp with time zone default now(),
  user_id text not null,
  model text not null,
  prompt_tokens integer not null,
  completion_tokens integer not null,
  cost numeric(10, 6) not null
);

The Shortcut: Don't Build It Yourself

As you can see, this is a lot of boilerplate. You need to maintain pricing for different models, build a robust logging system, and create a dashboard to view the data.

This is a problem I ran into, so I built LLMeter (llmeter.org). It's an open-source, AGPL-licensed dashboard you can self-host. It gives you a simple SDK to wrap your LLM calls and automatically handles the cost calculation, logging, and dashboarding for multiple providers (OpenAI, Anthropic, Mistral, etc.).

It turns the code above into something much simpler and gives you a production-ready dashboard out of the box.

Whether you build it yourself or use a tool, the takeaway is the same: if you're not tracking LLM costs on a per-user basis, you're driving with your eyes closed. Don't wait for the surprise bill to learn that lesson.

How to Stop One Customer From Blowing Up Your Entire LLM Budget

John Medina — Mon, 22 Jun 2026 14:06:46 +0000

So your SaaS is finally getting some traction. Congrats. Then you check your OpenAI bill and realize one power user just cost you $500 overnight running reports. Now what?

This isn't a rare problem. If you're building any multi-tenant AI app, your biggest financial risk is a single user with a runaway script or an unpredictable use case. Standard API rate limits are too crude—they punish all users and can kill legitimate usage. Manually watching your dashboard doesn't scale past your first few customers.

You need a way to track costs per user and enforce budgets automatically.

Most people start by trying to build this logic in-house. You can add a user_id to your API calls and log the token counts to your own database. Then you run a cron job to aggregate costs and check against a budget column in your users table.

It works, until it doesn't.

Provider differences: The way you calculate costs for OpenAI is different from Anthropic, and different again for OpenRouter. Your logic gets complex fast.
Timing issues: Cron jobs aren't real-time. By the time your job runs, a user could have already gone 2x over their budget.
Maintenance: It's another piece of infrastructure you have to build, test, and maintain. That's time you're not spending on your core product.

Tbh, I got tired of rebuilding this for every project.

So I built a simple, open-source tool to handle it: LLMeter.

It's a dashboard that sits on top of your existing LLM providers (OpenAI, Anthropic, etc.). You tell it which user made which API call, and it handles the rest.

Tracks costs per-user, in real-time.
Lets you set a budget for each user.
Sends you a webhook or email when a user hits 50%, 90%, or 100% of their budget.

You can then use that webhook to programmatically disable that user's access, switch them to a slower model, or just notify them. No more surprise bills.

It's not a proxy, so it doesn't add latency. It's just a simple, open-source dashboard you can self-host or use the managed version. Fwiw, it solved my own problem. Maybe it'll solve yours too.