DEV Community: John Medina

The 'AI Trust Tax': The Hidden 30% You're Paying on Every LLM Call

John Medina — Mon, 17 Aug 2026 14:08:11 +0000

PLATAFORMA: Dev.to

You've calculated your LLM costs down to the last token. You know the price of GPT-4o vs. Sonnet 3.5. You've optimized your prompts.

So why is your bill still 30% higher than you expected?

You're paying an invisible "AI Trust Tax."

This isn't about the inference cost. This is the tax you pay to make sure the model doesn't go off the rails. It's the cost of guardrail models, safety checks, evaluation calls, and moderation APIs.

It's the cost of:

Pre-flight checks: Running a prompt through a smaller, faster model (like Haiku) to check for PII or policy violations before sending it to the big, expensive model.
Post-flight analysis: Sending the model's output to a moderation API to check for hate speech or other unwanted content.
Red-teaming and evaluation: Constantly running your prompts and outputs through evaluation pipelines to measure quality and detect regressions.

These are all necessary evils. You can't ship a production AI app without them. But they are also silent budget killers.

The problem is that most monitoring tools only show you the cost of the final generation. They don't show you the cost of the entire chain of calls that it took to produce that generation.

This is where a tool that monitors all your API calls becomes essential. When I was building LLMeter, I made sure it could track every single call to every provider. That's how you uncover the hidden "trust tax."

Once you can see the full cost, you can start to optimize it. Maybe you can batch your moderation calls. Maybe you can run your pre-flight checks on a cheaper model.

But you can't optimize what you can't see. Stop paying the invisible tax. Get a monitoring tool that shows you the whole picture.

The Collapsing Stack: How Vendor Lock-in is Coming for Your AI-Powered Applications

John Medina — Mon, 10 Aug 2026 14:02:24 +0000

You chose Bun for its speed, but you might be getting more than you bargained for. What happens when your runtime, your CLI, and your LLM are all owned by the same company?

This is the new reality of the AI-powered stack. The lines are blurring between the tools we use to build our applications and the AI services we consume. And this "collapsing stack" is creating a new and dangerous form of vendor lock-in.

When a single company controls your runtime, they can start to influence your choice of AI provider. They can offer "seamless integrations" that are anything but seamless to disentangle. They can bundle services in a way that makes it impossible to switch one component without switching them all.

Think about it:

What if your runtime starts making metered calls to the vendor's AI service by default?
What if your CLI tool for deploying your application also "helpfully" provisions the vendor's AI services?
What if the terms of service for your runtime are updated to include clauses about your usage of their AI services?

This isn't just a hypothetical scenario. We're already seeing the first signs of this with Bun's rewrite in Rust by the same company that's behind the Claude models.

The only way to protect yourself is to maintain control over your stack. You need to use tools that are open, interoperable, and vendor-agnostic.

This is especially true for monitoring. You can't rely on your vendor's dashboard to give you an unbiased view of your costs. You need an independent tool that can monitor your usage across all your providers.

That's why we built LLMeter (https://llmeter.org). It's an open-source, self-hosted tool that gives you a single pane of glass for all your LLM costs. It's vendor-agnostic by design.

Don't let the collapsing stack lock you in. Choose your tools wisely, and maintain control of your own destiny.

Monitor LLM Costs with Prometheus & Grafana (Without a Proxy)

John Medina — Mon, 03 Aug 2026 14:07:58 +0000

PLATAFORMA: Dev.to

If you're running any observability stack, you've probably got Prometheus and Grafana at the core. You've got your CPU usage, your memory, your network I/O... but what about your single biggest, most volatile line item? Your LLM bill.

Getting that data into your existing dashboards is a pain. Most solutions want you to route all your traffic through their proxy, adding a point of failure and latency you don't need.

It doesn't have to be that complicated.

I'm a big believer in async, out-of-band monitoring. Your app should fire and forget logging events, not wait for a monitoring service to respond. This is why when I built LLMeter, I made sure it could plug into the tools you already use.

Here's how you can pipe your LLM costs into Prometheus without a proxy.

LLMeter exposes a /metrics endpoint that Prometheus can scrape, just like it does for your other services. All you need to do is add a new job to your prometheus.yml:

scrape_configs:
  - job_name: 'llmeter'
    scrape_interval: 15s
    static_configs:
      - targets: ['your-llmeter-instance.local:3000']

That's it. Now you can build Grafana dashboards that show your LLM costs right next to your other metrics. You can see how a spike in traffic to a certain endpoint correlates with a spike in your Anthropic bill. You can finally see the real cost of that new AI feature you just shipped.

This approach gives you:

Zero-latency monitoring: Your API calls go straight to the provider, not through a third-party proxy.
Real-time visibility: See your costs as they happen, not at the end of the month.
Unified dashboards: No more context-switching between your observability stack and your LLM provider's billing page.

Stop flying blind. Your LLM bill is part of your infrastructure cost. It's time to start treating it that way.

The Caching Illusion: Why Semantic Cache Won't Save Your LLM Budget

John Medina — Mon, 27 Jul 2026 14:02:57 +0000

Everyone tells you to just drop a semantic cache in front of OpenAI or Anthropic to cut costs. I tried it. tbh, it's mostly an illusion.

Here is the reality of AI agents: they don't ask the same questions twice. They inject dynamic context, timestamps, user-specific IDs, and multi-step reasoning traces into every single prompt.

I looked at 50,000 requests from a basic RAG setup. The cache hit rate? 4.2%.

A 4% cache hit rate doesn't save a $5k/mo bill. It just adds Redis infrastructure costs and a layer of latency you didn't need. The real problem isn't caching. The real problem is you don't know which user or which agent loop is burning the other 96% of the budget.

If you have a multi-tenant app, your biggest risk isn't missing a cache hit. It's one power user triggering a runaway agent loop that eats your entire API quota while everyone else gets rate-limited.

You need per-user cost attribution, not just a cache.

I built LLMeter (https://llmeter.org?utm_source=devto&utm_medium=article&utm_campaign=devto-caching-illusion-llmeter) exactly for this. It's an open-source (AGPL-3.0) dashboard that tracks costs per model, per user, per day. It supports OpenAI, Anthropic, DeepSeek, and OpenRouter.

Stop trying to cache dynamic agent loops. Start tracking who is actually spending your money.

Don't Trust OpenAI's Billing Alerts Alone

John Medina — Mon, 20 Jul 2026 14:04:57 +0000

PLATAFORMA: Dev.to

You set a hard budget limit in your OpenAI dashboard. You think you're safe. Then at the end of the month, you get a bill for 2x what you expected.

Happens all the time. Saw a few devs complaining about it again this week.

The "hard limit" isn't hard. It's a notification threshold that can be minutes or even hours delayed. By the time it triggers and you get the email, a runaway script or a high-traffic feature could have already burned through a lot more cash.

The issue is that usage data isn't processed in real-time. It's batched. So the dashboard you're looking at is always slightly in the past.

It's not a bug, it's how their billing infra is designed. But it's a problem if you're bootstrapping or giving per-user access to an LLM feature. One user having a field day can blow your whole month's budget.

The only real fix is to monitor costs on your end, request-by-request.

I built a simple open-source tool, LLMeter (llmeter.org), to deal with this. It tracks costs as they happen and has its own alerting system that's much closer to real-time. It also helps attribute costs per-user or per-feature, which is something OpenAI's dash doesn't do.

fwiw, whatever you use, don't rely on the official billing alerts alone for cost protection. You'll get burned eventually.

Everyone's sleeping on the easiest 50% discount for LLM APIs

John Medina — Mon, 13 Jul 2026 14:01:44 +0000

You see all these complex guides on prompt engineering, model routing, caching... all promising to shave a few percentage points off your LLM bill. They're not wrong, but they're missing the forest for the trees.

OpenAI and Anthropic are literally offering a 50% discount on their most powerful models. And it seems like nobody is talking about it.

It's called the Batch API.

It's not new, it's not sexy, and it doesn't involve any "AI magic". It's just a different endpoint. Instead of sending requests one by one, you bundle them up and send them in a single file. 24 hours later, you get a file back with the results.

The catch? It's async. Heavily async. You can't use this for a real-time chatbot. The 24-hour SLA is real.

But for any non-interactive workload? It's a no-brainer.

Summarizing articles for a newsletter.
Generating embeddings for a RAG pipeline.
Classifying user-generated content.
Running batch evaluations on a new prompt.

All these tasks are perfect for batching. You're probably running them on the real-time endpoint right now and paying double for the privilege.

The implementation is a bit different. You upload a JSONL file with your requests, start the batch job, and then poll for completion. It's a bit more work than a simple POST request, but it's not rocket science.

We wrote a more detailed breakdown of the pricing and gotchas here: https://simplifai.tools/blog/llm-batch-api-pricing

Honestly, before you spend a week implementing a caching layer, just check if your workload can be batched. It's the most boring, most effective way to cut your LLM costs. fwiw.

Your LLM Bill is a Ticking Time Bomb. Here's How to Defuse It.

John Medina — Mon, 06 Jul 2026 14:07:17 +0000

Everyone's rushing to bolt LLM features onto their products. A quick look at Upwork shows thousands of jobs for "LLM integration" and "AI automation". It's a gold rush. But very few are talking about the second-order effects. Specifically, the cost.

It's not about the per-token price on a vendor's website. That's just the entry fee. The real cost—the one that blows up budgets—is far more insidious. We're all dangerously skipping the rigor on cost analysis, and it's going to bite us.

The Real TCO of an LLM Feature

I saw a team blow through $20k in a single weekend. The culprit? A runaway agent caught in a retry loop. Their generic cloud budget alerts didn't even notice until Monday morning.

This is the new reality. Traditional tools like AWS Cost Explorer can't see inside an API call. They see you made a request to OpenAI; they have no idea if it was for a $0.01 summary task or a $100 recursive agent run. This is the "FinOps for AI" gap.

The Total Cost of Ownership (TCO) for your shiny AI feature isn't just (prompt_tokens * input_cost) + (completion_tokens * output_cost). It's:

Engineering Time: Hours spent debugging prompts, optimizing models, and firefighting when costs spike.
Scaling Risk: A feature that costs $10/day with 100 users will bankrupt you at 10,000 users if it scales linearly.
Cost of Errors: Bad or inconsistent outputs burn API credits and, more importantly, user trust.

How to Actually Do Cost Control

If you're building with LLMs and don't have a dashboard showing you cost-per-user or cost-per-feature, you're flying blind.

The first step is instrumentation. Every single API call needs to be tagged with metadata. Who is the user? What feature are they using? What's the conversation ID?

// Don't just call the API
// const response = await openai.chat.completions.create({...});

// Add your own context
const response = await trackCost(
  {
    userId: "user_123",
    featureId: "summarize_document",
    tenantId: "acme_corp"
  },
  () => openai.chat.completions.create({...})
);

Without this, you can't attribute costs. You can't tell which customers are profitable, which features are too expensive, or when a single user starts abusing the system.

You need a dedicated tool for this. I've been building LLMeter (llmeter.org) to solve this problem for my own projects. It's an open-source dashboard that monitors costs across providers like OpenAI, Anthropic, and Mistral. It gives you that crucial per-user and per-feature breakdown. Since it's AGPL, you can just self-host it and avoid vendor lock-in.

Whatever you use, the principle is the same: stop guessing. Start measuring. Your budget depends on it.

How to Track Per-User OpenAI Costs in Your Next.js App

John Medina — Tue, 30 Jun 2026 16:01:35 +0000

So you've integrated the OpenAI API into your Next.js app. Your users love it. But then you get the bill, and you have no idea who's costing you what. A single "power user" could be driving 90% of your costs, and you'd be flying blind.

Let's fix that. Tracking per-user costs isn't just a nice-to-have, it's essential for any application that exposes an LLM to end-users.

The Problem: The Un-instrumented API Call

Your code might look something like this. You have an API route in Next.js that calls OpenAI.

// pages/api/generate.ts
import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export default async function handler(req, res) {
  const completion = await openai.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'gpt-4o',
  });

  res.status(200).json({ result: completion.choices[0].message.content });
}

This works, but it's a black box. You know a call was made, but you don't know who made it.

The Solution: Attribute Every Call

To fix this, you need to attribute every single API call to a user. This means passing a unique user identifier with every request.

First, protect your API route to make sure you have a user session. Then, wrap your OpenAI calls in a helper function that logs the cost.

// lib/openai.ts
import OpenAI from 'openai';
import { getCostForModel, saveUsageToDatabase } from './billing'; // Fictional helpers

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function createChatCompletion(userId: string, options: any) {
  const completion = await openai.chat.completions.create(options);

  const promptTokens = completion.usage?.prompt_tokens ?? 0;
  const completionTokens = completion.usage?.completion_tokens ?? 0;

  // Calculate cost based on model (you have to maintain these prices)
  const cost = getCostForModel(options.model, promptTokens, completionTokens);

  // Save to your database
  await saveUsageToDatabase({
    userId,
    model: options.model,
    promptTokens,
    completionTokens,
    cost,
  });

  return completion;
}

You'd need a database table to store this, maybe in Supabase:

CREATE TABLE llm_usage (
  id bigserial primary key,
  created_at timestamp with time zone default now(),
  user_id text not null,
  model text not null,
  prompt_tokens integer not null,
  completion_tokens integer not null,
  cost numeric(10, 6) not null
);

The Shortcut: Don't Build It Yourself

As you can see, this is a lot of boilerplate. You need to maintain pricing for different models, build a robust logging system, and create a dashboard to view the data.

This is a problem I ran into, so I built LLMeter (llmeter.org). It's an open-source, AGPL-licensed dashboard you can self-host. It gives you a simple SDK to wrap your LLM calls and automatically handles the cost calculation, logging, and dashboarding for multiple providers (OpenAI, Anthropic, Mistral, etc.).

It turns the code above into something much simpler and gives you a production-ready dashboard out of the box.

Whether you build it yourself or use a tool, the takeaway is the same: if you're not tracking LLM costs on a per-user basis, you're driving with your eyes closed. Don't wait for the surprise bill to learn that lesson.

How to Stop One Customer From Blowing Up Your Entire LLM Budget

John Medina — Mon, 22 Jun 2026 14:06:46 +0000

So your SaaS is finally getting some traction. Congrats. Then you check your OpenAI bill and realize one power user just cost you $500 overnight running reports. Now what?

This isn't a rare problem. If you're building any multi-tenant AI app, your biggest financial risk is a single user with a runaway script or an unpredictable use case. Standard API rate limits are too crude—they punish all users and can kill legitimate usage. Manually watching your dashboard doesn't scale past your first few customers.

You need a way to track costs per user and enforce budgets automatically.

Most people start by trying to build this logic in-house. You can add a user_id to your API calls and log the token counts to your own database. Then you run a cron job to aggregate costs and check against a budget column in your users table.

It works, until it doesn't.

Provider differences: The way you calculate costs for OpenAI is different from Anthropic, and different again for OpenRouter. Your logic gets complex fast.
Timing issues: Cron jobs aren't real-time. By the time your job runs, a user could have already gone 2x over their budget.
Maintenance: It's another piece of infrastructure you have to build, test, and maintain. That's time you're not spending on your core product.

Tbh, I got tired of rebuilding this for every project.

So I built a simple, open-source tool to handle it: LLMeter.

It's a dashboard that sits on top of your existing LLM providers (OpenAI, Anthropic, etc.). You tell it which user made which API call, and it handles the rest.

Tracks costs per-user, in real-time.
Lets you set a budget for each user.
Sends you a webhook or email when a user hits 50%, 90%, or 100% of their budget.

You can then use that webhook to programmatically disable that user's access, switch them to a slower model, or just notify them. No more surprise bills.

It's not a proxy, so it doesn't add latency. It's just a simple, open-source dashboard you can self-host or use the managed version. Fwiw, it solved my own problem. Maybe it'll solve yours too.

Over-editing is a token tax: GPT-5.4 ships 6.5x more diff per fix than Claude Opus 4.6, and your bill notices

John Medina — Mon, 15 Jun 2026 20:13:04 +0000

A model is over-editing if its output is functionally correct but structurally diverges from the original code more than the minimal fix requires. Left unconstrained, the extended reasoning gives models more room to 'improve' code that doesn't need improving.

GPT-5.4 averages 0.395 normalized Levenshtein distance per edit. Claude Opus 4.6 averages 0.060. That is 6.5x more output tokens for the same class of fix, averaged across the benchmark. Pass@1 correctness is similar (0.723–0.912 across models), so the over-editing is paid waste, not paid capability.

What does 6.5x look like on a bill? A 50-engineer org doing 800 agent edits per engineer per month = 40k edits/mo. At average 500 output tokens per minimal fix × $15/M Opus 4.7 output = $300/mo. At 3,250 output tokens per over-edited fix = $1,950/mo. Delta is $1,650/mo per 40k edits, pure output-token waste with no correctness upside. Scale to your actual traffic.

Why 'just use a smaller model' isn't the answer: reasoning models got worse (not better) at minimal editing when given more reasoning budget. So you can't fix over-editing by paying more; you fix it by measuring the ratio and routing around it.

The metric CFOs actually need is over-edit ratio per agent: over_edit_ratio = output_tokens / minimum_required_tokens_to_achieve_green_tests. Infrastructure to compute this: log full diff of every agent edit, run patch-min on the diff offline, diff size ratio = your over-edit score.

Instrument over-edit ratio this quarter, treat it as a first-class SLO per agent (budget for <0.2 average), and route high-stakes "minimal" tasks to models whose published over-edit score is <0.1.

Attribution is the prerequisite for every other cost signal you'll want this year. LLMeter ships per-customer + per-agent attribution today. Over-edit ratio is the first quality-flavored metric where LLMeter's attribution layer is the right home.

Stop Paying for Failed AI Agent Retries

John Medina — Fri, 05 Jun 2026 14:08:15 +0000

When your AI agent fails a step and retries, you are paying for the exact same context window over and over again.

Most devs just stick a try-catch block around their LLM calls and call it a day. But tbh when an agent loops 5 times because of a hallucinated JSON schema, your cost per action just 5x'd. And standard dashboards? They just show a massive spike in "API Usage" without telling you it was a single runaway process.

I built LLMeter specifically to catch this. It tracks costs per-customer and flags anomalous retry loops in real-time. If you're running agents in production, you need to monitor this or your margins will disappear before you notice.

You can check it out at https://llmeter.org?utm_source=devto&utm_medium=article&utm_campaign=devto-stop-paying-for-failed-retries

Over-editing is a token tax: GPT-5.4 ships 6.5x more diff per fix than Claude Opus 4.6, and your bill notices

John Medina — Wed, 03 Jun 2026 14:08:46 +0000

Instrument over-edit ratio this quarter, treat it as a first-class SLO per agent (budget for <0.2 average), and route high-stakes "minimal" tasks to models whose published over-edit score is <0.1.