DEV Community: CJ Cummings

How to Control Overhead Cost for AI Products

CJ Cummings — Tue, 11 Aug 2026 13:00:00 +0000

Every LLM call in your AI product — a chat message, a doc summary, an MCP tool call — has a non-deterministic cost attached to it. Let a customer upload 1,000 documents instead of 3, and you're getting an invoice from Big LLM you didn't budget for.

The short version of how you get that under control: measure margin per account, per feature, and per vendor — not just cost. Set enforcement limits that guarantee your worst case. Then price so revenue moves with usage instead of trailing behind it.

Cost-to-deliver is the metric most teams reach for first, and it's a good start. But it's incomplete, because it's missing revenue. Controlling cost alone only ever caps your downside. The number that actually gives you control is margin-to-deliver.

Margin-to-deliver is the share of what you charge for a unit of delivered value that you keep after the vendor cost of producing it — (charged − overhead) / charged — measured per account, per feature, or per vendor.

Optimize margin-to-deliver and you've got two levers to pull: enforcement and pricing. Here's how we think about both, and the order we'd tackle them in.

Disclosure: I build Limitr, which does this. I use it as the worked example below because it's what I know, but the math and the instrumentation approach apply regardless of what you build on.

As soon as you have usage overhead, usage-based complexity exists in your product, regardless of pricing model and what you present to your customers.

This post fills the instrumentation gap between token cost-to-deliver and packaging, which should be simplified into customer language and presentation. Packaging itself is not covered here.

Usage Observability

An effective control strategy starts with observation. That's why we built an observe mode directly into our usage limits — you can't control what you can't see.

Vendor dashboards and API key segmentation might get you through the early days, but that approach falls apart the moment you need per-account, per-vendor, per-feature cost and margin analysis. If you don't believe that yet, check back in a few months once your product's grown up a bit.

The number you're after is margin-to-deliver. Here's an example:

Single account, per-feature usage, last 24 hours (2 vendors, 1 pipeline)

Claude Sonnet delivered: 35.67MTok
Google Gemini delivered: 42.5MTok
Outcomes delivered: 2,493
Charged: $53.78
Overhead: $42.45

An outcome here is one pipeline run. It may or may not mean anything to the customer — it means something to us.

Cost-to-deliver: $42.45, or $0.017 per outcome.

Revenue-per-outcome: $0.0216, so margin-per-outcome — (charged − overhead) / charged — comes out to 21.1%.

On average, every successful run nets us about 20% margin, for this account.

Now we have real numbers to work with. How much lower are we willing to let that margin go? How many more runs before it hits zero, or goes negative? Would a usage limit protect the bottom line, and should every plan or contract get one?

Attributing raw usage to outcomes

The hard part is typically the mapping. A pipeline run generally isn't just one call — it's several, possibly across multiple vendors, and you need to know which outcome each one belonged to before any of the above means anything.

In Limitr, an outcome is just another credit, tracked the same way tokens are. Every allow(...) call can carry metadata, and it's a common pattern to attribute usage within it, rolling usage up per agent, per feature, via region, the customer's team, or even per run margin and spend data using spend caps.

We can do this because Limitr is a local, context-aware engine that can track state over many enforcement checks. Pricing stays accurate with or without any of it, but the metadata is what lets you slice the analytics afterward. It's what makes a line item like "this pipeline consumed 132MTok of Sonnet" possible.

Tip

Put metrics in terms of real actions or outcomes wherever you can. Keep the raw usage data around for vendor-specific analysis (entitlements should always use the most discrete credit), but the margin-to-deliver KPI itself should be vendor-agnostic — documents read, uploads processed, tool calls made, whatever's meaningful to you.

That way, when you switch models, you're comparing against outcomes you (and your customer) care about, not an arbitrary token count that only means something to the vendor selling it to you.

Limitr measures real-time margin for every credit and token, and rolls them up (and translates them) automatically — this example is the simplified version.

Usage Enforcement

Once you can see margin-to-deliver per account, feature, and vendor, the first real lever you have to control it is enforcement — deciding what a user has access to and how much of it.

Single account on a "pro" plan @ $200/mo (flat or seats)

AI pipeline @ $0.017 avg cost-to-deliver per outcome
AI data aggregation @ $0.042 avg cost-to-deliver per refinement

$200/mo covers 11,764 pipeline runs at that cost. This account used 2,493 in 24 hours — a little over 4 days of runway, nowhere near a 30-day target. Left alone, this account pushes margin negative.

The simplest fix, without touching price: cap usage. One option is to limit the pro plan to 392 pipeline runs a day, and cut off data aggregation entirely. Worst-case margin-to-deliver is now 0% — you'll never lose money on this feature or account again.

But you may not make any money on it, either.

Usage Monetization

At this point, the case for usage-based pricing should be clearer. You want revenue to scale with overhead, so margin-to-deliver becomes something you set and optimize through revenue, not just something you defend by cutting cost.

There's more than one way to do this — plenty of other posts cover pricing strategy on its own. But every approach depends on being able to analyze and enforce usage first, which is the whole point of the last two sections.

A few of the options, for the example we've been using:

Credit model — flexible, supports top-ups, but can be confusing for users when the ROI isn't obvious
Cost-plus-margin — clean and accurate, but rigid, invoices that need explaining, hard to upsell, and it tells the customer exactly what your margin is
Outcome-based — a middle ground, packaged in the customer's language, but harder to guarantee a positive margin-to-deliver on

These aren't mutually exclusive, and the right answer usually differs by feature, by vendor, and by contract. Your objective changes too — margin this month, adoption next. That's the actual argument for keeping the choice in a policy instead of in code: you shouldn't have to ship a release to change your mind.

Here's what that policy might look like for our "pro" plan:

Single account "pro" plan Limitr policy

Monthly subscription: $200/mo (override for annual contracts)
Includes 5 seats, then $29.99/mo per additional seat (hard limit for annual)
Includes 30MTok Claude Sonnet tokens/day, then $5/MTok over (hard limit override for margin control on annual + usage governor for SLA)
Includes 40MTok Google Gemini tokens/day, then $4/MTok over (same overrides)
Includes 500 AI pipeline runs/day, then $0.02/run over (set to observe + unlimited on enterprise + analytics)
Includes 100 AI data aggregations/day, then $0.05/aggregation over (same)

The subscription and seats are doing a specific job here: they're the predictable floor that covers your fixed cost regardless of how the account behaves. Everything below them is the part that moves.

On the contract and invoice (anything presented to the customer), keep it to subscription, seats, and outcomes when possible — in the customer's language, framed as wins where you can. Average cost per outcome makes a good line item. Put the full cost breakdown further down if needed, in its own section, for whoever wants to dig in.

Example account usage, 1 month

Claude Sonnet delivered: 1,080MTok — 900MTok included, 180MTok over
Google Gemini delivered: 1,290MTok — 1,200MTok included, 90MTok over
Successful AI pipeline runs: 30,000 — 15,000 included, 15,000 over
Data aggregations: 4,500 — 3,000 included, 1,500 over
Seats: 5, all included

Worth noting: that's about 1,000 pipeline runs a day, down from the 2,493 we saw in the first 24-hour window. Usage moves. A plan with a visible included amount and a rate past it changes how an account behaves in a way a flat $200 never does — which is exactly why you want to be watching margin-to-deliver continuously, not modeling it once.

Invoiced

Line item	Detail	Amount
Pro Plan subscription	5 seats included	$200.00
AI Pipeline Runs	30,000 runs — avg $0.041/run	$1,220.00
Data Aggregations	4,500 aggregations — avg $0.092/aggregation	$415.00
Total due		$1,835.00

Usage details

AI Pipeline Runs — $1,220.00

Component	Amount
15,000 runs over plan @ $0.02/run	$300.00
Claude Sonnet — 132MTok over @ $5/MTok	$660.00
Google Gemini — 65MTok over @ $4/MTok	$260.00

Data Aggregations — $415.00

Component	Amount
1,500 aggregations over plan @ $0.05 each	$75.00
Claude Sonnet — 48MTok over @ $5/MTok	$240.00
Google Gemini — 25MTok over @ $4/MTok	$100.00

Overhead for the month came to $699 — 30,000 runs at $0.017 and 4,500 aggregations at $0.042. Against $1,835 charged, that's a margin-to-deliver of 61.9%, up from the 21.1% we started with. Nothing got cheaper. Revenue was just allowed to move with the usage driving the cost.

The included usage gives you a good baseline margin-to-deliver, depending entirely on any base platform fees. The cost vs price per credit changes the margin-to-deliver for every credit beyond included limits. And separating internal/external outcomes vs vendors vs included provides flexibility per account, feature, and vendor to move margin-to-deliver according to your goals.

Tip

A credit model could be used with this exact policy to provide even more flexibility for both the provider and consumer. Included, committed, and/or top-ups of abstract credits can be applied at different exchange rates across all other discrete credits, like tokens.

For your user, this actually simplifies what they're looking at, because it's one common unit, clearly scaled to make it easier for them to track, budget, allocate, and commit on. Could also allow them to place their own usage caps in a single unit, so that they never go over what they expect, regardless of how they're used.

Two levers, one usage policy

Cost-to-deliver tells you what an account costs. Margin-to-deliver tells you whether it's worth having, and gives you the numbers you need to actually stay in control.

Enforcement sets your baseline. A limit is a guardrail around the worst case. Cap pipeline runs at 392/day and you've decided, in advance, that this account cannot lose you money. It holds whether anyone's watching or not.

Pricing moves that baseline. Charge for overage and revenue rises with cost instead of being eaten by it. That's the 21.1% → 61.9% swing above.

Enforcement without pricing protects your margin by refusing actions — a 0% floor and a hard cap, for example. For some situations, like agent tool calls or the time an agent gets to converge on a turn, this layer of control makes sense regardless of pricing.

Pricing without enforcement can scale revenue with cost, right up until one run consumes 300x more AI overhead than accounted for.

Both require per-account, per-feature, and per-vendor observability. You can't set a limit you can't measure against, and you can't price an outcome you can't count.

Which is where this usually falls apart. Analytics sit with one vendor, limits sit in application code, prices sit in the billing system — and the three drift. The number you analyzed isn't the number you enforced on, and neither one is the number you invoiced.

This is what we built the Limitr policy for. One document defines the credits, what they cost you, what you charge for them, what each plan includes, and what happens at the limit — observe, soft, or hard.

That same document is what executes at the moment of the call. So the margin you're analyzing, the limit you're enforcing, and the line item you're invoicing all come out of one place, in real time, and they can't drift apart.

To see it in action, check out the live monetization demo — you can play with a simple policy and track actual usage right in your browser.

How to Actually Cap AI Spend for Your Users: 3 Edge Cases Everyone Misses

CJ Cummings — Wed, 01 Jul 2026 20:30:16 +0000

The spend cap for my first AI project was just a limit on an Anthropic API key.

Crossing your fingers that users don't find your limit, and that it's high enough for their usage anyway, is a sign that you need a real control layer.

AI spend control is not trivial

It seems like it would be easy, right? It's just a meter per user and a few if statements.

Then the questions and edge cases start. And don't really stop.

Is that meter counting a credit balance burning down, or raw spend counting up?
Does it include usage the customer's plan already pays for, or only the part that spills over?
If they've got a promo credit grant active, does that count against the cap too, or is the cap only for money actually billed?
Is the meter in my own abstract units or tokens, and how do I map that to all of my vendor costs?
Does the cap reset on its own schedule, or does it follow the plan, billing period, or token reset schedule?

None of these are exotic. They're among the first questions any real customer conversation raises, and they apply anywhere AI usage gets billed.

Limitr Open-Source Project

Defining spend well enough to cap it touches your pricing (vendors, tiers, entitlements), your credit system, and your billing period logic all at once.

That's why we added spend caps to the Limitr open-source project.

Limitr is a runtime for observing, enforcing, pricing, and optimizing usage-based software. It enables us to define plans, entitlements, usage limits, and credits as a single config document; managed in one place, enforced everywhere.

Using it to define, observe, and control costs will make your life much better, for free.

Limitr Cloud adds managed policies, real-time alerting, billing/payments integrations, analytics, and more on top of the same engine.

Edge 1: Overage-only vs. total spend

Say a customer's plan includes $50/month of AI usage before overage kicks in. You (or they) want to cap their overage at $20, not their total spend at $20, which would mean they can barely use the feature they're already paying for.

These are two different caps, and conflating them is a common DIY mistake. A cap that watches total spend will trip long before a cap that watches overage, and customers on a plan with generous included usage will hit a wall that makes no sense to them.

In Limitr, this is a flag on the cap itself:

await policy.addCustomerCap('cus_123', 20, {
  cap_id: 'ai_overage_cap',
  overage_only: true, // only counts spend beyond what the plan includes
});

Without overage_only, the cap counts from dollar one. With it, the first $50 of included usage doesn't touch the cap at all — only the spend that spills past the plan boundary does.

Edge 2: Does a credit grant count against the cap?

Say you gave this customer a $10 promo credit as a goodwill gesture after a support ticket. They start using it. Does that spend count against their $20 overage cap?

There's a real argument for both answers. If the cap exists to protect your margin, promo credit spend absolutely should count — you're still paying the model provider. If the cap exists to protect the customer from bill shock, credit-covered spend shouldn't count, because they're not being billed for it.

Most DIY spend-limit code picks one answer implicitly and never surfaces it as a decision — usually by accident, whichever way the meter happens to be wired. It's the kind of bug that doesn't show up until a customer emails asking why their "free" credits triggered a spending block.

await policy.addCustomerCap('cus_123', 20, {
  cap_id: 'ai_overage_cap',
  overage_only: true,
  ignore_grants: true, // spend covered by an active credit grant doesn't count
});

ignore_grants: true means the cap only tracks spend the customer is actually being billed for. Flip it off and grant-covered usage counts too, useful if the cap is really a margin guardrail rather than a customer-facing promise.

Edge 3: Independent reset schedules

Plans reset monthly. Caps often need to reset on a completely different cadence, a $15 weekly guardrail layered on top of a $50 monthly plan limit, for instance. If your reset logic is a single cron job that zeroes out "the counter," you don't have two limits, you have one limit with an identity crisis.

Caps in Limitr carry their own reset schedule, independent of the plan or credit they're watching:

await policy.addCustomerCap('cus_123', 15, {
  cap_id: 'ai_weekly_guardrail',
  overage_only: true,
  reset_sch: 'weekly:mon', // resets every Monday
});

That weekly guardrail and the customer's monthly plan meter now tick independently, on their own clocks, without needing to know about each other.

What this looks like end to end

Here's the shape of it, a customer on a plan with included Claude usage on a custom price, a monthly overage cap that ignores promo credits, and a tighter weekly guardrail underneath it:

// npm i @formata/limitr
import { Limitr } from '@formata/limitr';

// JSON, YAML, TOML, or STOF (default)
const doc = `
policy: {
    credits: {
        claude_sonnet_4: {
            description: 'Claude Sonnet 4 token'
            overhead_cost: 1.5e-7
            price: { amount: 0.0003 }
        }
    }
    plans: {
        starter: {
            label: 'Starter Plan'
            entitlements: {
                ai_chat: {
                    description: 'AI chat feature'
                    limit: {
                        credit: 'claude_sonnet_4'
                        mode: 'soft'  // allow overage & send overage events
                        value: 16667  // ~$50 included at $0.0003/token
                        resets: true
                        reset_sch: 'monthly:1'
                    }
                }
            }
        }
    }
}`;

const policy = await Limitr.new(doc);
await policy.createCustomer('cus_123', 'starter', 'user', 'Jane Doe', [], [], {
  email: 'jane@example.com',
});

// $20/month overage cap - doesn't count included usage or promo credits
await policy.addCustomerCap('cus_123', 20, {
  cap_id: 'ai_overage_cap',
  overage_only: true,
  ignore_grants: true,
  reset_sch: 'monthly:1',
});

// $15/week guardrail - independent clock, same overage-only logic
await policy.addCustomerCap('cus_123', 15, {
  cap_id: 'ai_weekly_guardrail',
  overage_only: true,
  ignore_grants: true,
  reset_sch: 'weekly:mon',
});

// Spend! Call into LLMs, upload files, run GPU jobs - Limitr handles it all
// Vendor and price agnostic - change in the config without touching code
if (await policy.allow('cus_123', 'ai_chat', 6420)) {
  // within plan + caps, allowed and recorded
} else {
  // spend capped, hard limit hit, usage governed, etc.
}

// Get current state information
const cap = await policy.customerCap('cus_123', 'ai_overage_cap');
console.log(`Current customer overage (USD): $${cap?.meter_value ?? 0}`);

Three flags, two independent clocks, one call to allow(). Everything upstream of that call — plan boundaries, grant interactions, reset timing — is already resolved by the time you get a yes or no.

Three easy decisions, one hard system

Each of those flags is a seemingly five-minute internal conversation on its own with a clear answer in the moment. Overage-only or total? Grants in or out? Reset on its own clock or the plan's?

But in a real system, both sides of each question turn out to have a real customer behind them, and reversing a decision later isn't free. Flip overage_only after an enterprise customer is already under contract, and you've quietly changed what they agreed to pay for. Ship the wrong ignore_grants default, and support finds out from an angry email, not a design review.

There's more to spend than a single dollar figure, too. Vendors don't all meter the same way, and mapping tokens, GPU-seconds, and flat fees onto one number is its own can of worms. That's a topic for another post.

If you're building metered AI features and want to poke at this yourself, make sure to check out Limitr.

We Deserve Better Than JSON as a DSL

CJ Cummings — Mon, 30 Mar 2026 14:00:00 +0000

It always starts innocently, you're creating a new config or endpoint with some dynamic behavior, thinking "KISS - Keep It Stupidly Simple". What's simpler than JSON or TOML? Some key-value pairs, maybe some nesting, we'll actually keep it clean this time.

Any experienced programmer knows what comes next.

Six months later you're hunting for a $ref in a sea of JSON, writing helper tools, trapped under a stack of legacy decisions, full of regret.

I've done it. You've done it. The entire industry has done it. GitHub Actions is YAML with a custom expression language bolted on. Terraform invented HCL because JSON wasn't expressive enough. OpenAPI is JSON Schema with extensions piled on top. Every AI framework has its own JSON-based tool definition format that's slightly different from the others.

The problem isn't the formats. JSON, YAML, TOML - they're all fine at what they do. The problem is that we keep asking them to carry logic they were never designed to hold, then building increasingly elaborate scaffolding when they inevitably buckle.

What if your data could just do what's needed?

Here's the part that made me want to build Stof.

import { stofAsync } from '@formata/stof';

const doc = await stofAsync`
    name: 'Stof'

    fn loaded() -> str {
        const stof = await Ext.fetch();
        parse(stof, self);
        self.say_hello()
    }
`;

doc.lib('Ext', 'fetch', async () => {
    return `fn say_hello() -> str {
        'Hello, ' + (self.name ?? 'World') + '!'
    }`;
});

console.log(await doc.call('loaded'));  // Hello, Stof!

The document starts without a say_hello function. It fetches more Stof from somewhere, an API, another service, an agent, then parses it into itself and calls the function that just arrived.

Stof runs in a WASM sandbox built in Rust and is just a document of data, like JSON (actually a superset of JSON). It can't touch your filesystem, network, or memory unless you explicitly bridge it to the host environment with doc.lib(). You control exactly what the context can reach.

This means a service can share its capabilities as Stof. Not a description of what it can do, but the actual logic. The consumer parses it into context and starts using it immediately, no client library, no SDK, no redeployment. Your system ships with certain capabilities and gains more at runtime.

The whole picture

Stof is a superset of JSON, so your existing data is already valid, but with functions, types, unit conversions, and async execution built into the format itself instead of a layer on top.

Instead of trying to replace existing interchange formats, Stof is the glue layer that works with all of them. Parse JSON, YAML, TOML, STOF, etc. into a single document at any time, add the logic that belongs, and send it anywhere. Export portions to the format your app expects internally.

const doc = await stofAsync`
#[type]
Server: {
    port: 8080
    host: 'localhost'
    secure: false
    MiB memory: 500GiB

    fn url() -> str {
        let url = self.secure ? 'https://' : 'http://';
        url += self.host + ':' + self.port;
        url
    }
}`;

// Parse JSON, YAML, TOML, binary, or more Stof into the same document
doc.parse(`Server "prod": {
    "host": "prod.example.com",
    "port": 443,
    "secure": true,
    "memory": "2GB"
}`);

console.log(await doc.call('prod.url')); // https://prod.example.com:443
console.log(doc.get('prod.memory'));     // ~1907 MiB (auto-converted from GB)

console.log(doc.stringify('toml', doc.get('prod') as string));
/*
host = "prod.example.com"
port = 443
secure = true
memory = 1907.3486328124998   # MiB
*/

The Server type defines shape, defaults, and behavior. When you parse new data in (JSON, YAML, TOML, whatever) and cast it to that type, it gets the functions and validation for free.

Schemas that don't drift

You know what's worse than writing a JSON Schema? Keeping it in sync with the thing it validates.

Here's the JSON Schema for a simple server config:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "port": {
      "type": "integer",
      "exclusiveMinimum": 1024,
      "maximum": 65536
    },
    "address": {
      "type": "string",
      "minLength": 1
    },
    "memory": {
      "type": "string",
      "description": "Memory in MB, must be at least 256"
    }
  },
  "required": ["address"]
}

That's a separate file, a separate format, a separate thing to maintain. And notice that memory, because JSON Schema has no concepts of units, the best you can do is write a comment and hope the user reads it. The validation logic for that field lives somewhere else entirely, probably in your application code.

In Stof, validation lives on the fields themselves:

#[type]
Server: {
    #[schema((target_val: int): bool => target_val > 1024 && target_val <= 65536)]
    int port: 8080

    #[schema((target_val: str): bool => target_val != "")]
    str address: "localhost"

    #[schema((target_val: MiB): bool => target_val >= 256MB)]
    MiB memory: 2GB
}

A port between 1024 and 65536, a non-empty address, and at least 256MB of memory. The last one is meaningful because Stof understands units as types, so pass "2GB" and it converts, pass in "100MB" and it fails. The schema can't drift from the data because it is the data.

A real production use case

Limitr is an open source pricing and enforcement engine built on Stof. The entire policy, plans, credits, limits, validation logic, lives in a single Stof document. It's a good example of what "data that carries its own logic" looks like when it gets past the toy stage.

Give it a try

The fastest way in is the playground, runs in your browser via WASM, no install needed.

npm i @formata/stof      # TypeScript/JavaScript
pip install stof         # Python
cargo install stof-cli   # CLI

Docs - reference
GitHub - source and issues
Discord - come tell us about your use case

There's also a VSCode extension for Stof syntax highlighting.

Apache 2.0.