DEV Community: batonyzeng-sketch

Before You Buy AI Tokens for Your Team: The 3 Checks Most Buyers Skip

batonyzeng-sketch — Sat, 14 Nov 2026 03:14:17 +0000

If your team is shopping for AI tokens — or, more precisely, for reliable access to large-model APIs — the conversation almost always opens the same way: "Which models do you support?" and "What's your price per million tokens?"

Those are the comfortable questions. They're also the ones that matter least.

The questions that decide whether the tokens you actually pay for move your business are the ones nobody asks until something breaks: a call times out in another country, an invoice can't be issued in the currency you need, an incident happens and nobody is clearly on the hook. By then the token budget is spent.

Here are the three checks worth doing before you sign.

1. Settlement and invoicing — the line item that isn't on the price tag

A per-token price is meaningless if you can't actually reconcile it. Three things decide that:

Which pricing caliber the quote uses. "Per token" can mean input and output priced separately, or bundled. "Per flow" can mean a failed retry counts as a new flow. The number looks comparable until you realize you're comparing two different cost structures.
The exchange-rate basis. Is the rate fixed on the quote date, or floating? A fixed rate means both sides absorb a slice of volatility; a floating rate means finance re-explains the variance every month.
The settlement-currency boundary. Which currencies can be collected, which invoices can be issued locally, and which cannot be promised — these are the lines that surface only when you're ready to move money.

A useful tell when you're evaluating a supplier: do they publish their price list, pricing caliber, and settlement boundary openly, so both sides reconcile against the same table? Some do. aiokeys.com puts its RUB price list, pricing caliber, and settlement-currency boundary on the site — not because it's the cheapest, but because "we can both see the same number" removes a round of email before you've even started.

2. Connectivity — where the API call actually originates

This is the one teams in a single market rarely think about, and teams operating across borders learn the hard way.

You buy the token. You wire up the integration. And in one market the calls sail through; in another they time out, or silently slow to a crawl. The cause is usually not the model. It's the network the calling device registers on.

Two 5G modes exist: NSA borrows the 4G core network for data; SA runs on an independent 5G core. If the device supports one and the local operator has only commercialized the other, the device quietly downgrades. Lights are on, signal shows full bars, the dashboard says connected — but it's registered on 4G, and your inference call runs on 4G expectations. In a market where SA isn't live yet, a "5G" device simply works as 4G LTE. Usable, but not what you priced for.

None of this shows up in a model spec. It shows up in the acceptance checklist — if you write one. The token is step two. Confirming the network can actually carry the call is step one. Buy the token first, and you may have bought something you can't invoke.

3. Responsibility — private deployment moves the request, not the accountability

A common reflex when cross-border data or compliance comes up: "We'll deploy it privately / in our own region." That solves one thing — the request doesn't cross a border. It does not solve the rest.

Who is notified when something fails? Who can read the request logs? Where is the data-retention period defined? Who is the responsible party when an incident hits? Those remain yours. Technology can be outsourced; the accountability cannot. A private deployment is a real trade-off — you gain a controllable data boundary and you take on operations, scaling, and model updates. Writing "more secure" on the proposal is not the same as writing down what you gained and what you kept.

The part that's actually the product

Tokens are, at this point, close to a commodity. The model list is a spreadsheet. The price per million is a number you can paste into a calculator.

What's not a commodity is the set of boundary conditions wrapped around the token: can you settle and invoice the way your business actually runs, will the call reach the model from where your team sits, and when it breaks, who answers. Those are the checks that separate "we bought AI" from "we can use AI."

So before the next token quote, run the three checks. And when a supplier shows you a transparent price page instead of a sales deck, that transparency is itself part of the deliverable.

Golden line: The token isn't the risky part. The conditions attached to it are.

Further reading: aiokeys.com/tseny — openly published RUB pricing, pricing caliber, and settlement-currency boundary.

The Corporate LLM Gateway: Why GPU-Hour Is the Wrong Metric

batonyzeng-sketch — Fri, 13 Nov 2026 05:59:12 +0000

An H100 80GB rents for anywhere between $1.30 and $12.29 per hour in public price lists. That is not a cherry-picked sample. It is the spread across providers tracked by Coloprice as of November 6, 2026, where the median sits at $3.90. Runpod lists the same card at $2.89 in Secure Cloud and from $1.99 in Community Cloud, billed by the second. Eight times the price for identical silicon, and that is still not the uncomfortable part of the job.

The uncomfortable part is that GPU-hour price does not decide anything in a corporate deployment. It does not even make the top three arguments once the project reaches security review and the CFO. I am writing this after several deployments where we calculated savings first and built the gateway second. Every time, the savings turned out not to be the reason the thing got built.

What follows is a cost model with the assumptions written down, the architecture of an exit point, the code for a minimal proxy, and three failure modes I have seen in production.

What actually breaks with direct access

Calling a foreign API straight from a corporate network runs into three problems, and none of them is bandwidth.

Keys get revoked. Providers see requests from certain regions and treat them as a regional policy violation: first a 403 on individual models, then a full key block. Recovery means correspondence in which you explain why your office IP ended up where it did. Over an eight-month pilot this happens two or three times, and each occurrence is downtime for a specific team rather than an abstract risk.

Payments do not reconcile. A corporate card does not go through, so you need an intermediary, and an intermediary means a contract, closing documents and VAT. A new cost line appears that was never in the project budget, and it lives separately from the technical work.

The third one is what usually stops the project. If a prompt carries a name, a phone number or bank details, you cannot prove afterwards that it did not. With direct access from applications there is no place where that would be recorded: every developer put their own key into their own service, and nobody retains a trace of what exactly left the perimeter. That is no longer about a 403. It is about data protection law (152-FZ in Russia) and about having nothing to answer an auditor with.

Recalculating the cost of a single request

Take a typical profile: an internal assistant with retrieval over company documents. 1.5 million requests a month, averaging 900 input tokens and 350 output tokens. That is 1.35 billion input and 525 million output tokens.

At an open-model price of roughly $0.15 per million input and $0.60 per million output, you get about $203 for input and $315 for output, so roughly $518 a month. The order of magnitude matters more than the precision here: substitute your own price and the conclusion stays in the same range.

Then the arithmetic breaks the original framing. If the cache hits 30 percent of the time, you save about $155 a month, which is around thirteen thousand roubles. An engineer who owns the gateway, the routes, the updates and incident triage costs 250,000 to 300,000 roubles a month in Russia, taxes included. That is five percent of their cost, and only with a good cache hit rate. At these volumes a gateway does not pay for itself through token savings. It pays for itself through something else.

What it actually pays for

A single personal data leak costs a legal entity three to five million roubles under part 12 of article 13.11 of the Administrative Code when 1,000 to 10,000 data subjects are affected, five to ten million for 10,000 to 100,000, and for a repeat offence the fine is calculated from turnover. For comparison, the annual payroll of that same engineer is 3.0 to 3.6 million. One incident costs more than a year of their work, and that number excludes reputational damage and projects frozen during the investigation.

Which answers the original question. You do not install a gateway to save on GPU-hours. You install it so there is one place where it is visible what left, what stayed, and why a particular decision was made. Savings arrive as a side effect rather than as the goal.

What the exit point does

Put it where traffic is already aggregated but has not yet left the perimeter: between the applications and the outside world. From there it takes on four jobs that cannot be done inside a client application.

Masking happens first, before anything is sent. Not after the response, not in the logs, but in the request body before the upstream call: entities are replaced with placeholders backed by a reversible dictionary so the answer can be reassembled. It is also fair to admit that masking breaks some workloads. Which ones is covered below, because it is not a detail.

Next is routing by sensitivity. Ticket classification, field extraction from contracts and everything else on a short context goes to a small model inside the perimeter. Long analysis, code generation and image work go to a large model through a tunnel, but only after masking. The rule lives in configuration rather than in a developer's head, otherwise in a month nobody can say why a given request went where it went.

Then quotas. A limit per team, a separate counter for experiments, a monthly ceiling per project. Without this one enthusiast with a batch script consumes the budget in a week, and you learn about it from the invoice.

And the decision log, without which everything above is pointless. For every request: an identifier, which masking was applied, which route was chosen, what the response cost. This log is what you show an auditor. Its absence is usually what "no compliance" actually means, even when everything was formally approved.

A separate note on streaming, because almost every hand-built gateway trips here. Chat returns the answer over SSE, and if the gateway buffers the whole body to count tokens and write the log, the user stares at nothing for twenty seconds and then gets the entire text at once. That feels worse than a slow but live answer. Read usage from the final chunk and write the log after the stream closes, otherwise you get either added latency or lost records when the connection drops.

Code: a minimal proxy

Below is the base I usually start from. The token comes from the environment, every request has an identifier, the timeout is bounded, and retries happen only on transport errors so that the same request is not billed twice.

import os, uuid, asyncio
import httpx
from fastapi import FastAPI, HTTPException, Request

app = FastAPI(title="corporate-llm-gateway")
UPSTREAM = os.environ["UPSTREAM_URL"]
TOKEN = os.environ["UPSTREAM_TOKEN"]

@app.post("/v1/chat")
async def chat(request: Request):
    payload = await request.json()
    rid = request.headers.get("x-request-id") or str(uuid.uuid4())
    payload = sanitize_pii(payload)                    # mask BEFORE leaving the perimeter
    route = pick_route(payload, request.headers)       # small model / large model / reject

    async with httpx.AsyncClient(timeout=httpx.Timeout(30.0, connect=5.0)) as client:
        for attempt in range(2):
            try:
                r = await client.post(
                    route.url,
                    json=payload,
                    headers={"Authorization": f"Bearer {TOKEN}", "x-request-id": rid},
                )
                if r.status_code in (429, 500, 502, 503, 504) and attempt == 0:
                    await asyncio.sleep(0.5)
                    continue
                log_decision(rid, route.name, r.status_code, r.headers.get("x-usage"))
                return r.json()
            except httpx.TransportError:
                if attempt == 1:
                    raise HTTPException(504, "upstream unavailable")

The sanitize_pii, pick_route and log_decision functions are deliberately left out: the first is an entity dictionary and regular expressions for your formats, the second is a rule table, the third writes to your logging system. What matters is not the implementation but the order. Masking and route selection run before the network call, not after it.

Three failure modes I have seen

A cache keyed on the wrong thing. Response caching looks like free money until the key omits the model version, the system prompt and temperature. This surfaces in support: answers meant for the basic tier reach premium customers because the key was built from the question text alone. Semantic caching is more honest about hit rates and more dangerous about consequences. Fifteen to thirty percent hits on a stream of similar requests is entirely achievable, but every hit is an answer to a similar question rather than to the same one.

Masking breaks code generation. Replace identifiers with placeholders and the model loses the structure: variable names and dictionary keys become identical labels, and the generated code does not compile. The fix is not to disable masking but to use different policies for different task types. Code goes through a route where source is allowed and personal data is not, while documents containing such data go through strict masking.

Failover that doubles the bill. When a route degrades, the gateway retries on a backup while the first request is neither cancelled nor untariffed, it is simply billed. On a stream of more than a million requests that is visible money and, worse, duplicated answers. You need an idempotency key and an explicit cancellation of the previous call on timeout.

How to test it

Synthetic pings are useless here. A test that shows something looks like this: record a real day of request traffic, replay it through the gateway, and drop one route halfway through the run. Then look at two numbers: the share of successful responses and the p95 latency. If both stay inside the contracted bounds, the architecture works. If they move, the problem is in the balancer or the timeouts rather than in the model, and that distinction matters during an incident review.

Test masking separately: run a set of requests with known entities and verify that none of them appears in the outbound log. That test should be written before the first incident rather than after it.

When you do not need a gateway

If you have one team, ten developers, no personal data in prompts and fewer than fifty thousand requests a month, you do not need a gateway. A hundred-line wrapper, a single key in a secret store and a ban on hardcoding keys solve the same problem more cheaply. I have seen projects that built a gateway "for the future" and then spent six months unable to explain who owns it.

A gateway becomes mandatory at exactly the moment a second application gets model access and the first personal data clause appears in a contract. Before that it only adds a service to your on-call rotation.

Takeaway

GPU-hour price is a poor metric for this problem because it answers "what does the hardware cost" while the decision is made on "what happens if data goes somewhere it should not". An eightfold spread on the same card is a good illustration of how secondary the first question is. Even if you pick the cheapest price, the answer to the second question does not change.

The practical conclusion is simple. Count the cost of one successfully handled request, including masking, routing and logging, rather than the cost of an hour of rental. And keep the exit point where the data boundary is: if you cannot show where a request was masked and where it went, you have neither compliance nor an SLA.

Sources

Prices and spreads were verified on November 11, 2026: the H100 80GB summary on Coloprice, rates on the Runpod pricing page. Fines are quoted from part 12 of article 13.11 of the Russian Administrative Code as amended with effect from May 30, 2025.