DEV Community: Bharath Koneti

You Picked a State of the Art Model. The API Is Making It Hard to Tell.

Bharath Koneti — Fri, 09 Oct 2026 15:23:26 +0000

The model can do the job. The service delivering it has a job to do, too.

You compare models, run your evaluation set, and find one that gets it. It handles the task, follows your instructions, and gives you answers your application can actually use. Then you put it behind an API. Some requests drag. Some answers stop just before the useful part. A retry saves the day, but the records make it look like the day never needed saving. Then something changes on the serving side, and the model with the same reassuring name starts acting differently.

Your users don't care how well it did on your benchmark last Tuesday. They care about the response they're staring at. Hard to blame them.

Choosing a capable model matters. Getting that capability to users consistently also depends on who's serving it, with what configuration, limits, routing decisions, and behavior under your workload. That doesn't make the API responsible for every bad answer. Models get things wrong. Prompts get things wrong. Integrations find remarkably creative ways to get things wrong. The point is to measure enough to tell those failures apart, instead of throwing them all into a bucket labelled “AI quality.”

At Inferock, our Baseline defines 19 quality requirements for evaluating model routes. They're part of a broader qualification framework. Here, we're looking specifically at the quality checks. One distinction matters before we get into them: these requirements are proposed. The current Baseline publishes no performance results and contains no completed route assessment. Minimums say what a route would need to meet to qualify; targets say what we're aiming for. Neither means a route has passed. So, before calling an inference service good, here's what we'd want to know.

Did we get an answer the application can use?

Say you're building a document assistant. It reads a policy, extracts the eligibility rules, and returns structured data so your application can display a comparison. The model might understand the document perfectly. Your application still needs the whole result, in the format it asked for.

Five requirements check different ways that handoff can fail:

Broken output (Q01): does a completed response satisfy its declared, supported schema?
Unexpected truncation (Q02): did generation stop early even though the output budget was sufficient?
Tool call validity (Q12): are the tool names and arguments structurally valid under the supported contract?
Unexpected filtered omission (Q15): was benign output unexpectedly removed, where the provider exposes filter evidence?
Clean stream completion (Q16): did a started stream end with the terminal or finish evidence its protocol requires?

Each of the five needs a different fix. A bigger output budget won't fix an invalid tool name. A stream that finishes properly can still deliver an object that fails your schema. And if the user deliberately cancels a request, that shouldn't show up as truncation caused by the provider.

For the document assistant, a lovely paragraph isn't a substitute for the structured object you requested. On the other hand, an object that parses perfectly can still contain the wrong eligibility rules. “Valid structure” and “correct answer” are two separate checks.

The quality definitions spell out those boundaries. They also make sure missing filter evidence doesn't get quietly interpreted as proof that nothing was omitted. “We didn't observe a problem” is less comforting when nobody could observe it in the first place.

Did it arrive before the user tried again?

The extraction can be correct and still take long enough that the customer hits submit a second time. Your model evaluation might call that a success. Your customer probably has a different word for it. Availability (Q08), latency distribution and generation speed (Q09), and retry amplification (Q17) cover this part of the experience.

Availability means eligible operations get a complete response, valid under their protocol, within their deadline, measured before retries. “The endpoint was reachable” doesn't quite cut it. Uptime measured over time and request success need to stay separate. Individual routes need to stay visible, too. One healthy provider shouldn't get to hide another provider's failures in an average.

For latency, a tokens per second number isn't enough. Measure when the first usable token arrives, the gaps between output tokens, and how long the full response takes. Look at the distributions, including the tail where some unlucky customer is sitting. A quick opening sentence followed by a long stall is still a long stall.

The Baseline's interactive profile shows how specific these measurements need to be: traffic without reasoning, 2,000 input tokens, 256 output tokens, a client in the same region, and an established verified session. For that profile, the proposed p95 limit for time to first token is two seconds at the minimum, with a target of one second. Full completion has a separate p95 limit: 13 seconds at the minimum, seven seconds at the target.

Those numbers belong to that profile. Long context and reasoning workloads have different requirements. Setup for a fresh session gets measured separately. Pulling out the fastest number and calling the whole service “fast” leaves behind most of the information that made the number useful.

Retries need their own accounting as well. Q17 measures total upstream attempts per logical operation, including failed attempts, and reports eventual completion separately. Its proposed minimum is no more than 1.020 attempts per operation; the target is 1.005. Across 10,000 logical operations, that allows 200 or 50 extra attempts, respectively. The ratios don't tell you where those attempts went, so keep the underlying records.

A retry can rescue the user's task. It doesn't retroactively make the first attempt successful. And replaying a tool with side effects is not an acceptable way to give your completion chart a little lift.

Conceptual illustration of the checks along a route. This is not a product screenshot or a completed route assessment.

Does it still work when something changes?

The document assistant passed your evaluation. Then the serving configuration changed. Does it still extract the right rules? Three requirements deal with this: unnecessary refusal (Q10), quality retained after changes (Q11), and grounded factual accuracy (Q14).

A valid safety refusal can count as an available service response. Whether that refusal was appropriate for the task is a different question. To check unnecessary refusals, you need prompts labelled benign and supported, matched to the task, language, and policy. Removing legitimate safety refusals to improve your completion percentage would certainly move the number. Just not in a direction worth celebrating.

To evaluate a change, you need a fixed reference. Use identical versioned tasks and sampling settings, and look at uncertainty alongside the scores. If a serving change lowers quality, making the lower score your new reference only shows that the goalposts are surprisingly portable.

Factual accuracy needs a clear scope, too. “Does this model tell the truth?” is too broad to be useful here. For the document assistant, you can use a fixed, answerable corpus, source material, and a declared rubric. That gives you a consistent way to score unsupported claims, missed answers, and unjustified abstentions. The Baseline's proposed 98% accuracy target applies to that kind of bounded evaluation. It says nothing about how truthful a route is on arbitrary prompts or for every model.

What a developer needs to know is whether this route still meets this workflow's requirements, on an evaluation someone else could understand and repeat. Without that context, a percentage is a decent badge and a lousy debugging tool.

Do the records agree about what happened?

The service also needs to keep trustworthy records. That means checking billed empty output (Q03), OpenAI token recount (Q04), Anthropic token cross check (Q05), request identity and duplicate charging (Q06), cache pricing anomalies (Q07), and known price before routing (Q19). You need to know which operation ran, what it delivered, and whether the records line up. Otherwise, your route comparison starts with a debate over whose spreadsheet to believe.

There are a few easy traps here:

A valid tool payload can be usable output even if the response contains no prose.
Legitimate retries can share an idempotency key.
A cache miss charged at its documented rate isn't automatically a cache pricing anomaly.

Token comparisons specific to a provider need comparable components, too. Message framing, reasoning, multimodal input, and documented system additions can all affect what you can reconcile. An unexplained difference deserves investigation. A counting estimate, on its own, doesn't prove overcharging.

The known price requirement is about having explicit terms before a billable routing decision. If the price is missing, record missing evidence. Don't helpfully turn it into a zero and congratulate yourself on the savings.

If the document assistant takes three attempts, we should be able to identify all three and reconcile what came back. The next investigation shouldn't start with everyone trying to remember which request happened when.

Did we get the route we approved?

The remaining two requirements are security and governance signals (Q13) and served model identity (Q18).

If you evaluated a particular model revision and configuration, silently serving a different one changes the basis of your decision. Route selection needs to respect the approved contract. A matching model label is useful evidence. It doesn't independently prove the exact weights that ran.

Security and governance checks need the same care with scope. Q13 covers confirmed critical exposures and required policy evidence within an authorized, bounded evaluation. It does not certify confidential computing or end to end encryption. The Baseline has a separate encryption and privacy category for its own requirements.

These checks belong in a quality discussion because customers need to know what service they actually received. A fast, fluent answer doesn't make an unauthorized model substitution or a critical exposure acceptable. Some requirements are gates. You don't get to average them away because the other charts are having a good day.

Measure the service your application actually uses

Those five questions cover all 19 quality requirements. In each case, we're assessing the whole route behind the model name. That assessment needs a defined scope: model revision, provider route, region, client and policy, configuration, workload, and traffic envelope. Change the scope, and you need to check whether the old evidence still applies. An interactive chat result doesn't establish long context performance. A quiet test doesn't tell you how an overloaded deployment behaves.

The Baseline's qualification rules also distinguish a genuinely optional check from a required capability the route doesn't support. Missing evidence isn't a pass. And a mandatory feature your workflow needs doesn't become “not applicable” just because that would make qualification easier.

You don't have to memorize 19 identifiers before shipping. You do have to decide what successful inference means for your application, then keep enough evidence to check whether you're getting it. For the document assistant, that means complete extraction, the right structure, grounded answers, an acceptable wait, and the approved serving configuration.

Where Inferock fits

At Inferock, we're building accountable inference around that distinction: access to models, visibility into what happened, and clear limits on what the evidence actually proves.

Inferock Watch lets you keep your existing provider accounts while gaining visibility into the quality, latency, and usage of calls routed through Inferock. Our inference workflow connects route and response evidence with checks, using active probes where a reference is needed. That workflow and the Baseline's proposed qualification requirements are related. They are not interchangeable proof that every requirement has been met.

To see it with your own calls, start with the Inferock Watch quickstart. Use the Baseline to ask better questions about the routes your application depends on.

You already spent time choosing a capable model. It would be nice if your users could tell.

Your LLM Dashboard Says “Success.” Show Us the Evidence.

Bharath Koneti — Wed, 07 Oct 2026 14:29:06 +0000

Following one request through Inferock, from a provider response to a finding an engineer can actually inspect.

Your agent needs to look up a customer’s account. It asks the model to call lookup_account, passing the account ID along with the other arguments.

The provider returns HTTP 200. The response arrives. The JSON parses.

Your dashboard logs another successful request. Meanwhile, your tool gets this:

{
  "region": "eu",
  "include_history": false
}

No account_id.

Congratulations. The braces match. The function would still like its arguments.

This is an illustrative example, but the distinction matters: the model API completed the request. Your application got something it couldn’t use. Both can be true, even if your dashboard only has room for one green badge.

At Inferock, we want you to be able to follow that gap. Which request produced the response? What was it supposed to return? What actually came back? Which check failed?

“The model did something weird” is a perfectly reasonable opening line for a debugging session. It’s a lousy place to end one.

Write down the requirement

Here’s the tool’s argument schema for our example:

{
  "type": "object",
  "properties": {
    "account_id": { "type": "string" },
    "region": { "type": "string" },
    "include_history": { "type": "boolean" }
  },
  "required": ["account_id", "region"]
}

The returned arguments are valid JSON. They’re not valid against this schema.

That gives us something specific to point at: account_id is required, and it’s missing. No debate about whether the response seemed helpful. No adjudication by vibes.

But the requirement has to be declared before the check means anything. If your application needs an account ID and never says so, an observer can’t reliably discover that expectation by staring harder at region and include_history.

Your application also needs to enforce the contract. Reject invalid arguments before executing the tool. A failure record helps you investigate what happened; it doesn’t replace validation at the execution boundary.

Give the request a trail

We’ll use Inferock Watch for this walkthrough. You connect your existing provider key, then authenticate gateway requests with a separate Inferock key. Your provider account still serves the inference.

Two keys, two jobs:

The provider key goes in provider setup.
The Inferock key goes in your application’s gateway authentication.

Our account setup guide covers that separation and the key replacement workflow. Mixing them up won’t make the setup simpler. It will just give you a different problem to debug.

The current guides cover the development gateway. The documented Chat Completions example uses these headers:

Authorization: Bearer <your Inferock key>
x-governance-provider: openai
x-request-id: account-lookup-001

Those headers handle authentication, routing, and request identity. A tool call request also needs the model, messages, and tool definition in its body.

Use the gateway base URL from your account settings, and give each new request a unique ID. That ID is how you connect an application event to the measured call later, rather than squinting at timestamps and hoping for the best. The First call guide has complete curl, Python, and Node.js examples.

For successful calls, the gateway preserves the upstream HTTP status and provider response content. No Inferock measurement envelope wraps the provider’s answer.

So yes, our broken account lookup can still come back through the gateway with HTTP 200. The status code keeps its usual meaning; we add evidence you can use to judge the output separately.

Find the call and evidence

After the provider result, the gateway emits a measurement event. Open Calls and find account-lookup-001.

Depending on what’s available, the call details can include the provider, model, request ID, attempt and failure class, timing, usage, linked findings, and redacted payload evidence.

For this lookup, you want answers to four questions:

Is this the request our application rejected?
Did the check have access to the declared tool schema?
What arguments did the model return?
Did JSON parsing succeed, and which schema requirement failed?

Keep those answers attached to the same request. An application log saying “validation failed” and a provider response saying “200” aren’t much help if you can’t establish that they’re talking about the same call.

The docs point you to Proof for finding evidence and proof packets. That’s where you move from “we counted a failure” to “here’s what supports the finding.”

Illustration only: the declared contract requires account_id, the response omits it, and the finding connects the two. This is not a product screenshot or a measured customer call.

The explanation should be refreshingly boring: the arguments parsed, the schema required account_id, and the returned object didn’t contain it.

Another engineer can inspect that reasoning. They can challenge it, too. If the wrong schema was supplied, the finding needs correcting. Transparency doesn’t get to stop being transparent when someone spots a mistake.

Know what the failure proves

The missing field supports a structural failure finding. It doesn’t explain why the model omitted it, prove that another prompt would have prevented it, or tell us what happened inside the provider’s infrastructure.

And a response with every required field isn’t automatically correct. The model could return the wrong customer’s account ID and pass this schema just fine. Your application may still need authorization checks, database verification, or other business rules before using it.

Inferock’s measurement methodology separates objective failures, signals that need thresholds, and flags that need review. A missing required argument belongs in the first category when the schema and response evidence support it.

Other claims need different evidence. A latency complaint needs a service expectation to compare against. A factuality claim needs ground truth. Putting them on the same dashboard doesn’t magically give them the same certainty.

There’s another boundary to check: did the request reach the model at all?

Inferock’s admission controls run after authentication and before the provider call. A rejected payload or exhausted quota doesn’t create a provider loss measurement. The limits guide explains the error reasons and recovery guidance.

That matters during an incident. A request rejected before inference and a completed provider response with invalid arguments are different failures. They need different fixes.

In our account lookup, we have a provider result and an output contract violation, so we can investigate the returned arguments. If admission had rejected the request instead, blaming the model would send us off debugging a response that never existed.

One generic error counter tends to blur all of this together. Very efficient at telling you something happened. Less impressive at telling you what.

Make the summary earn it

Calls lets you inspect the individual request. Proof holds the finding evidence. Home summarizes the reporting period.

The receipt guide documents states such as “No data yet,” “Partial data,” and “Verified,” along with the standard version and receipt generation time. Those details help you read a summary without pretending every number on the screen is equally established.

For engineers, the useful part is the path back to the request. An aggregate can tell you there’s a problem. The evidence underneath helps you decide what to change.

A measured failure also doesn’t automatically qualify for failure credit. Eligibility is a separate decision under the credit promise and the terms that apply to your account. The technical finding and the remedy status need to stay separate, however tempting it is to turn them into one convenient badge.

Back to our account lookup.

We started with a green status and a tool that couldn’t run. Now we have a request ID connecting the application event to the call, a declared requirement, a response that breaks it, and an explanation another engineer can inspect.

That’s enough to start doing useful work: check the tool definition and request construction, examine affected calls, and verify whether a change fixes the same failure.

We still have to do the engineering. Sadly, the dashboard has not volunteered.

This is what we mean by accountable inference at Inferock: route the call, attach observable evidence, and make the outcome inspectable, including the limits of what that evidence proves.

To follow the same path with your own provider account, start with the Inferock Watch quickstart and find your first request in Calls.

The green badge can stay. We’d just like it to come with an explanation.

A Stream Can Start. Finishing Is Another Matter.

Bharath Koneti — Mon, 05 Oct 2026 12:40:06 +0000

The first chunk proves an LLM response has begun. It does not prove the answer finished, and neither does it prove that your application can use it.

Imagine a support assistant helping a customer reconnect an integration. The first words arrive quickly: “Let’s get this working again.” Then come numbered steps. Check the connection. Open the integration settings. Reauthorize the account.

Halfway through the next instruction, the answer stops:

“Before you reconnect, make sure you…”

Make sure you what?

The UI has already displayed text that looks useful. In this example, the request began with HTTP 200. The customer sees something shaped like an answer, while the application has no confirmed completion.

Now the app must decide what it received: an answer, an interrupted answer, or something that failed before becoming usable. That decision needs evidence. A spinner disappearing is a surprisingly weak definition of success.

Streaming means the API sends pieces of an LLM response incrementally, while generation continues. With HTTP streaming over server sent events (SSE), the client can process those events and display text without waiting for the entire response. That improves perceived responsiveness: the customer starts reading instead of staring at an empty chat bubble. It does not make the early pieces a completion signal.

For our support assistant, “Let’s get this working again” is evidence of progress. It says nothing about whether the final instruction will arrive. Keep two concepts separate: content becoming available and the response reaching a confirmed terminal state.

Also distinguish text deltas from other events. A stream may carry lifecycle events, tool argument fragments, usage information, pings, and errors. A parser that only extracts displayable text can miss the evidence needed to decide whether the request succeeded. Claude’s streaming protocol explicitly includes these different event types.

The support answer that stopped halfway could have several explanations.

A connection interruption. A read error, timeout, or unexpected end before the required terminal signal leaves completion unconfirmed. Preserve the fragment and investigate the transport path; missing evidence alone does not identify which component caused the interruption.
An error inside an established stream. Claude, for example, can emit an error event containing overloaded_error. Your handler must inspect stream events as well as the initial HTTP status.
An output limit. Generation can stop because it exhausted its allowed output budget. Claude reports max_tokens; its documentation recommends raising the limit or continuing the response. Retrying unchanged does not address that limit.

Each case needs its own fix; one generic “LLM failed” bucket hides the difference.

Provider protocols also express completion differently. OpenAI’s Responses API exposes response.completed and response.incomplete events. Claude sends a final message_stop, with the stop reason supplied in message_delta. A terminal event therefore needs interpretation: Claude can finish its stream while reporting that generation hit max_tokens. The protocol ended; the support instructions may not have.

“Finished” hides three separate questions:

Did the transport end cleanly?
Did the provider report an acceptable generation outcome?
Did the accumulated content satisfy the application’s contract?

Treat them as three separate checks. Inferock’s methodology likewise considers stream completion, finish reason, delivered content, and structural validation when assessing broken output.

A clean connection close answers only the transport question. The provider’s terminal event and stop information answer the generation question. The application still owns the usability question.

For the support assistant, a declared contract might require a troubleshooting result containing instructions and a final verification step. If the model returns structured data, a missing closing JSON brace makes failure obvious. But a closing brace only proves syntactic closure: the object could still omit a required field.

Tool arguments make the distinction sharper. Claude streams tool inputs as partial JSON strings, which must be accumulated and parsed. Parsing successfully still does not establish that the arguments satisfy your tool’s schema.

For plain support prose, completeness is harder to prove. Define checks your application can actually enforce; do not pretend a period at the end establishes that every necessary instruction arrived.

The accidental promotion happens when rendering and success share the same handler:

on_text_delta: append_to_ui(delta)
on_connection_close: mark_answer_complete()

That second line assumes what it needs to establish.

Use an explicit partial state instead. Our interrupted assistant answer can remain visible, labeled “Response interrupted,” without becoming the final answer in conversation history or a downstream workflow. Rendering is provisional; committing is a separate decision.

Keep streaming. Just do not let the UI’s enthusiasm write your success criteria. Show progress promptly. Promote it deliberately.

When the support answer stops, “it cut off” is a symptom report. A useful call record preserves enough evidence to distinguish possible failure classes without inventing a cause. Inferock’s measurement approach starts with observable request, response, timing, usage, and outcome evidence.

For your application’s own record, keep:

Request identity: an application request ID and any available provider request ID.
Execution path: provider, API route, requested model, and attempt identity.
Timing: request start, first content chunk, last content chunk, and terminal event or error time.
Delivered content: what was accumulated at the layer you observed, under appropriate redaction and retention rules.
Provider outcome: terminal event, finish or stop reason, and any details the provider gives about an incomplete response.
Failure evidence: stream error, transport exception, timeout, or observed cancellation.
Application verdict: validation passed, failed, or never ran, with the failed check recorded.

These are proposed application fields; whether a gateway exposes all of them varies. Inferock documents streaming milestones “when available” and explicitly notes that provider fields differ.

Keep client cancellation separate from upstream failure when the evidence supports that distinction. If the customer presses Stop, record it. If your server reaches its own deadline, record that too. Otherwise, “provider interrupted” can become a convenient label for your own abort.

Finally, record where observation occurred. Gateway receipt is not proof of browser delivery. That boundary matters when comparing the call record with what the customer actually saw.

Inferock’s gateway documentation states that successful calls, streaming or not, preserve the upstream HTTP status and provider response content. The gateway does not replace the answer with a measurement envelope of its own.

Its first call workflow has you supply a request ID, then find the corresponding measured call in Calls. Call details can include provider, model, request ID, attempt and failure class, tokens, cost, timing, linked findings, and redacted payload evidence when those fields are available.

For our interrupted support answer, that gives engineers a concrete starting point: locate the request, inspect the recorded outcome, and compare it with the application’s interrupted state.

Inferock’s methodology for broken output considers delivered content, finish reason, stream completion, output usage, JSON parsing, and schema validation. It describes checks for output that is malformed, truncated, empty, or invalid under a declared contract.

The boundary matters. A call record helps investigate an interruption; it does not expose every internal provider cause or discover every requirement your support workflow forgot to declare. Inferock explicitly limits structural checks: they do not establish factual correctness, and tool argument validation needs a declared schema or objective contract.

Make the application’s rule explicit: chunks are provisional; success requires an acceptable provider outcome and application validation. That follows the distinction between provider stop information and checks against a declared response contract.

A practical sequence is:

Display incoming text, marked as still in progress.
Accumulate content while inspecting lifecycle and error events.
Require the expected terminal signal and interpret its outcome.
Validate the assembled result before committing success.
If either check fails, preserve the partial state and request ID.

Back in the support chat, “Before you reconnect, make sure you…” should not quietly become a completed answer. Keep it visible as an interruption, with a recovery path.

The assistant started helping. Your application still has to establish whether it finished.

The Retry Worked. What Broke the First Time?

Bharath Koneti — Fri, 02 Oct 2026 13:31:16 +0000

How we use Inferock Watch to make failed inference calls easier to investigate—even when the application recovers.

Imagine your application asks a model to summarize a document. The first attempt fails. Your retry logic kicks in, the second attempt returns an answer, and the user gets their summary.

The recovery worked. But when someone asks why the first attempt failed, your logs offer a timestamp, a generic exception, and the comforting knowledge that it worked eventually.

That leaves you with a decision to make and very little to base it on. Should you adjust a timeout? Change the retry policy? Investigate the provider? Leave everything alone?

“Probably transient” is doing an impressive amount of engineering here.

We built Inferock Watch to give teams more evidence for that investigation. Watch measures inference calls routed through your existing provider accounts and lets you inspect the recorded call and its linked findings. The problem we want to address is specific: recovering from a failed request should not leave you guessing about what happened to it.

Start with the two attempts in that document-summary example. They belong to one application task, but they are separate calls. If your application keeps only the final result, the successful response can obscure the failed attempt that preceded it.

That matters even when the user receives an answer. A recurring failure can add waiting time to otherwise successful tasks. It can also train a team to accept retries as normal before anyone has established what keeps triggering them. Increasing the retry count may improve completion rates, but it does not explain the underlying behavior.

The first step is to make each attempt findable. When routing a request through Inferock, supply a unique x-request-id and record it in your application logs. If the application retries, give that attempt its own request ID and associate both IDs with the same application task.

Your application keeps the task-to-attempt association; Inferock records each routed call under its request ID. Together, those records let you inspect the failed attempt and its retry without mistaking the final answer for the full history of the task.

With Watch, you connect your provider account and send calls through the Inferock gateway using an Inferock key. Successful calls preserve the upstream HTTP status and provider response content. Your application receives the model response, while measurement happens alongside the request flow. The first-call guide covers the request ID and gateway behavior.

Now return to the failed attempt. In the development app’s Calls view, you can look it up using its request ID. The documented record includes provider and model, attempt information, failure class, token usage, cost, timing, and linked findings. Redacted payload evidence is available where present. The Proof view provides finding evidence and proof packets. These views are described in Receipts + ledger.

You can compare that record with what your application observed. Did your client report a timeout? What timing did the gateway record? Is there a failure classification or a linked finding to examine? What evidence is available, and what is still missing?

Those questions produce a more useful investigation than sending the same prompt again and hoping the failure is feeling cooperative.

They also help keep the diagnosis honest. Suppose your application rejected a response because a required field was missing. That application validation result belongs in your logs. A gateway record alone cannot establish every requirement your product has. You need both the call evidence and the application’s reason for rejecting the answer before deciding what to change.

There is another boundary worth checking: whether the request reached a provider at all. Inferock applies admission limits after authentication and before calling the provider. If a request is rejected at that stage, it does not create a provider-loss measurement. For example, a gateway 429 caused by an account limit needs to be investigated using its reason and reset guidance. Blaming the model would send you in the wrong direction. The Limits guide documents that behavior.

None of this makes a retry wrong. Recovery is useful. We want the user to get their summary. We also want the engineer investigating the incident to have enough evidence to choose a sensible next step.

That might mean fixing request construction, changing how the application handles a response, respecting a limit reset, or taking a documented provider failure to support. Sometimes the available evidence will still be incomplete. An unresolved question is a better starting point than a confident diagnosis built around the last successful attempt.

This is where transparency becomes useful in practice. You can inspect the recorded call, follow a finding to its evidence, and compare it with your own application logs. That gives the team something concrete to review when deciding whether a recurring failure needs a code change or further investigation.

If this sounds familiar, start with Inferock Watch setup and add request IDs to the calls you route through it. The next time a retry succeeds, you should be able to investigate the attempt that made it necessary.

Why AI Providers Shouldn’t Grade Their Own Bills

Bharath Koneti — Wed, 30 Sep 2026 18:12:25 +0000

Why the future of AI routing is an independent layer that credits back what providers break.

Let’s look at a bill that looks completely normal.

Your app asks a model to return a JSON object. The model starts well enough, gets halfway through a field, and stops. Your parser has to respond. Your app retries. The second attempt works, the user never sees the first one, but the dashboard does and it records two API calls. Somewhere in the usage column, the broken attempt sits quietly beside the useful one.

There may be no billing error here. Tokens were processed and generated. The meter counted them. But you paid for a response your software could not use, then paid again to finish the job.

That distinction is the point. A provider can bill correctly under its terms while delivering a failed result for your application. We need to account for both facts. Right now, the industry counts tokens precisely and broken work rather poorly.

The bill knows what ran. Your product cares what arrived.

Compute has always charged for resources rather than outcomes. An EC2 instance bills while it is running, even if your application on it crashes or sits idle. That bargain is clear: you rent the machine; you own the code. AWS describes the charge in terms of the instance’s running state.

With a model API, you control much less of the execution path. The provider decides how to run the model, when to stop it, what usage to report, and how to price that usage. You buy input and output tokens. Your app needs an answer that meets its request. Those are different units.

If you ask for prose, a cut off paragraph may be mildly annoying. If you ask for a tool call with a required account_id, a response ending after “account_” is a failed transaction. The tokens on the invoice are real and so is the failure.

The cause matters.

Did the model hit an output limit you set? Did the provider end the stream? Did your client stop reading? Each calls for a different remedy, but a month-end invoice cannot tell you which happened. Engineering sees retries, finance sees spend, and support sees a user error. Those should be three views of the same event, tied to its charge.

The company holding the meter also writes the rules

Provider billing teams are not sitting around plotting how to collect forty cents from your malformed JSON. That would be a terrible if true but no.

The structural issue is simpler. The provider runs the service, reports usage, publishes the billing rules, and often holds the most detailed account of what happened inside its system. You get an API response, your own logs if you kept them, and are delivered an invoice later. When those records disagree, the provider’s record tends to be the starting point for the conversation.

Even when everyone acts in good faith, customers need their own receipt. What was requested, what arrived, and what was charged.

To be fair, providers do expose useful signals. OpenAI’s response events can include an incomplete status, an incomplete reason, and token usage details. Google’s Gemini API exposes separate usage metadata for candidate, thinking, and cached tokens. These are valuable fields, and they make independent checking possible in some cases. They do not, by themselves, tell you whether your application received a usable answer or whether a given failure qualifies for a refund. OpenAI’s streaming reference and Google’s token documentation show how much nuance is already present in the raw records.

We want customers to have a second ledger. It should preserve the provider’s usage report and check it against delivery, completion, pricing, and the result that the app expected. A modest, verifiable finding beats a dramatic refund estimate made from guesswork.

There is an important distinction here. A billing error means the charge conflicts with measurable usage, a rate, a discount, or a published rule. A service failure means the call did not produce the result your application needed. A single event can be both. It can also be only one of them. The accounting should keep those possibilities separate.

What actually gets lost between the request and the invoice

The failure modes are mundane, which makes them easy to bury in ordinary traffic.

An empty response with billed usage. Zero visible characters alone does not prove the meter is wrong, models may use billed reasoning tokens, return a tool call, or represent a refusal separately. The question is whether the call delivered anything usable under the agreed contract after those token classes and output types are accounted for.

A truncated response. The model returns most of a JSON object, then stops before the closing braces. If your output cap was too low, fix your configuration. If the upstream stream ended unexpectedly, investigate the route. Either way, the receipt needs the finish reason, bytes delivered, usage, and schema result to show what the failed call cost.

A retry cascade. An agent hits an error, retries, times out, retries again, and finally succeeds. The customer sees one task; the provider may see several attempts. We would not claim every error is billed. We do want to know which attempts were processed and charged, and what the whole task cost. The successful final call is a flattering but incomplete number.

A token or cache mismatch. Visible text is not always the full token count; hidden reasoning and other token classes can appear in usage. A suspected cache hit is not a confirmed discount, either. Compare the provider’s usage categories, applicable model rates, and observed charge before calling anything an overcharge. Sometimes the result is a strong billing candidate. Sometimes it is a question for review.

Here is a small example.

Record only attempt 3 and you see success. Record all 3 and you see what success cost.

OpenRouter has already shown that a clearer remedy is possible for one narrow category: it announced zero charge for responses with zero completion tokens under its stated conditions. That is a useful precedent. It also illustrates why the details matter. Zero completion tokens is a specific rule; it does not automatically cover every partial stream, invalid schema, retry, or answer a customer cannot use.

If you have seen a failure mode we missed, bring it to inferock-bench on GitHub. The measurement rules are public, and issues help us make the receipts more accurate.

Why the receipt belongs in the routing path

Many teams send traffic through multiple providers, frameworks, and fallback routes. Then finance asks why inference spend rose and gets four dashboards with four definitions of “request.” A routing layer gives selected traffic one accountable path and records the call while it happens.

You cannot reconstruct a broken stream from a month-end invoice. You need the requested output, upstream route, bytes delivered, finish state, usage, and retries. A receipt created in the request path keeps those facts together and connects multiple attempts to the task that caused them.

That is the idea behind Inferock. You send selected inference traffic through one accountable path; the path keeps route, timing, response, usage, and charge evidence attached to each observed call. It can report objective failures such as empty or malformed output when the evidence supports them, and it can keep weaker findings in a separate review category. The methodology describes the measurements and their limits.

There are two ways to use that accountability. The difference matters.

With Bring Your Own Keys(BYOK), you keep your provider accounts and pay those provider bills directly. Inferock measures traffic and gives you receipts and loss reporting. If the evidence points to a provider billing issue, you have a better record to investigate or dispute it.

With Managed inference, Inferock operates the upstream provider relationship for approved traffic. That creates room for a service credit when an eligible, objective failure meets the agreed terms. See current pricing and mode description.

The rule starts with evidence. Did the call pass through the measured path? Is there a priced usage or charge record? What failed, and was the cause upstream, client-side, or a configuration limit? Does it meet the service terms? The receipt should show the answer and any gap in the evidence.

A new standard for accountable inference

There is a tempting version of this argument where every ugly model response is called an overcharge and every provider is cast as a villain. It would make for a loud post. It would also make it harder for anyone serious to build a fair system around the problem.

Here is the standard we are willing to defend.

Every material model call should have a customer-readable receipt. That receipt should connect the request, upstream route, delivered response, completion state, usage, applicable rate, observed charge, and any retry to one logical operation. It should say what can be verified and what cannot. It should separate a provider-recognized billing error from a service failure measured by an independent standard.

We also want teams to stop accepting “the API returned 200” as proof that the job was done. A valid HTTP status is useful. A valid result for your customer is the thing you actually needed. The gap between them has a cost, and that cost should be measurable.

We’re building Inferock around that conviction. You should be able to point to a call and say: this is what we asked for, this is what arrived, this is what the meter recorded, this is what failed, and this is what happens next. If the evidence is incomplete, we should say that in the same place. If a Managed failure qualifies for a credit, the credit should be visible there too.

Providers will keep improving their meters. They should. We want them to publish clearer rules for partial streams, failed calls, hidden token classes, and cache charges. Their own credit policies should become easier to understand and use. But the only record of a failed purchase should not come from the company that sold it.

That is the standard we want to see across AI infrastructure: a bill you can check, a failure you can prove, and a remedy whose limits are clear before you need it.

If you want to see this on your own traffic, request an Inferock invite.

If your model spend is growing, you already have enough uncertainty in the system. Your invoice should not be another one.