Short answer: write one append-only attribution event for every model attempt, before a logistics charge is finalized. Include the request, shipment, vendor, model, and outcome identifiers; then reconcile retries and token usage from that event stream. This is how a Node.js service can later compare quality without guessing which model vendor served a request.
The business constraint is a prepaid logistics balance. If the balance reaches zero unattended, label generation and exception triage stop at the worst possible time. A meter that cannot explain one invoice line is a billing control failure, not an observability inconvenience.
I once joined usage by a provider trace ID and counted a retry twice. The staging report showed 2.1x the expected calls. Nothing was wrong with the model response; my join key described attempts inconsistently. That incident changed my invariant: an attempt ID is local, immutable, and unique even when an upstream trace is reused.
Architecture decision record: what must be true?
The ledger has five invariants. Every attempt gets a stable attempt_id. The request_id groups attempts caused by one user action. shipment_id selects the billable logistics work. vendor and model capture the serving route at the time of the call, rather than whatever configuration is current during a later report. Finally, status moves through started, succeeded, or failed exactly once per attempt.
Keep prompt content out of this record by default. Store a normalized SHA-256 fingerprint and a retention class; content logs create a separate privacy and access-review problem. Token counts are nullable until a response supplies them. A missing count should be visible as missing, not silently converted to zero.
Here is the decision matrix I use for a prepaid account:
| Design | Attribution accuracy | Failure boundary | Use it when |
|---|---|---|---|
| Transactional usage ledger | High; joins to shipment and invoice IDs | The write path needs durable storage | Charges must be explainable |
| Provider dashboard export | Medium; retries and delayed exports are hard to join | Export timing can miss a balance alert | Trend checks are enough |
| Trace-only metadata | Variable; sampling can omit attempts | A sampled trace cannot be a meter | Debugging latency, not billing |
The rejected option is “read the dashboard at month end.” It is valid for a rough capacity review, but it cannot protect a prepaid balance in real time. The meter belongs beside the request path, with a separate alert consumer.
How can a Node.js service record vendor and model per request for quality metrics?
The application can be written in Node.js while the persistence contract stays language-neutral. Record started before sending the request, update the same row on a response, and make the update idempotent. A timeout is an outcome to reconcile, not permission to create a second row.
from dataclasses import dataclass
from datetime import datetime, timezone
import hashlib
@dataclass(frozen=True)
class ModelAttempt:
attempt_id: str
request_id: str
shipment_id: str
vendor: str
model: str
status: str
prompt_sha256: str
input_tokens: int | None
output_tokens: int | None
created_at: str
def fingerprint(normalized_prompt: str) -> str:
return hashlib.sha256(normalized_prompt.encode("utf-8")).hexdigest()
def upsert_attempt(store, attempt: ModelAttempt) -> None:
store.execute(
"""INSERT INTO model_attempts
(attempt_id, request_id, shipment_id, vendor, model, status,
prompt_sha256, input_tokens, output_tokens, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(attempt_id) DO UPDATE SET
status=excluded.status,
input_tokens=excluded.input_tokens,
output_tokens=excluded.output_tokens""",
(attempt.attempt_id, attempt.request_id, attempt.shipment_id,
attempt.vendor, attempt.model, attempt.status,
attempt.prompt_sha256, attempt.input_tokens, attempt.output_tokens,
attempt.created_at),
)
started = ModelAttempt(
attempt_id="att_8f31",
request_id="req_204",
shipment_id="ship_7712",
vendor="routing-provider-a",
model="quality-model-v3",
status="started",
prompt_sha256=fingerprint("classify delayed pallet"),
input_tokens=None,
output_tokens=None,
created_at=datetime.now(timezone.utc).isoformat(),
)
upsert_attempt(db, started)
In a real Node.js process, pass the selected vendor and model into this function at dispatch time. Do not infer them from a response after a fallback: the event should preserve the route that actually received the request. Keep the code path boring. Billing systems reward boring.
What should quality, spend, and balance reports join on?
Join quality labels to request_id when one user action can retry, and join spend to attempt_id when each attempt is billable. Group both by vendor and model as they were recorded, then slice by shipment lane, warehouse, and prompt version. A model score without its prompt revision can make a regression look like vendor drift.
For the prepaid balance, consume successful usage events and reserve a small amount when an attempt starts. Release the reservation on a definitive failure. Unknown outcomes stay in a reconciliation queue until the provider receipt or a timeout policy resolves them. Never treat a network timeout as free usage.
The alert should name the remaining balance, the reservation total, and the oldest unresolved attempt. A single “low balance” gauge is too vague for an on-call engineer who has to decide whether to pause a route or investigate duplicate accounting.
Short retention helps.
Testing and operational boundaries
The failure chain worth rehearsing is mundane: a dispatch worker reserves 400 tokens, the process dies after the provider accepts the request, and a replacement worker retries with a new attempt ID. If the first receipt arrives after the retry, a naive consumer charges both attempts and leaves the prepaid balance 800 tokens lower. The repair is a reconciliation rule, not a clever timeout: retain the original request_id, mark the late receipt against its existing attempt_id, and let a policy decide whether the second attempt is billable. Store the provider receipt reference as evidence, but never use it as your only local key because fallback vendors may issue unrelated receipt formats. In the same test, advance the clock across the reservation lease, deliver events out of order, and verify that the balance cannot become negative. I also run the scenario with two workers and an interrupted database transaction; the expected result is one committed state transition and one alert, not two compensating writes that hide the race.
Use a fake model client and a temporary database to test: fallback from vendor A to vendor B, duplicate delivery of the same response, a process crash after started, a delayed token receipt, and two workers racing to finalize one attempt. Assert that one attempt_id produces one billable result and that the vendor/model pair never changes during an update.
Emit counters for started, succeeded, failed, and unknown; histogram the time spent in unknown. Sample traces for debugging, but export the attribution event unsampled. Keep secrets in a managed store and grant the writer only the minimum database permission; OWASP's guidance is a useful baseline for that boundary.
The catch is storage and governance. An append-only ledger costs more than a dashboard export, and shipment identifiers may still be sensitive even when prompts are hashed. This approach is not suitable when the team cannot define retention or provide durable storage; stick with a provider export for exploratory metrics, and accept that it cannot enforce a live balance ceiling.
I'm not sure a single quality score will survive every lane or language. Your mileage may vary. Preserve the raw dimensions so the next comparison can answer that question instead of rewriting history.
Top comments (0)