The operational constraint changes the design: a routing decision is not proof that a model vendor served a request. Record the vendor after the upstream attempt reaches a terminal outcome, beside an internal request ID, tenant ID, model alias, outcome, and timing. On a developer-tools platform that issues and revokes one scoped key per tenant, store the key ID. Never store the secret.
TL;DR: emit one structured audit event for every completed attempt, including failures. Take the vendor identifier from the adapter that made the call, not from routing configuration. Later, join quality scores by request ID and group them by vendor ID. This produces comparison metrics without confusing routing intent with delivery evidence.
How should you record which model vendor served a request?
The before model looks plausible. A router logs selectedVendor, invokes an adapter, and returns. It breaks as evidence under retries: vendor A can time out, vendor B can answer, and the first log still says A. Recording successes alone introduces another blind spot because later comparisons omit availability failures.
The after model separates two records. An access event says which tenant-scoped credential identity was authorized. A request event says which adapter attempt reached success, error, or timeout. An internal operation ID can connect retries, while each attempt retains its own request ID.
That distinction matters.
Picture the path. The tenant key ID enters the authorization boundary. A request ID crosses the router. The adapter supplies a controlled vendor ID. The audit sink accepts a terminal event. Later, an evaluator attaches a score to that request ID.
Keep prompts and responses out of this event unless the evaluation design truly requires them. Their access and retention rules may differ from operational metadata. Short record. Clear purpose.
OWASP's secrets guidance recommends identifying secrets, establishing rotation criteria, revoking credentials, and logging who requested and used them. A stable tenant key ID gives those lifecycle actions a subject without exposing credential material.
| Field | Question it answers | Keep out |
|---|---|---|
requestId |
Which attempt produced this event? | Prompt text |
operationId |
Which attempts belong to one user operation? | Credential value |
tenantKeyId |
Which credential identity was authorized? | Key fragments |
vendorId |
Which adapter attempted or completed the call? | Derived hostnames |
A copyable TypeScript boundary
Make provenance part of the adapter contract. Do not reconstruct it later from a hostname, model alias, or environment variable. This union is intentionally closed so misspellings cannot silently create a new reporting bucket.
type VendorId = "vendor-a" | "vendor-b" | "vendor-c";
type Outcome = "success" | "error" | "timeout";
type ServedResult = {
vendorId: VendorId;
upstreamRequestId?: string;
output: string;
};
type AuditEvent = {
eventVersion: 1;
operationId: string;
requestId: string;
tenantId: string;
tenantKeyId: string;
modelAlias: string;
vendorId: VendorId;
outcome: Outcome;
startedAt: string;
finishedAt: string;
durationMs: number;
upstreamRequestId?: string;
};
interface ModelAdapter {
readonly vendorId: VendorId;
generate(input: string, signal: AbortSignal): Promise<ServedResult>;
}
interface AuditSink {
append(event: AuditEvent): Promise<void>;
}
Measure a single attempt in one place. performance.now() supplies elapsed time; ISO timestamps support correlation across systems. A successful response carries the vendor that answered. The adapter identity remains available when an attempt fails before returning a response.
import { performance } from "node:perf_hooks";
import { randomUUID } from "node:crypto";
type AttemptInput = {
operationId: string;
tenantId: string;
tenantKeyId: string;
modelAlias: string;
prompt: string;
};
async function runAttempt(
input: AttemptInput,
adapter: ModelAdapter,
audit: AuditSink,
signal: AbortSignal,
): Promise<{ requestId: string; output: string }> {
const requestId = randomUUID();
const startedAt = new Date();
const start = performance.now();
try {
const result = await adapter.generate(input.prompt, signal);
await audit.append({
eventVersion: 1,
operationId: input.operationId,
requestId,
tenantId: input.tenantId,
tenantKeyId: input.tenantKeyId,
modelAlias: input.modelAlias,
vendorId: result.vendorId,
outcome: "success",
startedAt: startedAt.toISOString(),
finishedAt: new Date().toISOString(),
durationMs: Math.round(performance.now() - start),
upstreamRequestId: result.upstreamRequestId,
});
return { requestId, output: result.output };
} catch (error) {
await audit.append({
eventVersion: 1,
operationId: input.operationId,
requestId,
tenantId: input.tenantId,
tenantKeyId: input.tenantKeyId,
modelAlias: input.modelAlias,
vendorId: adapter.vendorId,
outcome: signal.aborted ? "timeout" : "error",
startedAt: startedAt.toISOString(),
finishedAt: new Date().toISOString(),
durationMs: Math.round(performance.now() - start),
});
throw error;
}
}
Awaiting audit.append adds the sink to request latency, but it establishes a crisp rule: the application does not report success before recording its event. A buffered sink shortens the synchronous path and can tolerate a brief storage interruption. It also creates work. The process must drain safely during shutdown, bound its queue, and expose rejected writes, queue depth, and oldest-event age.
Here is the retry case in concrete terms. An operation begins with one operationId. Attempt one gets a fresh requestId, records vendor A, and ends as timeout; attempt two gets another request ID, records vendor B, and ends as success. The evaluator scores only the output from attempt two, while availability reporting counts both terminal events. If the implementation instead updates one row from A to B, the final row looks tidy but destroys the timeout evidence. If it logs the routing choice before the call and never writes the outcome, it preserves intent but cannot prove service. Two attempts, two events, one operation. That shape is slightly more expensive to query than a single mutable row, yet it preserves the exact distinction a vendor-risk review needs.
Test three paths before deployment: direct success, timeout followed by fallback, and audit-write failure. Assert that serialized events contain neither the prompt nor the secret. Also verify that the fallback creates a second attempt under the same operation ID. Otherwise a storage key based on one request ID can erase the history you meant to preserve.
What should quality metrics join on?
Join each evaluator result to the internal request ID, then aggregate by vendor ID and model alias. Keep the evaluation record separate. Scores may arrive later, and a changed rubric may recompute them; an evaluationVersion field makes that history explainable.
Do not join quality results on the tenant key ID. One key can authorize many requests, and rotation can replace a key while the tenant stays the same. The key ID answers an access question: which credential identity was used? The request ID answers an execution question: which output was scored?
Every comparison should carry its denominator. A mean quality score without attempt and failure counts can make a vendor with unscored failures appear stronger. Compare the same model alias, evaluation version, workload slice, and time window. Those dimensions do not remove selection bias, but they expose enough of it for review.
Isn't the application log enough?
Usually, no. Free-form logs help diagnose behavior, but message formats change and operational retention may not match audit needs. A versioned event can travel through the same logging pipeline; the distinction is its documented schema, controlled identifiers, and monitored delivery.
Do not claim tamper resistance unless the complete storage path provides it. An insert-only application interface is a useful constraint, not a security guarantee. Restrict writers, separate read access, define retention, test restoration, and alert on failed appends. For credentials, distinguish issuance, use, rotation, and revocation so a reviewer can reconstruct lifecycle events without retrieving a secret.
A useful alert is intentionally dull: page on sustained audit-write failures and warn when buffered-event age exceeds the recovery objective. Dashboard request outcomes by vendor ID. Keep output scoring in the evaluation system because operational health and response quality answer different questions.
The main limitation is latency. Synchronous audit writing is not suitable when the request path cannot tolerate a storage dependency; a durable buffer is the better fit there, with extra recovery work as the trade-off. This event is also insufficient for semantic quality analysis by itself because it intentionally excludes the prompt and response. Put evaluation inputs behind a separate access and retention policy instead of expanding a broadly readable operations record. Finally, an application event cannot establish tamper resistance on its own. That requires storage and access controls outside this TypeScript function.
Why not derive the vendor during analysis?
Because the clues can change. A model alias may route to multiple adapters. A gateway can hide a downstream hostname. Configuration shows what should have happened, not which attempt completed. Capturing a controlled ID at the adapter boundary preserves the fact while the code still knows it.
Define what vendorId means for your comparison. If one organization operates the gateway and another supplies the model, use separate gatewayId and modelProviderId fields rather than overloading one string. Collect the downstream identity only when it is reliably available and permitted. An explicit unknown value in a later schema is more honest than guessed provenance.
Start with one rule: every terminal attempt produces one event, and no event contains credential material. Schema versions, evaluator versions, access controls, retention tests, and delivery alerts then turn a small Node.js wrapper into evidence suitable for vendor-risk review.
Version 1 can stay small.
Top comments (0)