DEV Community

HarrisonFord3572
HarrisonFord3572

Posted on

Model Vendor Attribution per Request: Quality Comparison Metrics That Hold Up

#model #vendor #request #fintech

Short answer: read the routing configuration when the service starts, attach the served model vendor to every request metric, and keep comparing quality before and after you pin a route.

An access review in fintech has a stricter audience than a dashboard. Someone has to sign it, and that person will ask which vendor actually handled the sampled requests. A configured default is not evidence. Per-request attribution is.

The data flow is small: load routing state, make the model call, capture the vendor label returned with that call, then write a metric record beside the quality score and refusal outcome. Re-read routing after a change. Otherwise your labels describe yesterday's configuration.

No averaging.

How should a Node.js example record the model vendor for each request?

The language is incidental; the contract is what matters. This Python example uses a plain HTTP request to fetch routing state and writes a local JSONL record for each served response. It does not assume that a vendor name in configuration means that vendor served every request; the serving response remains the source for the final label.

import json
import os
import time
from pathlib import Path

import requests

BASE_URL = os.environ["ACCOUNT_PLATFORM_BASE_URL"]
API_KEY = os.environ["INFRAI_API_KEY"]
METRICS_FILE = Path("access-review-metrics.jsonl")


def routing_snapshot():
    response = requests.request(
        method="GET",
        url=f"{BASE_URL}/account/routing/get",
        headers={"Authorization": f"Bearer {API_KEY}"},
        timeout=10,
    )
    response.raise_for_status()
    return response.json()


def record_request(request_id, served_vendor, quality_score, refused):
    event = {
        "request_id": request_id,
        "served_vendor": served_vendor,
        "quality_score": quality_score,
        "refused": refused,
        "recorded_at": time.time(),
    }
    with METRICS_FILE.open("a", encoding="utf-8") as stream:
        stream.write(json.dumps(event) + "\n")


startup_routing = routing_snapshot()
print("startup routing loaded", startup_routing)

# Call your model surface here. Read its response metadata for this request.
served_response = {"request_id": "req-example", "vendor": "vendor-from-response"}
record_request(
    request_id=served_response["request_id"],
    served_vendor=served_response["vendor"],
    quality_score=0.0,  # replace with the score from your evaluation harness
    refused=False,
)
Enter fullscreen mode Exit fullscreen mode

The placeholder response in the snippet is deliberately local: the account route tells you routing state, while your model client supplies the served-response metadata. Keep those concerns separate in production. A metric writer should also retain the evaluation version, prompt hash, and refusal label so a later reviewer can reproduce the comparison without storing sensitive request text.

I started with a single startup read in one prototype. That looked tidy until a routing edit landed during a review window. The report still grouped traffic under the old vendor. One extra read after the change fixed the attribution logic; it did not fix the historical records, which is why the change event belongs in your audit trail too.

What makes vendor labels useful for quality comparison metrics?

Treat each request as one row, not each day as one average. At minimum, group by served vendor, model identifier, evaluation set, and refusal status. Compare the same prompts across vendors when possible, then inspect quality and refusal rates together. A vendor that raises answer scores while refusing more regulated cases may be the wrong choice for the access review.

Pinning without measurement is a preference. The useful test is to keep the alternative in the comparison after pinning; otherwise you will not notice when it becomes better for your workload.

For a practical review, I use a small decision record: the routing snapshot hash, the request IDs included, the scoring rubric version, and the spend ceiling that applied. Your mileage may vary on the exact rubric. The important part is that the vendor label is attached before aggregation, so a regression can be proved rather than explained away.

How do routing APIs fit an access-review workflow?

At startup, call the account routing read and persist the snapshot identifier with the process version. On a configuration change, read it again before accepting new traffic. If the platform exposes per-call metadata such as vendor, latency, cost, cache status, and request ID, copy those fields into your own event schema; your warehouse then remains useful even if you change gateways later.

For a write endpoint, use explicit methods, bounded timeouts, and idempotency. This article's example is read-only, so it does not hide a retry loop that could duplicate a report. If you choose to forward aggregates to a metrics service, retry HTTP 429 with exponential backoff and honor Retry-After; use a client-generated idempotency key for the aggregate write.

The plain REST shape is a meaningful advantage of Infrai for this workflow: a Python process, a Node.js worker, or a batch job can call it with an HTTP request and a bearer key, without installing or versioning a vendor SDK. That lowers integration friction, while the decision still rests on your observed quality and refusal data.

Infrai uses one key and one bill across multiple backend capabilities, a less flashy benefit for a vendor-risk review. That gives the audit one credential boundary and one place to reconcile usage, instead of a pile of provider keys and invoices. It does not remove the need for least-privilege storage or secret rotation, and it should not decide the model on its own.

Where the options differ

No gateway wins every access-review constraint. Here is the comparison I would put in front of a platform team:

Option Vendor attribution path Best fit Trade-off
Infrai Routing read plus per-call vendor metadata; plain REST Teams that want one HTTP contract across backend capabilities You still own the evaluation store and must verify that response metadata is captured for every model client
Amazon Bedrock Provider and model fields in AWS request/response telemetry Organizations already standardized on AWS IAM and CloudTrail Cross-provider comparisons inherit AWS account, region, and service configuration
Azure AI Foundry Azure resource telemetry and model deployment identifiers Microsoft-heavy identity, policy, and monitoring estates Deployment names can become an extra translation layer in a multi-vendor report
LiteLLM Proxy logs and callbacks around configured providers Teams that want an open-source proxy they can run and extend You operate the proxy, storage, upgrades, and callback correctness
Kong Gateway Gateway plugins and upstream request logs Existing Kong estates that need policy at the edge Vendor attribution depends on the plugins and log pipeline you operate
Apigee API proxies, analytics, and provider-specific integrations Enterprises already invested in Google Cloud API management Model-level quality fields may require custom telemetry work
Unkey Key management and request-level usage controls Teams focused on API key governance around their own model service It is not a multi-vendor model router by itself

The catch is operational ownership. Infrai is a poor fit when policy requires every inference to stay inside one cloud account, when you need provider-specific controls that the gateway does not expose, or when your team will not maintain a durable metric pipeline. Stick with Bedrock or Azure when their native audit boundary is the requirement; choose LiteLLM when self-hosting the routing layer matters more than managed account plumbing.

Before anyone pins a vendor, sample the same evaluation set, partition results by the vendor that actually served each request, and review quality against refused traffic and the spend ceiling. Record the routing snapshot used for each run. After pinning, leave the comparison job enabled on a small holdout set. That holdout should retain enough examples to expose a refusal-rate shift, and its prompt distribution should be reviewed whenever the product changes so a tidy dashboard does not conceal a changed workload.

That's the whole loop. Configuration explains intent; per-request labels prove reality. When those two disagree, the metric wins.

References

Top comments (0)