DEV Community

nathanielbrooks0360
nathanielbrooks0360

Posted on

Write and Read Back Verification for Provider Routing Preference One Capability at a Time

#routing #reliability #billing #observability

Rolling a platform onto a new provider routing preference gives you two honest options, and the fast one is the wrong one: write the policy for every capability at once and watch the aggregate error rate, or write it for a single capability, read the effective config back, prove the change with one real request, and only then move to the next. Use the second. The first optimizes for calendar time, which nobody pages you about, while the second optimizes for attribution accuracy — which is the thing that keeps a prepaid balance from reaching zero unattended on a Saturday, when the only humans awake are the ones teaching a weekend cohort.

A control-plane write is a request, not a result.

That distinction sounds pedantic until you've reconciled a month where the routing change reported success and the invoice says a different upstream served most of the traffic. Configuration propagates through caches, staged rollouts, regional replicas and fallback chains, and each of those layers is permitted to disagree with your intent for a while. The write returns 200 with a revision id. What the data plane actually does with that revision is a separate fact, and it's the only fact the billing pipeline cares about.

Our student-facing API is Node.js; the verifier is a separate Go binary on a schedule. That split is deliberate, because a checker that shares the application's process, config cache and credentials will cheerfully confirm its own assumptions.

Why does a provider routing preference need a write test and read back loop?

There are three states in play and teams routinely collapse them into one. Desired is what you wrote. Effective is what the control plane admits it is holding for that capability right now. Observed is which upstream actually answered a request and got charged for it. A read-back of the effective config closes the gap between the first two, and only a probe — one real, billable request, tagged as synthetic — closes the gap to the third.

Attribution accuracy is the axis I care about most here, and it's a billing property before it's an engineering one. We run a prepaid balance per capability group, forecast days-to-zero from the trailing burn rate, and alert when the forecast drops under a floor of seven days. If a transcription request is attributed to the provider you intended rather than the one that served it after a fallback, the forecast is not noisy — it's confident and wrong, which is much worse, because a wrong-but-confident forecast suppresses the alert that would have bought you a week of lead time. Capacity planning against bad attribution is just arithmetic on fiction. I'd rather have no forecast than a smooth one built on it.

The Node service never re-derives routing at request time. It loads a small policy artifact that the verifier is the only writer of, and it fails closed when the artifact is stale:

{
  "capability": "speech_to_text",
  "order": ["primary_stt", "secondary_stt"],
  "revision": "rev_8f21c4",
  "verified_at": "2026-09-11T02:14:07Z",
  "max_age_seconds": 5400,
  "on_stale": "reject_writes"
}
Enter fullscreen mode Exit fullscreen mode

The week the balance forecast was smooth and wrong

The failure mode I keep meeting in reviews has the same shape every time. Someone applies a preference change across four capabilities in one commit because they're all "the same kind of change". Three take effect. The fourth sits behind a provider-side fallback chain that only engages on retry, so the happy path looks exactly as intended, dashboards stay green, and the retry path — perhaps two percent of calls, perhaps twelve during an upstream slowdown — quietly bills a different account.

Nothing breaks. That's the trap.

Error rate is flat, latency is flat, the SLO burn rate never moves, and the first signal is a prepaid account crossing its floor days later with no matching rise in traffic. I assumed for a long time that reading the effective config back was sufficient, and it isn't: effective config describes the router's intent, not the upstream's behaviour under retry. The invariant we ended up writing down is short. A routing preference is applied when, and only when, a request tagged to that capability comes back naming the provider you asked for, and the usage record for that request lands in the account you expect. Two facts, both observed, one capability at a time.

Verifying one capability at a time in a job you can run in CI

The loop is write, poll the effective config until the revision matches, send one probe, then compare what answered against what you asked for. Deadlines everywhere, and a non-zero exit so the same binary works as a deploy gate and as a synthetic check:

package main

import (
    "context"
    "encoding/json"
    "errors"
    "fmt"
    "net/http"
    "os"
    "time"
)

type Preference struct {
    Capability string   `json:"capability"`
    Order      []string `json:"order"`
    Revision   string   `json:"revision"`
}

type Probe struct {
    ServedBy  string `json:"served_by"`
    AccountID string `json:"account_id"`
    Units     int    `json:"units"`
}

// ApplyAndVerify writes one capability's preference, waits for the control plane
// to report that revision as effective, then spends one real request to find out
// who answers. Any disagreement is a hard stop: the caller keeps the old policy.
func ApplyAndVerify(ctx context.Context, c *Client, want Preference) error {
    rev, err := c.PutPreference(ctx, want)
    if err != nil {
        return fmt.Errorf("write preference %s: %w", want.Capability, err)
    }

    deadline := time.Now().Add(90 * time.Second)
    backoff := 500 * time.Millisecond
    for {
        eff, err := c.EffectivePreference(ctx, want.Capability)
        if err == nil && eff.Revision == rev {
            break
        }
        if time.Now().After(deadline) {
            return errors.New("effective config never reached revision " + rev)
        }
        time.Sleep(backoff)
        if backoff < 8*time.Second {
            backoff *= 2
        }
    }

    p, err := c.Probe(ctx, want.Capability, "synthetic-routing-check")
    if err != nil {
        return fmt.Errorf("probe %s: %w", want.Capability, err)
    }
    if p.ServedBy != want.Order[0] {
        return fmt.Errorf("capability %s: asked for %s, served by %s",
            want.Capability, want.Order[0], p.ServedBy)
    }
    if p.AccountID != c.ExpectedAccount(want.Capability) {
        return fmt.Errorf("capability %s: usage billed to %s", want.Capability, p.AccountID)
    }
    return json.NewEncoder(os.Stdout).Encode(map[string]any{
        "capability": want.Capability, "revision": rev,
        "served_by": p.ServedBy, "units": p.Units,
    })
}
Enter fullscreen mode Exit fullscreen mode

Three details in there earn their keep. The probe carries a stable idempotency key per revision so a retried verification doesn't double-charge the prepaid account and skew the very burn rate you're trying to measure. The probe is tagged as synthetic, because attribution you pollute with your own health checks is attribution you can't reconcile against an invoice later. And the reconciliation step is deliberately outside this binary: response headers tell you who answered within milliseconds, the provider's usage export tells you who got paid, and those two agree on a lag measured in hours, so the fast check gates the deploy while a nightly job compares probe ledger against exported usage and pages on drift.

Run the whole thing per capability, serially, with a gap between capabilities.

Serial is slower and it's the point — a parallel sweep gives you one timestamp for four changes, and when the burn rate moves next Tuesday you have no way to attribute the move to any one of them.

Buy, build, or borrow the control plane

The buy-vs-build call here comes down to where your attribution record is born, and I'd rank on-call load above feature count:

Approach Attribution comes from On-call load Lock-in Where it stops helping
Provider console only Vendor's own usage report Low High No cross-provider view; reconciliation is per-vendor and manual
Managed API gateway policy Gateway access logs Medium Medium Policy language may not express per-capability preference
Self-hosted routing proxy Your logs plus upstream response headers High Low You own the upgrade treadmill and the metering pipeline
Dedicated metering service Usage events keyed for dedup Medium Medium Needs a stable event schema before it pays for itself

Two data points worth having before you argue about this in a design review. LiteLLM's proxy config expresses fallbacks as an ordered list per model, which is the same shape as a routing preference and makes the read-back question concrete rather than abstract. OpenMeter's event model — subject, timestamp, idempotency key — is roughly the minimum schema that lets you reconcile an internal ledger against a vendor invoice at all; anything less and the nightly job can't tell a duplicate from a retry.

Neither of those removes the loop. They change who operates it.

Where this loop is the wrong investment

If you have one provider per capability, no prepaid balance, and post-paid billing with a credit line, the catch is that all of this machinery buys you almost nothing — stick with the provider's console and a monthly export. The loop earns its cost when prepaid credit, multiple upstreams per capability, and unattended nights are all true at once.

It also doesn't support the case people most want it to. A read-back proves configuration, and a probe proves one request; neither proves that the ten thousand requests between two probes were attributed correctly, because a probe is a sample and the fallback path is a rare event by construction. Sampling can't see rare events reliably, which is why the nightly reconciliation against exported usage is the load-bearing control and the probe is the fast gate in front of it. If your provider's usage export lacks a per-request identifier, the reconciliation degrades to comparing totals, and I'm not sure that's worth building — comparing aggregates finds a drift of ten percent and misses a drift of one, and one percent on a prepaid balance is exactly the kind of slow leak that surfaces as an outage instead of a graph.

Treat the routing preference as configuration with an audit trail, give the verifier short-lived credentials scoped to a single capability rather than the platform-wide key it will otherwise inherit, and keep the probe ledger for as long as you keep invoices.

Sources

Top comments (0)