Per-Capability Provider Routing at the Gateway: Residency Rules and Spend Caps

#architecture #security #api #go

Cap the workload, not the router. If a marketplace needs one enrichment job to stop spending before the invoice arrives, the smallest mechanism that actually does it is a credential scoped to that single workload with a hard budget attached, plus a per-capability routing rule that excludes the vendors that job may never reach. Pinning one provider is the more tempting move — a single vendor id, one line of config, done — and I use it last, because a pin freezes a decision that would otherwise keep improving, and because a pin describes your configuration rather than your traffic.

The order matters. Decide what you can prove, then what you can spend, and only then express a routing preference. Do it the other way around and you'll be reading the invoice to find out what your own system did.

What the enrichment bill is actually made of

Take a marketplace that enriches seller listings at upload time: translate the description into three locales, caption the thumbnail, screen the whole thing for prohibited goods. Three capabilities, three vendor pools, one API gateway sitting in front of all of them. Assume 40,000 listings a day and two model calls per listing — 80,000 calls a day, around 2.4 million a month — and the invoice stops being mysterious. Vendor charges are the bill. The gateway's own CPU is a rounding error against them, and the decision log, at one row of a few hundred bytes per call, costs less to keep for a year than the inference it describes costs in an afternoon.

One term dominates: calls allowed, multiplied by the unit price of whoever served them.

Which is why the interesting control is not the router. Routing changes who gets paid; it does not change whether the call happens at all. A preference is an optimization on the second-largest term in the invoice, while a cap is the only thing that touches the first. So the routing rules in this design exist to honor constraints you are obliged to honor anyway — residency, contractual exclusions, a measured quality gap — and the spending question gets answered somewhere else entirely.

How should a marketplace pin or exclude a provider per capability without losing the audit trail?

Start from default routing and change it only when the reason fits in one sentence you would be willing to defend in review. Routing is set per capability for a good reason: pinning image captioning to one vendor should not freeze the translation model you spent a quarter evaluating.

Prefer exclusion to pinning. An exclusion is subtractive and survives the vendor list changing — when the router adds a provider next month, an exclude rule still means exactly what it meant on the day it was written, while a pin quietly becomes a decision to skip everything that arrived after it. Pins also fail in the audit direction. "We pinned vendor A" is a statement about configuration; what an auditor asks for is a statement about traffic, and the two drift apart the first time a fallback fires. Write the policy per workload, per capability, with the reason inline:

{
  "policy_version": "2026-09-01",
  "workload": "listing-enrichment",
  "capabilities": {
    "text.translate": { "exclude": ["vendor-c"], "regions": ["DE", "IE", "FR"] },
    "image.caption": { "pin": "vendor-a", "regions": ["DE"], "reason": "TICKET-4471, expires 2026-12-01" },
    "moderation.screen": { "regions": ["DE", "IE"] }
  },
  "budget": { "period": "day", "cap_minor_units": 250000, "currency": "EUR", "on_exhaustion": "reject" }
}

The record that matters is written before the call leaves the gateway, not after the response comes back.

// Decision is resolved and persisted before the upstream request is made.
type Decision struct {
    IdempotencyKey string    // one enrichment attempt, one key, supplied by the caller
    Workload       string    // "listing-enrichment"
    Capability     string    // "text.translate"
    KeyFingerprint string    // SHA-256 of the credential; never the credential itself
    Vendor         string    // resolved vendor id
    Region         string    // ISO 3166-1 alpha-2 code where processing happens
    Rule           string    // "exclude:vendor-c" | "pin:vendor-a" | "default"
    PolicyVersion  string
    DecidedAt      time.Time
}

// Resolve applies one capability's policy. Exclusions and the region list are subtractive;
// a pin is an assertion that must still satisfy residency after the subtraction.
func Resolve(p CapabilityPolicy, pool []Vendor, capability string) (Decision, error) {
    allowed := make([]Vendor, 0, len(pool))
    for _, v := range pool {
        if slices.Contains(p.Exclude, v.ID) || !slices.Contains(p.Regions, v.Region) {
            continue
        }
        allowed = append(allowed, v)
    }
    if len(allowed) == 0 {
        return Decision{}, fmt.Errorf("no vendor for %s inside regions %v", capability, p.Regions)
    }
    if p.Pin != "" {
        for _, v := range allowed {
            if v.ID == p.Pin {
                return Decision{Capability: capability, Vendor: v.ID, Region: v.Region, Rule: "pin:" + p.Pin}, nil
            }
        }
        // A pin outside the residency list is a policy conflict, not a reason to fall back.
        return Decision{}, fmt.Errorf("pinned vendor %s is outside regions %v", p.Pin, p.Regions)
    }
    chosen := allowed[0] // router preference order, already filtered
    return Decision{Capability: capability, Vendor: chosen.ID, Region: chosen.Region, Rule: "default"}, nil
}

slices.Contains landed in the standard library in Go 1.21, which is the only reason this reads as six lines instead of twenty. Look at what the record does not contain: no credential, no prompt, no listing body. A fingerprint of the key answers "which credential reached vendor A in Frankfurt at 03:14" without turning the audit trail into a second copy of the data you were trying to keep inside one region. OWASP's secrets management guidance argues the same thing from the other direction — the credential is supposed to be short-lived, scoped and rotatable, and anything that pins your logs to a specific secret value fights that.

Residency is an access claim, not a routing preference

GDPR Chapter V, Article 44 in particular, treats moving personal data to a third country as something you need a lawful basis for, and the basis has to exist at the moment of the transfer rather than at the moment somebody asks. A routing preference cannot carry that weight. In most gateways a preference is a hint the router is free to satisfy or ignore, and a hint is not evidence.

So enforce the region list as a filter with a hard failure, the way Resolve does above, and record the region as a field rather than deriving it later from a response header you happened to log.

Test the decision instead of inferring it. Give the gateway a dry-run path that resolves the policy and returns the Decision without making the upstream call, then assert it in CI: translation never resolves outside the EU region list, moderation never resolves to the excluded vendor, a pin whose region was revoked fails the build. Those assertions are a few lines each and they catch the class of mistake that otherwise surfaces as a regulator's question eighteen months later. In production the same four fields — vendor, region, rule, policy version — belong on the span as attributes, so a trace can answer the residency question without a join against a table that may have been rotated out from under you. OpenTelemetry's semantic conventions cover the HTTP and generative-AI parts of that span already; the routing rule is the part you add yourself.

Caps that hold before the invoice arrives

A budget you reconcile at month end is a report, not a cap. To stop spending, the check has to run in the request path, against the same credential the workload presents, and it has to survive retries without drifting.

Retries are where naive counters die. The enrichment worker times out at 30 seconds, the vendor finishes at 31, the job retries, and one listing gets charged twice while the counter says once. Run it the other way — increment first, crash before the call — and the counter runs ahead of reality, which is the failure nobody notices because it doesn't look like one and slowly strangles throughput. The fix is the same boring fix as everywhere else in payments: an idempotency key supplied by the caller, a uniqueness constraint doing the locking, and a reservation row that a settlement job later reconciles line by line against the vendor's own usage record.

// Reserve debits the workload budget exactly once per idempotency key.
func (l *Ledger) Reserve(ctx context.Context, d Decision, estimateMinor int64) (int64, error) {
    tx, err := l.db.BeginTx(ctx, nil)
    if err != nil {
        return 0, err
    }
    defer tx.Rollback()

    // The insert is the lock. A replayed attempt updates its row to the value it already had,
    // so RETURNING still yields exactly one row and the workload is charged once.
    const reserve = `
INSERT INTO spend_reservation (idempotency_key, workload, capability, vendor, region, amount_minor, period_day)
VALUES ($1, $2, $3, $4, $5, $6, CURRENT_DATE)
ON CONFLICT (idempotency_key) DO UPDATE SET amount_minor = spend_reservation.amount_minor
RETURNING amount_minor`
    var amount int64
    if err := tx.QueryRowContext(ctx, reserve, d.IdempotencyKey, d.Workload, d.Capability,
        d.Vendor, d.Region, estimateMinor).Scan(&amount); err != nil {
        return 0, err
    }

    const spentToday = `SELECT COALESCE(SUM(amount_minor), 0) FROM spend_reservation
WHERE workload = $1 AND period_day = CURRENT_DATE`
    var spent int64
    if err := tx.QueryRowContext(ctx, spentToday, d.Workload).Scan(&spent); err != nil {
        return 0, err
    }
    if spent > l.capMinor {
        return spent, ErrBudgetExhausted // fail closed: the call never leaves the gateway
    }
    return spent, tx.Commit()
}

Two things there are deliberate. The insert is the lock, so there is no advisory lock and no SELECT ... FOR UPDATE dance around a counter row. And exhaustion fails closed, because a marketplace that keeps enriching listings past its cap has converted an operational limit into an invoice surprise, which is the exact outcome the cap existed to prevent.

None of this is Go-specific. I write ledgers in Go out of habit; the same three steps — resolve, reserve, record — fit a Node.js gateway middleware, an Envoy external-authorization service, or an NGINX auth subrequest just as well, as long as all three happen before the upstream call and for every capability rather than for the one that embarrassed you last quarter. When the cap does trip, return a problem document as described in RFC 9457, carrying the workload, the period and the limit. A bare 402 tells the calling worker nothing it can act on, and "shed load or escalate to a human" is a decision that needs the numbers.

One more piece of hygiene: every pin and every exclusion in that policy file carries a ticket id and an expiry date, and the expiry is enforced by a test. Rules without expiry dates are how a two-week vendor incident becomes permanent architecture.

What I stop keeping, and what that costs when a dispute lands

Record	Kept for	Why that number
Routing decisions: vendor, region, rule, policy version, key fingerprint	12 months, most recent 3 immediately queryable	PCI DSS v4.0.1 requirement 10.5.1 sets that floor for audit logs, and the dispute windows I care about sit inside it
Reservations and settlements	12 months	reconciled against the invoice line by line, so it has to outlive the invoice
Request and response payloads	7 days, then a hash	this is the term that carries both the storage cost and the residency exposure

Payloads are the expensive thing and the dangerous thing at the same time. They are also the only part of the trail that a per-capability residency rule cannot fully protect, because once you copy a listing body into a central log bucket you have performed the transfer you just spent a week preventing at the router.

Here is the price of dropping them. When a seller disputes a takedown four months later, the trail proves which vendor handled the call, in which region, under which policy version, against which credential, for what reserved amount — and it cannot reproduce the text the model actually saw. The decision record answers the access question. It does not answer the content question, and those are not the same audit.

The catch is that all of this is overhead if your workload talks to one vendor under a committed contract, where a daily counter and a billing alert beat every line above. It is also the wrong shape for interactive traffic: a hard reject at the cap is correct for a background enrichment queue and hostile on a checkout path, where the better move is to degrade to a cheaper capability tier and let the limit bite somewhere the buyer cannot feel it. And I'm not sure seven days is the right payload window for every marketplace — if your appeals process runs longer than your retention, you have moved the failure from the invoice to the support queue, which is a worse place to discover it.

DEV Community