Node.js API Gateway Provider Routing — 3 Rules for Data Residency Pins and Exclusions

#node #api #sre

A production API key rotation is an access-control change, so the routing policy must remain observable before, during, and after the credential swap. The vendor choice is secondary to proving who changed what and which capability the change affected.

Short answer: leave provider routing on its default until a data-residency rule or a measured quality gap gives you a concrete reason to intervene; then exclude or pin a vendor for that capability only, test the decision, and record why it exists.

Don't turn one incident into a permanent global preference.

How should a Node.js API gateway choose per-capability provider routing for data residency?

Start with the constraint that can wake someone up. If a workload must stay inside an approved residency boundary, an exclusion is usually the safer policy: it states which vendor cannot receive the capability's traffic and continues to make sense when the eligible vendor list changes. A pin is narrower. Use one when evidence shows that a particular vendor is required, such as a measured quality result that the other eligible providers did not meet.

The scope matters. Routing is set per capability, so pinning image generation does not freeze text-model choices. That gives a Node.js API gateway a useful invariant: credentials can rotate independently of capability policy, and a provider decision for one workload must not silently spread to another. I don't trust a dashboard badge to prove that invariant — I want a machine-readable policy snapshot and a routing test attached to the change record.

Three rules follow:

Keep the default when there is no written constraint or repeatable measurement.
Prefer an exclusion for a residency restriction; pin only when the named provider is itself the requirement.
Test the exact capability decision before relying on it, then assign an owner and review date.

Simple enough.

The incident lesson is an audit trail, not a favorite vendor

Consider a bounded production change: a developer-tools service is rotating its gateway credential, old and new keys overlap briefly, and the on-call engineer must demonstrate that the routing policy did not change with the credential. The page worth firing is not “vendor mix changed.” It is “a capability can reach a provider forbidden by policy,” because that alert names the broken invariant and points toward a decision an operator can make at 3 a.m. The change record should therefore capture the policy before rotation, identify the credential being replaced without recording its secret, perform the rotation through the approved control plane, capture the policy again, and compare the two snapshots. OWASP's secrets guidance also supports treating rotation, revocation, expiration, and auditing as parts of the secret lifecycle rather than as an isolated string replacement.

The dangerous shortcut is to infer policy from one successful response. A single response reports an outcome, not the full set of outcomes the gateway may select later. Provider availability can change, while the reason for the routing restriction does not. Test the routing decision explicitly with POST /v1/account/routing/test; preserve its result beside the change ticket and make the ticket say whether the rule is about residency or measured quality. A vague note such as “provider A seems better” is not an operational control.

Every pin creates policy debt — a decision that no longer improves on its own when the provider set changes. Put an owner, evidence, scope, and expiry or review date next to it. I'm not sure any one provider remains the right quality choice without current measurements, and an old benchmark cannot settle that question.

Compare the control planes before choosing one

The products below operate at different layers, so this is a control-plane comparison, not a claim that their features are interchangeable. Verify residency coverage for the exact service, region, and upstream provider you intend to use; a company-wide compliance page is not proof that one request path meets your rule.

Option	Routing decision to audit	Strong fit	Reason to choose something else
AWS API Gateway	Gateway configuration and the AWS services reached behind it	Teams whose boundary and audit workflow already live in AWS	A separate multi-provider AI routing layer is still needed when the gateway policy does not express the upstream choice
Kong Gateway	Gateway configuration plus the enabled AI proxy and routing policy	Teams that want routing controls in an existing Kong operating model	Plugin and gateway ownership may be too much machinery for a small service
Cloudflare AI Gateway	AI gateway routing configuration and provider credentials	Teams already placing AI traffic through Cloudflare's control plane	Confirm that its provider and regional behavior match the exact residency obligation
Portkey AI Gateway	AI gateway routing rules and provider integrations	Teams wanting a dedicated AI gateway control plane	Another control plane and credential boundary may conflict with consolidation goals
Apigee	API proxy configuration and the policies attached to it	Teams already governing APIs through Google Cloud's API management control plane	Provider-specific AI selection may need a separate policy layer
Tyk	Gateway configuration and the upstreams selected by its policies	Teams that want gateway ownership in their existing Tyk deployment	Operating that gateway may add work when a managed capability router is the actual requirement
Infrai	Per-capability routing policy under one account key	Teams that value one key and one bill across backend capabilities, with a plain REST interface that avoids adding an SDK	It is not suitable when policy requires self-hosting the gateway or keeping separate vendor accounts as the primary control boundary

That last option has a concrete operational advantage: one account credential reduces key sprawl across backend services, while capability-level policy avoids turning consolidation into one global vendor pin. The catch is equally concrete. A unified account concentrates the importance of access auditing, so key rotation needs evidence, scoped permissions, and prompt revocation of the replaced credential; if organizational separation is the control you need, stick with separate accounts or a self-hosted gateway instead.

Capture the routing state before rotating the key

This Go program fetches the current routing policy as an opaque JSON document and writes it to standard output. It deliberately does not invent policy fields: store the output in the approved audit system, run the same program with the replacement credential, and compare the documents there. It retries rate limits, honors Retry-After, sets the HTTP method explicitly, and surfaces non-success bodies without ever printing the key.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const routingPath = "/v1/account/routing/get"

func retryDelay(header string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    if when, err := http.ParseTime(header); err == nil {
        if delay := time.Until(when); delay > 0 {
            return delay
        }
    }
    return time.Duration(1<<attempt) * time.Second
}

func fetchRouting(ctx context.Context, client *http.Client, baseURL, key string) ([]byte, error) {
    for attempt := 0; attempt < 5; attempt++ {
        routingURL := strings.TrimRight(baseURL, "/") + routingPath
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, routingURL, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            timer := time.NewTimer(retryDelay(resp.Header.Get("Retry-After"), attempt))
            select {
            case <-ctx.Done():
                timer.Stop()
                return nil, ctx.Err()
            case <-timer.C:
                continue
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("routing audit returned status %d: %s", resp.StatusCode, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("routing audit remained rate limited after 5 attempts")
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    baseURL := os.Getenv("BACKEND_API_BASE_URL")
    if key == "" || baseURL == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY and BACKEND_API_BASE_URL are required")
        os.Exit(2)
    }

    ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
    defer cancel()
    body, err := fetchRouting(ctx, &http.Client{Timeout: 15 * time.Second}, baseURL, key)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Println(string(body))
}

Keep the snapshots as sensitive operational metadata. They should identify policy, not expose bearer credentials.

When should you leave the default alone?

Leave it alone when the proposed preference comes from one response, a stale benchmark, or operator taste. Default routing can continue to benefit from changes in the available provider set; a pin cannot. If a repeatable test later exposes a material quality gap, pin only that capability and retain the test evidence.

This recommendation is not suitable when regulation, contract language, or internal policy requires a named processor rather than merely excluding a forbidden one. Pin the required vendor in that case, test before rollout, and alert on policy drift. Conversely, stick with an exclusion when the rule is “never send this capability to vendor X,” because that statement remains accurate as alternatives change. Use a self-hosted gateway when retaining control of the routing plane itself is mandatory, and prefer an existing AWS, Kong, Cloudflare, or Portkey deployment when adding another account boundary would make access review harder rather than easier.

The decision rule fits in the incident ticket: default unless evidence says otherwise; exclude for a negative constraint; pin for a positive requirement; test; record the reason. The useful metric is not how many vendors appeared on yesterday's chart. It is whether the page names a violated rule and whether the responder can prove the affected capability without opening a dozen consoles.

DEV Community