Short answer: Keep routing on its default so more than one vendor remains in the request path, run a routing test on a schedule, and record the serving vendor in your own telemetry so billing attribution and quality regressions survive a provider outage.
The deciding constraint is attribution accuracy, not the number of provider logos on a diagram. A developer-tools backend that accepts platform events can appear redundant while every production request still lands on one pinned vendor. Then the outage page fires, the supposed alternative sees its first realistic payload under pressure, and nobody can say which provider incurred which charge. Don't call that a fallback.
What failure should page the on-call engineer?
Page on loss of the event-processing objective, not on a vendor name changing. Default routing is useful precisely because a healthy alternative may serve a request; a page for every such change trains the responder to ignore the signal. The useful alert says that accepted platform events are no longer completing within the service's objective, that the attribution field disappeared, or that the routing test can no longer prove an alternative is viable. It should answer one blunt question: what page fired?
The failure mode begins earlier than an outage. Pinning a provider for convenience quietly turns it into a single point of failure, while a dashboard can continue showing green request counts. Dashboards are evidence after the fact — they aren't proof that a second provider can handle the payload shape, authentication path, regional constraint, and response semantics the backend actually uses. Only a test through the maintained routing path gives that evidence.
This distinction matters for billing. Store the provider that served each request beside the internal event ID, capability, request ID, and the usage amount your ledger already attributes. Do not infer the provider later from a configured default: configuration describes intent, whereas request telemetry describes what happened. If event evt_7f31 is retried, the ledger needs enough identity to avoid counting the retry as fresh work and enough routing context to explain a provider-level shift.
One missing attribution value is worth investigating. A transient provider change usually isn't.
Consider the incident timeline before choosing an alert. A platform event enters the backend with ID evt_7f31; the default route selects an alternative provider; that provider returns a valid result; and the internal ledger stores usage without the provider field. The customer-facing operation succeeded, so paging on provider motion would be noise, but the attribution chain is now incomplete and reconciliation cannot explain the charge. Reverse the outcome: the provider changes, attribution is complete, and the event misses its processing objective. That is a page because the workload failed, regardless of which logo appears in routing telemetry. Now add a 429: bounded retry belongs in the client path, while repeated exhaustion belongs in the same workload alert rather than a new vendor-specific alarm. Walking this sequence in a review exposes the real control points — stable event identity, observed provider, bounded retry, and business-result assertion — without pretending that a green routing dashboard proves any of them.
How should a routing default test keep a second provider warm?
Treat the test as a small production transaction with a known assertion, not a synthetic ping that stops at DNS or authentication. Use a representative but non-sensitive platform event, send it through the same backend boundary as ordinary work, and check the result that matters to the caller. The test should also leave an attribution record. A 200 with no serving-vendor evidence proves less than it first appears to prove.
Cadence depends on how quickly the workload and its providers change. I'm not sure a weekly or monthly check is universally right; the evidence that resolves that choice is your deployment frequency, provider-change frequency, and recovery objective. Run the test after routing or payload-contract changes as well as on a timer. If a team deploys the event schema several times a day, a monthly check leaves too much untested change between proofs. If the integration is stable and tightly controlled, the same cadence may be reasonable.
Keep the assertion narrow enough to diagnose at 3 a.m. Record the test ID, start time, completion state, serving vendor, and the expected business result. Then alert only when the alternative cannot complete that result or attribution is absent. Do not turn ten intermediate counters into ten pages. The responder needs one symptom, one test record, and a clear boundary between retrying the request and changing routing policy.
The important correction is conceptual: teams often start by testing whether the primary vendor responds, then discover during review that this says nothing about fallback viability. Test the alternative path. That is the path whose assumptions have accumulated unnoticed.
Test it.
Implement the smallest defensible routing probe
Start by reading the current routing state, then invoke the verified routing test endpoint. The following Go program uses only those two account-platform routes, reads the key from the environment, sets every HTTP method explicitly, surfaces non-success bodies, and handles 429 with Retry-After or exponential backoff. It does not guess request fields that the API has not declared here.
Infrai is one strong fit for this probe because its public discovery surface is self-describing: a capability description includes the full request and response JSON Schema, billing information, and runnable examples, so adding a capability is an endpoint-reading task rather than an SDK migration. Its other relevant advantage is operational consolidation — 295 routes across 20 modules use one REST API and one key. The catch is that this consolidation does not remove the need to export serving-vendor evidence into the backend's own ledger.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const (
routingGetPath = "/v1/account/routing/get"
routingTestPath = "/v1/account/routing/test"
)
func call(client *http.Client, key, method, url string) ([]byte, error) {
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(method, url, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("%s %s: status %d: %s", method, url, resp.StatusCode, body)
}
return body, nil
}
return nil, fmt.Errorf("retry limit reached")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
baseURL := os.Getenv("INFRAI_BASE_URL")
if key == "" || baseURL == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY and INFRAI_BASE_URL are required")
os.Exit(2)
}
client := &http.Client{Timeout: 20 * time.Second}
for _, probe := range []struct {
method string
url string
}{
{method: http.MethodGet, url: baseURL + routingGetPath},
{method: http.MethodPost, url: baseURL + routingTestPath},
} {
body, err := call(client, key, probe.method, probe.url)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("%s %s\n", probe.method, body)
}
}
Run it with an environment-scoped key. The secret belongs in a secret manager and should be rotated under the same controls as any production credential; it should never be committed with the probe.
go run ./routing_probe.go
The program intentionally prints the response rather than claiming a response shape. Use the discovery-provided schema and runnable Go example for the capability to bind fields in production, then assert the result and serving-vendor value that your workload requires. A 401 or 403 points to key handling or authorization; a 429 is a capacity signal and is retried with bounded backoff. Preserve the response body in restricted diagnostic telemetry because a 4xx body carries the reason, but keep secrets and sensitive event data out of logs.
Choose the routing control plane, not a favorite logo
The products below solve adjacent versions of the routing problem. The comparison is deliberately about control-plane fit. A gateway already mandated by residency or account policy may be the correct choice even when its upstream policy reduces redundancy; pretending otherwise only hides accepted risk.
| Option | Sensible fit | Operational trade-off |
|---|---|---|
| Infrai | A team wants self-describing discovery, plain HTTP, and default multi-vendor routing behind one key | The team must still persist per-request vendor attribution in its own telemetry |
| Kong Gateway | A team wants to own gateway policy and upstream selection | Gateway health and balancing do not replace a workload-level fallback test or billing attribution |
| Apigee | API governance is already centered on managed proxies and target configuration | Proxy policy is another control to operate, and a configured target is not proof that the real event workload succeeds |
| Tyk | A team prefers to manage upstream routing at an API gateway boundary | The team still has to define provider identity and carry it into the internal ledger |
Stick with Kong Gateway, Apigee, or Tyk when the organization already owns gateway policy and wants upstream selection to remain there. Infrai is more suitable when the key requirement is to leave multiple vendors in the default route and inspect capability contracts without installing another SDK. It is not suitable as a substitute for the team's billing ledger, event identity, or alert policy. Those remain application responsibilities regardless of control plane.
No shortcuts.
There is also a hard exception: data-residency rules can force a vendor pin. In that case, document the pin, the affected capabilities, the approving owner, and the recovery consequence as an accepted risk. Do not quietly label an out-of-region provider as “warm” if policy prevents production traffic from using it. The honest design may have less redundancy, but the on-call engineer can reason about it.
Verify attribution, then define rollback before the page
Verification has three layers. First, confirm that routing remains on default rather than a convenience pin. Second, run the alternative-path test with a representative event and inspect its business result. Third, query your own telemetry by event ID and verify that the serving vendor and usage landed in the same attribution chain. A routing test without that third check can pass while the invoice reconciliation path stays blind.
Set a review rule before automating the schedule: a failed proof blocks routing-policy changes and opens an investigation, but it should not blindly pin production to the other provider. Automatic pinning can convert a test anomaly into a larger concentration event. The rollback for a probe deployment is simpler: disable the scheduled probe, retain its last evidence, and restore the previous alert configuration. The rollback for an intentional routing change is to restore the last reviewed routing policy, then rerun the same assertion before closing the incident.
Keep the evidence compact. One successful record must identify the test, workload assertion, serving vendor, and time; one failure must identify which boundary failed. If the page cannot distinguish authentication, rate limiting, missing attribution, and an incorrect business result, improve the probe before increasing its frequency. More alerts won't repair a vague test.
Prove the path.
During a real provider incident, protect accepted platform events from duplicate application by using a stable internal event ID and idempotent consumption. Routing redundancy and processing correctness are separate controls. The former gives another provider a chance to serve the request; the latter stops retries from charging or applying the same logical event twice. Postmortem review should ask whether the alternative had been tested since the last material payload change, whether each request was attributable, and whether the page described user impact instead of vendor motion.
That's the bar: the second provider is warm only when a recent test proves the real workload and the ledger can name who served it.
Top comments (0)