๐ November 20, 2026 โข AWS ML Blog
On November 20, 2026, AWS announced general availability of Amazon SageMaker HyperPod Global Inference Gateway. Per AWS, it is one SageMaker-managed endpoint that routes inference requests across every registered HyperPod cluster, in a single Region or across several. The company describes it as the global tier on top of the routing it already runs inside each cluster.
The per-cluster Inference Gateway, introduced earlier, picks the model pod inside a cluster using signals like KV cache utilization and queue depth, and that layer stays unchanged. The new global tier picks the cluster instead. In the AWS example, a team serving 4 models across 5 clusters in 2 Regions tracks 20 endpoints without gateways, 5 with per-cluster gateways, and 1 with the global gateway โ 20 endpoints become 1.
The problem, per AWS, shows up at traffic peaks: requests queue on a saturated cluster while GPUs in another cluster, sometimes in another Region, sit idle, and failover used to mean manual reconfiguration. The gateway scores healthy clusters on 4 signals โ KV cache headroom, queue depth, Region proximity and recent error rate โ with default weights 0.20, 0.25, 0.40 and 0.15, which operators can override with values from 0.0 to 1.0 as long as all four are set and sum to 1.0. Changes via UpdateGlobalInferenceGateway apply to new requests.
Failover is automatic. When a cluster fails, traffic moves to the next-best cluster and clients change nothing, a failed request is retried once on another cluster unless streaming has already started, and if no healthy cluster remains the gateway returns HTTP 503. Traffic returns gradually after recovery, per AWS. Clients send OpenAI-compatible requests signed with AWS Signature Version 4 (SigV4) and authorized through IAM, and requests reach clusters only over AWS PrivateLink or the AWS backbone, with AWS stating that prompts and completions are not logged or stored.
Setup takes two steps: publish each cluster by enabling spec.globalRouting in its InferenceGatewayConfig, then call CreateGlobalInferenceGateway with the clusters to group, and clusters can be added or detached later with CLI commands. The gateway is generally available in 13 AWS Regions across the Americas, Europe and Asia Pacific. Per AWS, platform teams get capacity overflow and cross-Region failover without building their own load balancers, DNS or cross-Region networking. Clients call one signed endpoint instead of tracking one endpoint per model per cluster.
The AWS blog also lists caveats: a published cluster that belongs to no gateway receives no traffic while its load balancer keeps running and is billed, a model missing from every scheduler in the configuration returns 404, and identity-token checks should be on for all clusters serving a model or for none. In the beta, per the same blog, Jinhyung Park of Bagelcode saw the gateway route requests across two AWS Regions through a single endpoint and keep serving when one cluster could no longer serve the model, which he sums up as cross-Region failover without having to build or maintain their own routing layer.
Sources:
https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemaker-hyperpod-global-inference-gateway-one-endpoint-across-regions-and-clusters/
https://www.unite.ai/aws-routes-hyperpod-inference-across-regions-from-one-endpoint/
https://inite.ai/en/news/aws-adds-one-endpoint-routing-across-hyperpod-gpu-clusters
https://aitoolmaps.com/news/2026-11-20/aws-adds-global-inference-gateway-to-amazon-sagemaker-hyperp
https://aiagentnews.io/stories/aws-adds-a-global-inference-gateway-for-routing-across-sagemaker-hyperpod
https://guthlabs.ai/news/story/aws-sagemaker-hyperpod-global-inference-gateway-2026-11-20
Sources:
- https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemaker-hyperpod-global-inference-gateway-one-endpoint-across-regions-and-clusters/
- https://www.unite.ai/aws-routes-hyperpod-inference-across-regions-from-one-endpoint/
- https://inite.ai/en/news/aws-adds-one-endpoint-routing-across-hyperpod-gpu-clusters
- https://aitoolmaps.com/news/2026-11-20/aws-adds-global-inference-gateway-to-amazon-sagemaker-hyperp
- https://aiagentnews.io/stories/aws-adds-a-global-inference-gateway-for-routing-across-sagemaker-hyperpod
- https://guthlabs.ai/news/story/aws-sagemaker-hyperpod-global-inference-gateway-2026-11-20
Continue the conversation: https://t.me/AIAutomationArchitecture ยท Bluesky
Top comments (0)