SageMaker HyperPod Governance: Project Access to Quotas

Sharing a GPU cluster across teams sounds simple — until everyone wants the same H100s at 2 AM. AWS's own administration guidance for SageMaker HyperPod is blunt about one thing: connecting a cluster to a Unified Studio project gives teams a workspace, not a security boundary. Project membership, tenant permissions, and accelerator quotas each need an explicit design. Here's what the vendor guidance actually says.

1. Start with who can touch the infrastructure

Before deciding how teams share compute, lock down who can create it. AWS documents three IAM condition keys for granular control over CreateCluster and UpdateCluster:

• sagemaker:InstanceTypes — restrict which instance types callers may choose
• sagemaker:VpcSubnets — pin cluster creation to approved subnets
• sagemaker:VpcSecurityGroupIds — control which security groups apply

This is the infrastructure API layer. Tenant and workload controls sit below it — different knobs, different layer.

2. Projects are collaboration, not isolation

Unified Studio lets a project launch workloads on a connected HyperPod cluster, but AWS explicitly describes projects as collaboration boundaries — clusters remain managed through SageMaker AI interfaces. For EKS clusters, the real runtime boundary is the classic tenant-isolation stack: a namespace per tenant, RBAC, per-tenant service accounts, EKS Pod Identity roles, default-deny network policies, and tenant-specific storage plus KMS keys.

One subtle gotcha from the guidance: AI Studio users can see all EKS cluster tasks by default, and on Slurm, every user can view, manage, and interact with available tasks. Project membership ≠ task privacy. Check that behavior explicitly.

3. Quotas depend on your orchestrator — and they're not interchangeable

EKS and Slurm get completely different capacity mechanisms, and AWS is explicit that Slurm has no equivalent of EKS's lending-and-borrowing model:

Concern	EKS	Slurm
Team allocation	HyperPod task governance quotas	Accounting w/ hierarchical accounts
Scheduling policy	Priority classes + preemption	QoS, priority, fair-share, partitions
Spare capacity	Lending and borrowing	No documented equivalent

On EKS, the task governance add-on installs Kueue (Kubernetes-native quota management) — a real dependency with prerequisites: K8s 1.30+, an existing HyperPod node in the cluster, and you must uninstall any existing Kueue first. After install, SageMaker manages KueueManagerConfig, ClusterQueue, and LocalQueue resources for you.

The takeaway

Governing a shared accelerator pool is a layered design problem: IAM condition keys at the API layer, namespaces + identity + network + storage at the runtime layer, and orchestrator-specific quotas at the scheduling layer. Skip any layer and you get either blocked teams or surprise GPU bills.

Go deeper

The full walkthrough covers each control with links to the underlying AWS documentation: SageMaker HyperPod governance: from project access to workload quotas.

---

Originally published on Agentic Affairs (https://agentic-affairs.com)