DEV Community

Fernando Azevedo
Fernando Azevedo

Posted on Edited on Originally published at fernando.moretes.com

Sharing a GPU cluster on HyperPod: three designs and when each one pays off

#aiagents #sagemakerhyperpod #eks #gpu

The question that usually opens the shared GPU cluster discussion is "how do I split the machines across teams?". The question that decides whether the design survives is a different one: when the research team's training job gets evicted at 2 AM to make room for an inference endpoint, who finds out, how many GPU-hours were thrown away, and which cost center absorbs them? The reference architecture AWS published on 10/08/2026 for SageMaker HyperPod on EKS answers a good part of that. Here I put it side by side with the two alternatives I see most often and say under which conditions each one is worth its maintenance cost.

The real choice: three ways to split GPUs

Option A, one shared HyperPod EKS cluster with Task Governance: the design in the post. Each team gets a hyperpod-ns-<team> namespace, its own SageMaker domain for Studio users, and a permission set in IAM Identity Center for CLI users. Fairness across teams belongs to Task Governance, built on Kueue: guaranteed quota per team, lending of idle capacity, and priority classes.

Option B, one cluster per team, ideally in separate accounts: the boundary is the AWS account. There is no noisy neighbor because there is no neighbor. In exchange, each team carries its own capacity floor, and GPUs idle in one cluster cannot rescue a full queue in another.

Option C, a shared cluster with namespaces and ResourceQuota only: this is what shows up when the platform starts as a single-team cluster and other teams trickle in. It works until the first week of contention, when the quota becomes a hard ceiling and arrival order decides who gets to train.

The difference between the three is not the cost of standing up the cluster. It is what each one demands in continuous operation: reviewing quotas, debugging Kueue admission, maintaining the UID/GID webhook, reconciling chargeback. That is the cost that stays for years, and it is the one I compare on.

From login to GPU: where each boundary in design A is enforced

Identity decides the namespace, the namespace decides the queue, the queue decides whether the pod runs on its own or borrowed quota, and the namespace becomes the chargeback key.

🔐 IAM Identity Center: identidade

  • Permission set AWSReservedSSO_TeamA_* (security)
  • SageMaker domain TeamA-domain + TeamA-role (security)

🟧 AWS: HyperPod EKS control plane

  • EKS access entries escopo: hyperpod-ns-team-a (security)
  • Admission webhook UID/GID via sessionName (compute)
  • DynamoDB mapa POSIX (data)

🤖 Task Governance: Kueue

  • LocalQueue hyperpod-ns-team-a-localqueue (messaging)
  • ClusterQueue cota garantida + borrow limit (messaging)
  • Idle resource sharing IdleResourceSharing: Enabled (messaging)

🟧 AWS: pool de GPU compartilhado

  • Nós HyperPod health checks + auto-recovery (compute)
  • FSx for Lustre /fsx/TeamA (storage)

📤 Saída: custo e visibilidade

  • Kubecost custo por namespace (data)

Flows

  • idp -> ps: federates groups
  • user -> ps: aws sso login
  • user -> dom: Studio
  • ps -> ae: kubectl
  • dom -> ae: execution role
  • ae -> wh: pod created
  • wh -> ddb: resolves UID/GID
  • wh -> lq: queue-name label
  • lq -> cq: admission
  • cq -> idle: over quota: borrows
  • cq -> gpu: admits pod
  • gpu -> fsx: checkpoint
  • gpu -> cost: usage per namespace

Isolation: a namespace is an organizational boundary, not a security one

The post itself states it plainly: a namespace is an isolation boundary, not a hard security boundary, and the design targets several teams within one company, not tenants who do not trust each other. I agree, and I add what that means in a regulated environment.

Design A stacks four controls: EKS access entries scoped per namespace (one for the domain role, another for the AWSReservedSSO_<permission-set>_<id> role), RBAC that returns Forbidden on cross-namespace calls, a default-deny NetworkPolicy per namespace, and S3 restricted to the team's prefix. All four run on the same kernel, the same NVIDIA drivers and the same control plane.

Silent failure number one: a NetworkPolicy without a CNI that enforces it is just YAML. If the Amazon VPC CNI does not have network policy enabled, kubectl apply succeeds, the object exists, and not a single packet is blocked. Test with a team B pod trying to open a connection to a team A service before you call the boundary done.

My rule is this. If the teams share the same regulatory regime and the same data classification, a namespace is enough. If one team trains on customer data under LGPD and another on synthetic data, or if a BACEN 4.893 auditor is going to ask for demonstrable segregation, the right boundary is the account, and the design goes back to B.

Fairness: quota, borrowing, and who gets evicted

Task Governance splits two policies. Compute prioritization decides how idle capacity is handed out (first-come first-serve or fair-share) and how tasks are queued (arrival order or task ranking with priority classes). Compute allocation sets, per team, the guaranteed quota, a fair-share weight from 0 to 100, and the lending posture: Lend and Borrow, Lend, or Don't Lend. The borrow limit can be a percentage, up to 10,000% of quota, or absolute per instance type.

The documentation example shows the mechanics: team A with a quota of 6 using 2, team B with a quota of 5 using 4, a 4-unit job lands on team B and 3 units are borrowed from A. What the documentation also says, and what defines operations, is that a task running on borrowed quota can be preempted by a higher-priority task when capacity runs out.

Silent failure number two: a job without kueue.x-k8s.io/priority-class gets the lowest priority. An inference endpoint deployed without the label becomes the first eviction candidate in the cluster.

Preemption has a price in GPU-hours. A 16-GPU training run on borrowed capacity with a checkpoint every 30 minutes loses up to 8 GPU-hours per eviction, 4 on average. If the cluster evicts that job twice a day, that is roughly 240 GPU-hours a month thrown away without tripping a single alert. That is why I treat borrowed capacity like spot: only jobs with checkpoints on FSx and idempotent resume run on it.

Chargeback: who pays for idle and borrowed GPUs

The post uses Kubecost grouped by namespace, and since the namespace is already the team, no extra workload tagging is needed. That works for the question "how much did team A consume". On its own it does not answer two questions that show up at the first month-end close.

The borrowed GPU: when team B runs on team A's quota, the pod lives in B's namespace. Charging B is correct, but team A will look at the report and see it paid for a guaranteed quota someone else used. Decide upfront whether the guaranteed quota is a fixed price (team A pays for the reservation, used or not) or usage-based. Both rules are defensible; changing the rule mid-quarter is not.

The idle GPU: capacity not allocated to any quota needs an owner in the report. Split Cost Allocation Data for EKS, which started covering GPU, Trainium and Inferentia on 09/02/2025, splits cost per pod and reports unused capacity separately, with a 9:1 GPU versus CPU and memory weighting in the documentation example. That example is an EC2 instance. HyperPod nodes are ml.* instances billed through SageMaker, so confirm on your bill whether that data shows up before you swap Kubecost for CUR. I would keep Kubecost as the chargeback source and use CUR to reconcile the total.

The three designs side by side

Criterion A: shared HyperPod + Task Governance B: one cluster per team C: namespaces + ResourceQuota
Isolation boundary Namespace + access entry + NetworkPolicy AWS account Namespace + RBAC
One team's idle GPU Lent, with limits and preemption Stays idle Stays idle (quota is a hard cap)
Contention Fair-share by 0–100 weight + task ranking None across teams Arrival order
Chargeback Per namespace, needs a borrowing rule Account bill, trivial Per namespace
Parts to maintain Quota policies, UID/GID webhook, Kubecost, dashboards N clusters, N upgrades, N pipelines Few, until the first dispute
When it fails badly Job without priority-class or checkpoint Low utilization becomes fixed cost Large team monopolizes the cluster

Decision matrix

A: shared HyperPod + Task Governance

Pros

  • One team's idle GPU becomes another's capacity, with IdleResourceSharing covering even unallocated capacity
  • Explicit priority: production inference preempts experiments
  • One driver and EKS upgrade for every team

Cons

  • A namespace will not pass a strong-segregation audit
  • The UID/GID webhook and its DynamoDB map become yours to run
  • Preemption wastes GPU-hours when jobs do not checkpoint

Verdict: Default for teams in one company under one regime

B: one cluster per team

Pros

  • Segregation provable by account, IAM and bill
  • A bad upgrade's blast radius stays within one team

Cons

  • Each team pays its own GPU floor, even when idle
  • Platform effort multiplied by N

Verdict: When data or regulation differ across teams

C: namespaces + ResourceQuota

Pros

  • Nothing new to learn

Cons

  • No borrowing, no cross-team priority, no fair-share
  • GPU disputes become meetings, not policy

Verdict: Only as a step until the second team arrives

Guaranteed quota is not guaranteed capacity: The documentation says to keep the sum of quotas below cluster capacity: 20 instances, quotas summing to under 20. But HyperPod pulls unhealthy nodes out of rotation, and idle resource sharing already excludes NotReady or unschedulable nodes from its math. Team quotas do not shrink with them. In a 16-node cluster with 2 nodes recovering, quotas summing to 16 promise capacity that does not exist at that moment. I keep at least one node per instance type outside any quota.

What I would do: I would start with design A, but with three rules written down before the first job: any manifest without kueue.x-k8s.io/priority-class is rejected by an admission policy, every job running on borrowed quota writes checkpoints to FSx at a known interval, and the billing rule for borrowed quota is signed off by the cost center owners. I would also make the UID/GID webhook's failurePolicy an explicit decision: Fail so the cluster stops scheduling when DynamoDB does not answer, Ignore only if you accept pods with the wrong POSIX identity writing to /fsx. I have seen shared platforms die not from a technical limit, but because the first unexplained eviction made a team ask for its own cluster. The lesson: in a shared cluster, predictability is worth more than utilization.

Verdict

Use Option A (shared HyperPod with Task Governance) when the teams belong to the same company, handle data with the same classification, and someone owns the quota policies and Kueue observability (kubectl describe workload has to become routine, not archaeology). Use Option B (cluster per account) when one team operates under LGPD with customer data and another does not, or when the audit requires segregation by account; accept paying for idle GPUs as the price of isolation. Treat Option C as a transitional state: as soon as a second team competes for GPUs, move to A. The gain in design A is not having a single cluster, it is lending idle GPUs without negotiating in a meeting, and it only pays off if evictions are predictable and chargeback is agreed upfront.

Rating: A, com regras de admissão e checkpoint /

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Top comments (0)