From SSH Chaos to Self-Service GPU Workspaces with Coder and HAMi

Disclosure: I designed the architecture, production rollout, and validation for this project. Parts of the Coder implementation were developed with Claude as an AI coding assistant, while all design decisions, testing, and production validation were performed on a real Kubernetes cluster.

TL;DR: Shared bare-metal GPU box → Coder on Kubernetes with per-user CPU/RAM/disk/VRAM quotas (forked) → HAMi for VRAM-isolated GPU sharing across 3× L40S. Fork: ranjbaryshahab/coder feat/per-user-resource-limits. Real gotchas inside.

The problem: one GPU box, many data scientists, endless conflicts

Our data-science team shared a single bare-metal GPU server over SSH. You can guess how that went: conflicting Python environments, someone's training job eating all the VRAM, "who installed CUDA 12.4 and broke everything," and no isolation between people. It worked — until it didn't, constantly.

So I joined that server to our Kubernetes cluster and started looking for something better.

The false starts (that taught me what I actually needed)

My first instinct was the obvious one: "just use containers." I told the team to work in Docker.

They pushed back — and they were right. For model training they need a real, persistent environment: SSH in, install things, run long jobs, use the GPU directly. A throwaway docker run isn't that.

So I tried the middle ground: one Ubuntu pod on Kubernetes with SSH access for everyone. Better than bare metal… but I'd just recreated the original problem inside a pod. Still no isolation, still no resource limits — one person could still starve everyone else, and I had no idea who was using what.

That's when the real requirements clicked:

• Per-user isolation (their own environment, their own home)
• Resource quotas (CPU / RAM / disk / GPU VRAM) so nobody hogs the box
• Self-service — spin up a workspace, pick VS Code / Cursor / a web IDE, without me in the loop
• Still real SSH and GPU for training

Enter Coder

Coder fit almost perfectly: self-hosted developer workspaces on your own infrastructure, with SSH, VS Code Desktop, browser IDEs, and Terraform-defined templates. On Kubernetes each workspace is a pod with a persistent home volume.

Almost — because the one thing I needed most, per-user resource quotas, is an Enterprise feature. Coder OSS doesn't provide pooled per-user resource quotas like "this user gets 32 CPU cores, 64 GB RAM, and 48 GB of VRAM to distribute across all of their workspaces."

So I forked Coder OSS and added the missing capability.

What I added to Coder OSS

• A custom_user_limits table + a self-contained API (one Go file, five migrations) — deliberately kept small and additive so the fork stays easy to rebase onto upstream.
• Pooled per-user budgets: CPU, memory, disk, and GPU VRAM are a total the user spends across all their workspaces (0 = unlimited).
• Enforcement in the Terraform template, not in Coder's core — preconditions reject a build that would blow the budget, with human-readable errors ("CPU quota exceeded: 12 of 32 cores already used…"). Worst case is a failed build with a clear message, never corrupted state.
• The template authenticates as the workspace owner (coder_workspace_owner.me.session_token) against self-service endpoints — so there's no admin token baked into templates, nothing to leak or rotate.
• Admin Usage page (live per-workspace CPU/mem/disk/VRAM) and a self-service Quota page for users.

One subtle bug worth sharing: the naive pooled check has a race — two workspaces created at the same instant each exclude themselves, count "everyone else," and both slip past the budget. The fix is a deterministic total order (count only workspaces whose build is strictly older), so of two concurrent creates exactly one loses. No locks, no reservation table.

Building a fork with AI assistance

I didn't hand-write every line of the fork. I used Claude to implement much of the Go, SQL, TypeScript, and Terraform after describing the features and constraints in plain language. That sped development considerably, but it didn't replace engineering judgment.

The architecture, trade-offs, production testing, and validation were still mine. In several cases the AI made confident but incorrect assumptions—for example, treating the L40S as having 48 GB of usable VRAM. Production testing exposed those mistakes, and they were corrected before rollout.

AI turned out to be a useful implementation partner, but a poor substitute for verification. Running everything on a real cluster remained essential.

Then the GPUs fought back

With Coder live, I hit the hard problem. We have 3× NVIDIA L40S on that node. Two engineers each created a workspace asking for 48 GB of VRAM — and Kubernetes scheduled them onto the same physical GPU. Neither could train.

That's when I learned NVIDIA time-slicing shares GPU compute time, not memory. The scheduler treats "GPUs" as interchangeable slots and is completely blind to VRAM — so two "48 GB" jobs happily collide on one 46 GB card and OOM each other. The vanilla options all failed my use case:

• Time-slicing → no memory isolation (the collision above)
• Whole-GPU → only 3 concurrent users, most VRAM wasted
• MIG → not supported on L40S
• MPS → shares are uniform; can't mix a 48 GB and a 10 GB tenant

I wasn't looking for GPU virtualization—I was looking for VRAM-aware scheduling. That's what eventually led me to HAMi.

HAMi to the rescue

HAMi (a CNCF project) does exactly what I needed: pods request nvidia.com/gpumem in MiB, and HAMi's scheduler bin-packs by free VRAM and hard-caps each pod via CUDA interception. Inside the container, nvidia-smi shows exactly the memory you were granted — nobody can eat a neighbor's VRAM.

Mixed sizes now pack cleanly onto whatever card has room; a request that doesn't fit waits instead of silently colliding.

The gotchas that cost me the most (airgapped cluster + shared prod):

• Rename the managed resource so HAMi's webhook never touches our H100 production node — one operator, two GPU nodes, zero blast radius.
• Per-node handover via labels — the GPU-Operator keeps the driver/toolkit, HAMi takes scheduling on just the L40S node.
• The kube-scheduler image isn't on Docker Hub → needed a mirror.
• "48 GB" is marketing; usable is 46,068 MiB. Cap your VRAM picker at 44 GiB or requests never schedule.

Where it landed

The platform is now used daily by our data-science team. Data scientists now open Coder, click New workspace, pick VS Code Desktop / Cursor / a browser IDE, choose their CPU/RAM/VRAM within their budget, and go — SSH and GPU included, fully isolated, no more stepping on each other. I can see exactly who's using what, and one workspace can't starve the box.

And it's all observable — Coder + GPU metrics in Grafana:

Open source

The Coder fork (per-user quotas for OSS) is public:

👉 github.com/ranjbaryshahab/coder — branch feat/per-user-resource-limits

Backend, migrations, the three UI pages, and the race-free quota logic are all there. The GPU enforcement lives in the workspace template (HAMi nvidia.com/gpumem requests) and the deployment repo.

What I'd tell you if you're doing this

• Keep a fork minimal and additive. One Go file, isolated migrations, template-side enforcement. Rebasing over upstream stays cheap.
• Put enforcement where the blast radius is small. Template preconditions can't corrupt core state — worst case is a clear error.
• 0 = unlimited needs a fail-closed path, or a broken token silently hands out infinite resources.

What started as a single shared SSH server evolved into a self-service GPU platform with isolated workspaces, pooled user quotas, and VRAM-aware scheduling. More importantly, the operational burden disappeared: data scientists provision their own environments, and the infrastructure enforces fairness automatically.