If you have ever tried to hand out Kubernetes clusters on GPU servers to several teams or customers, you know the problem. Provisioning machines, forming clusters, keeping them isolated, upgrading them and replacing broken nodes all turn into glue scripts very quickly.
NVIDIA Kubernetes Engine (NVKE) is an open source project that tackles this. It is a management plane you install on your own NVIDIA GPU infrastructure, and it creates and operates Kubernetes clusters on demand. Think of it as the thing that lets a platform team run something like a managed Kubernetes service, but on bare metal that you own.
Heads up: NVKE is currently a preview. APIs, configuration and behavior can change between releases, and the project does not recommend it for production workloads. This article covers what that means in practice further down.
The short version
- You run the NVKE management plane. Your users run workloads on the clusters it creates.
- Every workload cluster gets its own Kubernetes control plane, its own etcd and its own node pools.
- Users talk to their cluster with plain
kubectland the Kubernetes API. They do not need to know NVKE exists. - Machines are provisioned through NVIDIA Infra Controller (NICo), which is the only supported infrastructure provider in the preview.
- License is Apache-2.0, and the code is written in Go.
Who this is for
NVKE is aimed at platform teams: the people who own a pool of GPU servers and need to give other teams or customers their own isolated Kubernetes clusters.
If you only want to run workloads on a single GPU cluster, this is probably more than you need. If you are building an internal "Kubernetes as a service" on top of GPU hardware, keep reading.
How it is structured
An NVKE installation has two kinds of Kubernetes cluster.
1. The management cluster. This runs the NVKE management plane and the infrastructure controllers. You create it with the nvkectl command line tool, which uses a temporary seed cluster to bootstrap it. Installing the management plane services is a separate step after that.
2. Workload clusters. These are the clusters your users actually run applications on. The management plane creates and operates them through NICo, Cluster API and a node agent.
The README describes NVKE as a self-hosting factory for Kubernetes clusters, and explicitly says it is not a Kubernetes distribution and not a Cluster API flavour. A throwaway seed cluster stands up the management cluster, and the management plane then provisions and manages many isolated workload clusters.
Here is the flow in plain terms:
seed cluster (temporary)
|
v
management cluster (runs the NVKE management plane)
|
| requests machines through NICo
v
workload clusters (own control plane, etcd, node pools)
|
v
your users, using kubectl
What NVKE does for you
Creates independent workload clusters. You request a cluster through the NVKE API or nvkectl. NVKE provisions the machines through NICo and forms a Kubernetes cluster with its own control plane.
Manages the cluster lifecycle. You can create, scale and delete node pools, upgrade Kubernetes and replace nodes. In the preview, some of these steps are manual and have limits, so read the node pool guide before relying on them.
Establishes node identity. Nodes attest themselves through SPIFFE/SPIRE, using their TPM, before they receive join credentials. This means a machine has to prove what it is before it can join a cluster. Encryption of etcd data is available per cluster.
Connects builds to deployment. A companion project, NVIDIA Kubernetes Recipes (NVKR), builds the node images and NVKE components from pinned inputs. NVKE deploys those artifacts, while NVKR owns their build and release evidence.
Try it without GPU hardware
You do not need GPUs or an NVIDIA account to evaluate NVKE. The local quickstart creates a development management plane and a workload cluster on simulated machines.
What you need:
- A Linux host or VM, amd64 or arm64
- At least 6 CPUs, 12 GiB of free memory and 100 GiB of free disk
- Docker, virtual TPM support and the other tools listed in the guide
Follow the local quickstart through workload cluster creation, and confirm the cluster's nodes report Ready. That is your success check.
On macOS, use the local site guide instead. It runs the evaluation inside a Colima virtual machine.
Install it on real infrastructure
For a real installation you bring your own:
- Machines
- Networking
- Node images
- Registries
- Access to NICo
Start with Plan your installation. It covers the prerequisites and the order of operations: prepare nodes, bootstrap the management cluster, then install the management plane services.
The platform and version reference lists the validated combinations and pinned dependencies. Check it before you pick hardware or versions.
Build from source
If you want to read the code, hack on it or contribute, you can build everything locally.
Requirements:
- Go 1.27.3, the version pinned in
go.work - GNU Make 4 or later
- Public Go modules reachable on the first build
- Optional: Flox, which supplies the repository's development tools (and GNU Make 4 on macOS)
git clone https://github.com/dsx-ai-factory/nvke.git
cd nvke
make build
This builds the Go workspace and writes seven binaries to bin/:
| Binary | Notes |
|---|---|
agent |
The node agent, which runs on Linux |
updater |
|
controller |
|
nvkectl |
The command line tool |
capd-reboot-controller |
|
nvke-os-image-check |
|
management-engine |
Building does not start a management plane or create a cluster. It only produces the binaries.
The repository is a multi-module Go workspace, and the workspace guide explains how the modules and outputs fit together.
What developers should know before starting
This is the section to read twice.
1. It is a preview, so expect change
APIs, configuration and behavior can change between releases. Do not plan a production rollout around the current behavior. See Preview status and known limitations.
2. API authentication is off by default
After installation, the API uses plain HTTP and authentication is turned off. You must configure API authentication and protected access before you let any callers in. Do not expose a fresh install to a network you do not control.
3. The management cluster does not look after itself
The management cluster does not upgrade, scale or replace itself. There is also no management-plane upgrade procedure and no backup and restore procedure yet. Plan for that operational gap.
4. FDE node images are for evaluation only
The FDE node images provide no data-at-rest confidentiality. Treat them as an evaluation tool, not a security feature.
5. NICo is the only infrastructure provider
For the preview, NVIDIA Infra Controller (NICo) is the sole compatible infrastructure provider. If your hardware is managed some other way, NVKE does not support it yet.
6. Some lifecycle steps are manual
Node replacement and Kubernetes upgrades have manual steps and preview limits. The node pool guide documents them.
A sensible path for a first evaluation
- Read What is NVKE to understand the operator workflow.
- Run the local quickstart on a Linux VM and get a workload cluster to
Ready. - Read the Architecture doc to see the data flows between components.
- Read the known limitations page and decide whether the preview gaps matter for your use case.
- If you have GPU hardware and NICo access, move on to planning a real installation.
Where to find the docs
| I want to... | Go to |
|---|---|
| Prepare nodes and bootstrap the management cluster | Bootstrap the management cluster |
| Install NVKE services | Deploy path README |
| Build node images and get release artifacts | Get NVKE artifacts from NVKR |
| Create and manage customer clusters | Create a workload cluster and Manage node pools |
| Give a customer access | Access a workload cluster |
| Inspect cluster health and telemetry | Observe NVKE and its clusters |
The project also publishes documentation at docs.nvidia.com/nvke.
Closing up
NVKE gives platform teams a way to run a managed Kubernetes service on their own NVIDIA GPU hardware, with isolated clusters, TPM-based node attestation and a lifecycle managed through an API and nvkectl. It is early, and the preview limitations are real, especially the disabled API authentication and the missing management-plane upgrade and backup procedures.
Repository: github.com/dsx-ai-factory/nvke
Top comments (0)