DEV Community

Raphael Gab-Momoh
Raphael Gab-Momoh

Posted on

Install and set up Kubeflow for ML on EKS.

#kubernetes #aws #machinelearning #eks

What is Kubeflow?

Kubeflow is like a dedicated playground for machine learning on Kubernetes. Imagine having a magic toolbox that not only helps you build and train your models but also handles all the nitty-gritty details of deploying and scaling them in a Kubernetes environment. It's like having a personal assistant that makes sure your machine learning workflows run smoothly, letting you focus on the fun part—experimenting and creating cutting-edge models.

Step 1: Set Up AWS CLI

You can install the AWS CLI from the official [documentation](https://docs.aws.amazon.com/cli/)
This command can also be used to install AWS CLI

sudo apt-get update && sudo apt-get install -y awscli
Enter fullscreen mode Exit fullscreen mode

aws cli downloaded
After the download, we need to configure it using the command

AWS configure
Enter fullscreen mode Exit fullscreen mode

This step will prompt you to enter your AWS Access Key ID, AWS Secret Access Key, default region, and default output format (optional).

account configured

Install EKS Cluster

The raw aws eks create-cluster command shown above works, but only if you already have a VPC, subnets, and an EKS service role with the right trust policy set up beforehand — that's three extra prerequisites most people don't have lying around. The faster, more common path is eksctl, which creates the cluster, the VPC, the subnets, and a managed node group in one command:

eksctl create cluster \
  --name kubeflow-eks \
  --region us-east-1 \
  --nodegroup-name ml-nodes \
  --node-type m5.xlarge \
  --nodes 2 \
  --nodes-min 2 \
  --nodes-max 4 \
  --managed
Enter fullscreen mode Exit fullscreen mode

m5.xlarge (4 vCPU, 16 GB RAM) is a reasonable floor for Kubeflow — its control-plane components (Istio, Dex, the Kubeflow dashboard, Notebook controller, Pipelines) are memory-hungry even before you run a single training job. This takes 15-20 minutes; eksctl is provisioning real VPC, subnet, and IAM resources underneath, not just the cluster.

Once it finishes, point kubectl at the new cluster:

aws eks update-kubeconfig --name kubeflow-eks --region us-east-1
kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

You should see your node group's instances in Ready state before moving on — Kubeflow's installer will fail in confusing ways if the nodes aren't ready yet.

Step 2: Install Kubeflow

Kubeflow doesn't ship as a single Helm chart — it's a collection of components (Istio, Dex, cert-manager, the Kubeflow Pipelines UI, Notebook controller, KServe, and more) glued together with Kustomize overlays in the official kubeflow/manifests repo. Clone it and check out a release tag that matches a Kubernetes version your EKS cluster actually supports:

git clone https://github.com/kubeflow/manifests.git
cd manifests
git checkout v1.9.0
Enter fullscreen mode Exit fullscreen mode

Kubeflow's components have interdependencies that kubectl apply -k alone can't always resolve on the first pass — a CRD from one component might not exist yet when another component's manifest tries to reference it. The project's own documented workaround is to retry the apply in a loop until every resource is created:

while ! kustomize build example | kubectl apply --server-side --force-conflicts -f -; do
  echo "Retrying to apply resources"
  sleep 20
done
Enter fullscreen mode Exit fullscreen mode

This can take several passes and several minutes — that's expected, not a sign something's broken, as long as the error messages are about missing CRDs rather than something else.

Step 3: Verify the Install 

kubectl get pods -n kubeflow
Enter fullscreen mode Exit fullscreen mode

Every pod should eventually reach Running. Then reach the dashboard by port-forwarding the Istio ingress gateway rather than exposing it publicly on a fresh cluster:

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80
Enter fullscreen mode Exit fullscreen mode

Open http://localhost:8080 — the default login is user@example.com / 12341234, which you should treat as a placeholder to change immediately, not a real credential to leave in place, if this cluster is anything more than a throwaway lab.

Closing Thoughts

The gap between the raw aws eks create-cluster command and a running Kubeflow dashboard is bigger than it looks from the AWS CLI reference page alone — a working node group, a Kustomize-based multi-component install with real interdependency ordering issues, and a default credential you have to remember to rotate. None of that is a knock on Kubeflow itself; it's a genuinely capable ML platform once it's up. It's just worth knowing the real shape of the setup before starting, rather than assuming one aws eks create-cluster call and a Helm install away.

Top comments (0)