How to upgrade to istio 1.2.4 using Helmfile

We’ve faced an interesting challenge with our istio installation. We were running the istio version 1.1.7 for a while and wanted to upgrade to the most recent stable version. This is an easy task if you follow the upgrade instructions in the istio website.

We use helmfile to manage the multiple standard applications in our cluster. In this helmfile is our istio installation. Unfortunately, I wasn’t able to find a straightforward way of running helmfile sync and having the istio installation upgraded.

I’d like to share with you our step-by-step on how we upgraded our istio version.

Helmfile

Prior to the migration, we had a helmfile that declared our istio installation like this:

helmDefaults:
  kubeContext: my_cluster
  atomic: true

repositories:
- name: incubator
  url: https://kubernetes-charts-incubator.storage.googleapis.com/
- name: istio
  url: https://storage.googleapis.com/istio-release/releases/1.1.7/charts/

releases:
  - name: istio-init
    namespace: istio-system
    chart: istio/istio-init

  - name: istio
    namespace: istio-system
    chart: istio/istio
    version: ~1.1.7
    values: [./istio/values.yaml]

Before jumping into the migration itself I recommend reading the istio upgrade notes.

Step 1 - Check your Custom Resource Definition (CRD)

If you have set up in your file the istio-init chart, like we did, then you might bump into the famous error of not being able to install the new charts because of existing CRD.

For us the simplest way was to drop some of them and let the istio chart recreate them.

Note that it’s recommended that you backup your current custom resource data, before proceeding with the upgrade.

$ kubectl get crds | grep ‘istio.io\|certmanager.k8s.io’ | cut -f1-1 -d “.” | \
    xargs -n1 -I{} sh -c “kubectl get —all-namespaces -oyaml {}; echo —“ > $HOME/ISTIO_1_0_RESTORE_CRD_DATA.yaml

Now let’s change our helmfile to pull the new istio version:

repositories:
- name: incubator
  url: https://kubernetes-charts-incubator.storage.googleapis.com/
- name: istio
  url: https://storage.googleapis.com/istio-release/releases/1.2.4/charts/

releases:
  - name: istio-init
    namespace: istio-system
    chart: istio/istio-init

  - name: istio
    namespace: istio-system
    chart: istio/istio
    version: ~1.2.4
    values: [./istio/values.yaml]

We’ve written a simple script that will try to cleanup a few things like ClusterRole, ClusterRoleBinding, Attributes, Metrics definitions and PodDisruptionBudget. Be careful here, please take some time to read the script and only delete what you can.

#!/bin/sh

# Download the most recent version of the charts
curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.2.4 sh -

cd istio-1.2.4

# install the new version’s CRDs
helm upgrade --install --force istio-init install/kubernetes/helm/istio-init --namespace istio-system

## Check if all jobs are done
kubectl get job --namespace istio-system | grep istio-init-crd

# Here is where the cleanup will start
# First we have to remove the current istio’s cluster roles
kubectl delete clusterrole istio-egressgateway-istio-system istio-ingressgateway-istio-system

# Now the cluster role bindings
kubectl delete clusterrolebinding istio-egressgateway-istio-system istio-ingressgateway-istio-system istio-kiali-admin-role-binding-istio-system

# Moving on, to remove the Kubernetes attributes
kubectl delete kubernetes attributes

# Next, we delete all metric CDRs that istio had installed
kubectl delete metric requestcount requestduration requestsize responsesize tcpbytereceived tcpbytesent tcpconnectionsopened tcpconnectionsclosed

# Finally we remove the PodDisruptionBudget
kubectl delete poddisruptionbudget istio-egressgateway istio-egressgateway istio-ingressgateway istio-pilot istio-policy istio-telemetry

Note that the commands in this script can return something like:

Error from server (NotFound): metrics.config.istio.io "requestcount" not found

This means you don’t have the CRD installed, and when you run the istio upgrade, it will install it for you.

Updating istio

Finally, run the helmfile sync command to upgrade your istio installation.

$ helmfile -f helmfile.yaml sync

To confirm if everything worked run:

$ istioctl version

client version: 1.2.4
citadel version: 1.2.4
egressgateway version: 94746ccd404a8e056483dd02e4e478097b950da6-dirty
galley version: 1.2.4
ingressgateway version: 94746ccd404a8e056483dd02e4e478097b950da6-dirty
pilot version: 1.2.4
policy version: 1.2.4
sidecar-injector version: 1.2.4
telemetry version: 1.2.4

Check your Kiali installation and new dashboard

kubectl -n istio-system port-forward (kubectl -n istio-system get pod -l app=kiali -o jsonpath='{.items[0].metadata.name}') 20001:20001

Go to Kiali Console and see your new version.

Conclusion

That is it, a simple istio upgrade. I’m sure there are better ways of doing this, but this one was the easiest and quickest that worked well for us.

If you know better ways of doing this, please share it as it will be very helpful.

Thanks 👋