Mitz

Posted on May 23, 2020

Istio 1.6.0 Canary Upgrade Trial

#istio

While I am playing with Istio 1.5, it has become 1.6. Time flies...

https://istio.io/news/releases/1.6.x/announcing-1.6/

Istio LTS has 3 months release cycle. And the previous LTS becomes EOL after 3 months from the new LTS is released. Therefore I think it's important how to make it easy to handle upgrade.

The good news is that Istio 1.6.0 supports Canary upgrade of Istio itself. It sounds interesting to me.

https://istio.io/blog/2020/multiple-control-planes/

So I would like to try it.

Environment

I prepared a GKE cluster.

❯ kubectl get nodes
NAME                                       STATUS   ROLES    AGE   VERSION
gke-cluster-1-default-pool-019aa731-6ztc   Ready    <none>   21m   v1.16.8-gke.15
gke-cluster-1-default-pool-019aa731-spbj   Ready    <none>   21m   v1.16.8-gke.15
gke-cluster-1-default-pool-019aa731-ztjz   Ready    <none>   21m   v1.16.8-gke.15

And prepared config.yaml file to install Istio which is a custom configuration for Istio install. I simply added Stackdriver integration, though it's not related to this article because I don't use anything of Stackdriver this time:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: default

  addonComponents:
    prometheus:
      enabled: false

  values:
    global:
      proxy:
        tracer: "stackdriver"
      logAsJson: true
    telemetry:
      v2:
        enabled: true
        prometheus:
          enabled: false
        stackdriver:
          enabled: true
          logging: true
          monitoring: true
          # topology: true
          configOverride: {}

Canary upgrade

I followed this article:

https://istio.io/docs/setup/upgrade/

It introduces 2 types of upgrade:

Canary upgrades
In place upgrades

I will try canary upgrades this time from 1.5.4 to 1.6.0.

Install Istio 1.5

At first, I install 1.5.4:

❯ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.5.4 sh - 

❯ cd istio-1.5.4

# I saved config.yaml with the contents I wrote above.
❯ bin/istioctl manifest apply -f config.yaml                   
- Applying manifest for component Base...
✔ Finished applying manifest for component Base.
- Applying manifest for component Pilot...
✔ Finished applying manifest for component Pilot.
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
  Waiting for resources to become ready...
- Applying manifest for component IngressGateways...
✔ Finished applying manifest for component IngressGateways.


✔ Installation complete

Then, install our favorite bookinfo app:

# Enable sidecar auto injection
❯ kubectl label namespace default istio-injection=enabled
namespace/default labeled

# Deploy Bookinfo services
❯ kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
service/details created
serviceaccount/bookinfo-details created
deployment.apps/details-v1 created
service/ratings created
serviceaccount/bookinfo-ratings created
deployment.apps/ratings-v1 created
service/reviews created
serviceaccount/bookinfo-reviews created
deployment.apps/reviews-v1 created
deployment.apps/reviews-v2 created
deployment.apps/reviews-v3 created
service/productpage created
serviceaccount/bookinfo-productpage created
deployment.apps/productpage-v1 created

# Deploy Gateway
❯ kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml
gateway.networking.istio.io/bookinfo-gateway created
virtualservice.networking.istio.io/bookinfo created

# Get IP address of LB
❯ export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

# Open the app with browser (My machine is Ubuntu)
❯ xdg-open http://$INGRESS_HOST/productpage

I can see a familiar bookinfo page.

Upgrade to Istio 1.6

Download 1.6.0:

❯ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.6.0 sh - 
❯ cd istio-1.6.0
# I saved config.yaml with the contents I wrote above.

Currently, my Istio version is 1.5.4:

❯ bin/istioctl version                                                     
client version: 1.5.4
control plane version: 1.5.4
data plane version: 1.5.4 (7 proxies)

Run upgrade

Then, upgrade the Istio following this document:

https://istio.io/docs/setup/upgrade/

hm... though it says the command is like this:

$ istioctl install --set revision=canary

I guess I should specify my config.yaml file, because it would be deafult profile otherwise. So I checked the reference of the command:

https://istio.io/docs/reference/commands/istioctl/#istioctl-install

It has -f option, so I can try this:

❯ bin/istioctl install --set revision=canary -f config.yaml
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Installation complete

And it seems worked. The result of watching pod is as follows:

❯ kubectl -n istio-system get pods -w
NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-7f76b9fc9f-ffr4j   1/1     Running   0          4m29s
istiod-6558b7b7f4-9d2dk                 1/1     Running   0          4m50s

# I ran install command here

istiod-canary-848948494f-vrpz8          0/1     Pending   0          0s
istiod-canary-848948494f-vrpz8          0/1     Pending   0          0s
istiod-canary-848948494f-vrpz8          0/1     ContainerCreating   0          0s
istiod-canary-848948494f-vrpz8          0/1     Running             0          9s
istiod-canary-848948494f-vrpz8          1/1     Running             0          10s
istio-ingressgateway-ccbb848f5-b94mh    0/1     Pending             0          0s
istio-ingressgateway-ccbb848f5-b94mh    0/1     Pending             0          0s
istio-ingressgateway-ccbb848f5-b94mh    0/1     ContainerCreating   0          1s
istio-ingressgateway-ccbb848f5-b94mh    0/1     Running             0          8s
istio-ingressgateway-ccbb848f5-b94mh    1/1     Running             0          10s
istio-ingressgateway-7f76b9fc9f-ffr4j   1/1     Terminating         0          5m37s
istio-ingressgateway-7f76b9fc9f-ffr4j   0/1     Terminating         0          5m43s
istio-ingressgateway-7f76b9fc9f-ffr4j   0/1     Terminating         0          5m44s
istio-ingressgateway-7f76b9fc9f-ffr4j   0/1     Terminating         0          5m44s

Finally, it became like this:

❯ kubectl get pods -n istio-system
NAME                                   READY   STATUS    RESTARTS   AGE
istio-ingressgateway-ccbb848f5-b94mh   1/1     Running   0          7m16s
istiod-6558b7b7f4-9d2dk                1/1     Running   0          13m
istiod-canary-848948494f-vrpz8         1/1     Running   0          7m29s

What does it mean?

It adds istiod-canary keeping the original istiod as it is.
Regarding istio-ingressgateway, it stops the original one and deploys new one.

I see... I looked into istio-ingressgateway and found it became 1.6.0:

❯ kubectl -n istio-system get pods istio-ingressgateway-ccbb848f5-b94mh -o yaml | grep image: 
    image: docker.io/istio/proxyv2:1.6.0
    image: istio/proxyv2:1.6.0

The version command shows this result:

❯ bin/istioctl version
client version: 1.6.0
pilot version: 1.5.4
istiod version: 1.6.0
data plane version: 1.5.4 (6 proxies), 1.6.0 (1 proxies)

Ah, it was like this before:

❯ bin/istioctl version                                     
client version: 1.5.4
control plane version: 1.5.4
data plane version: 1.5.4 (7 proxies)

It was separated from control plane version into 2 parts: pilot version and istiod version.

Btw, it is istio-ingressgateway which uses 1.6.0 in data plane version. It seems intended to update ingress-gateway:

https://github.com/istio/istio/issues/23923#issuecomment-630892188

At this point

Data Plane remains 1.5.4. It should be 1.5.4, even though I restart the pods. Let's try:

❯ kubectl rollout restart deployment
deployment.apps/details-v1 restarted
deployment.apps/productpage-v1 restarted
deployment.apps/ratings-v1 restarted
deployment.apps/reviews-v1 restarted
deployment.apps/reviews-v2 restarted
deployment.apps/reviews-v3 restarted

❯ bin/istioctl version   
client version: 1.6.0
pilot version: 1.5.4
istiod version: 1.6.0
data plane version: 1.5.4 (6 proxies), 1.6.0 (1 proxies)

Looks good. They keep using 1.5.4.

Upgrade Data Plane

As I tried now, each Pod still uses the previous version when we just install new istiod-canary. We need to modify namespace label in order for the Pods to start using the new version.

❯ kubectl label namespace default istio-injection- istio.io/rev=canary
namespace/default labeled

❯ kubectl get namespaces default --show-labels
NAME      STATUS   AGE   LABELS
default   Active   48m   istio.io/rev=canary

I removed istio-injection label from the default namespace, where I installed bookinfo, and added istio.io/rev=canary label. The reason why we need to remove istio-injection label from the namespace is it has higher precedence than revision label for backward compatibility.

Now they should start using new version when I restart them:

❯ kubectl rollout restart deployment
deployment.apps/details-v1 restarted
deployment.apps/productpage-v1 restarted
deployment.apps/ratings-v1 restarted
deployment.apps/reviews-v1 restarted
deployment.apps/reviews-v2 restarted
deployment.apps/reviews-v3 restarted

Wait for a while the Pods restart and check version:

❯ bin/istioctl version
client version: 1.6.0
pilot version: 1.5.4
istiod version: 1.6.0
data plane version: 1.6.0 (7 proxies)

It is updated, yay! I also checked one of the container and found it used newer version of istiod:

❯ bin/istioctl proxy-config endpoints details-v1-7bc48f795d-2fwzr --cluster xds-grpc -ojson | grep hostname
                "hostname": "istiod-canary.istio-system.svc",

Out of curiosity

I wondered whether I can write revision option in my config.yaml file rather than setting it as a command line option.

As a result, it works with this config:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: default

  revision: "v160"

  addonComponents:
    prometheus:
      enabled: false

  values:
# omitted

At the first time, I wrote v1.6.0 but it didn't work, so I guess we can't use period as a revision name:

❯ bin/istioctl install -f config.yaml
Error: failed to apply manifests: errors occurred during operation

My thoughts

(1) How to clean up old version?

I can still see old version resources such as istiod, sidecar-injector etc. But I'm not sure how to clean them up.

❯ kubectl get pods -n istio-system
NAME                                   READY   STATUS    RESTARTS   AGE
istio-ingressgateway-ccbb848f5-b94mh   1/1     Running   0          62m
istiod-6558b7b7f4-9d2dk                1/1     Running   0          68m
istiod-canary-848948494f-vrpz8         1/1     Running   0          62m

❯ kubectl get mutatingwebhookconfigurations
NAME                                                 CREATED AT
istio-sidecar-injector                               2020-05-23T13:57:59Z
istio-sidecar-injector-canary                        2020-05-23T14:03:42Z
pod-ready.config.common-webhooks.networking.gke.io   2020-05-23T13:49:48Z

Actially, it showed errors when I tried to install another istiod after I removed the original istiod. It seems to me the validation hook keep using old istiod and shows error?

❯ bin/istioctl install -f config.yaml                               
✔ Istio core installed                                                                                                                                                           
2020-05-23T15:31:02.866368Z     error   installer       failed to create "EnvoyFilter/istio-system/metadata-exchange-1.6-v160-2": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: no endpoints available for service "istiod"
2020-05-23T15:31:03.008138Z     error   installer       failed to create "EnvoyFilter/istio-system/stackdriver-filter-1.6-v160-2": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: no endpoints available for service "istiod"
2020-05-23T15:31:03.164115Z     error   installer       failed to create "EnvoyFilter/istio-system/tcp-metadata-exchange-1.6-v160-2": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: no endpoints available for service "istiod"
✘ Istiod encountered an error: failed to create "EnvoyFilter/istio-system/metadata-exchange-1.6-v160-2": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: no endpoints available for service "istiod"
failed to create "EnvoyFilter/istio-system/stackdriver-filter-1.6-v160-2": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: no endpoints available for service "istiod"
failed to create "EnvoyFilter/istio-system/tcp-metadata-exchange-1.6-v160-2": Internal error occurred: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: no endpoints available for service "istiod"

(2) Should I keep using the name "canary"?

I used "canary" for the revision name following the tutorial. But it seems I need to keep using the name "canary" for this version.

So I feel I should use a name which I can keep using it for the version such as v160 etc?

(3) The unit isn't pod but namespace

Currently, it only supports to change namespace label for the canary upgrade. I thought it would be convenient if I can upgrade pods one by one, but hm... it might be enough good.

Summary

It is a good direction that we can deploy multiple versions of istiod and gradually change to use the new version.
I need to remember IngressGateway is upgraded at once without having multiple versions.
I'm looking forward to seeing new information about cleanup. I hope it is added in 1.7.

And I started to subscribe this issue:

https://github.com/istio/istio/issues/23889

It was fun!

DEV Community