From Istio 1.3.x to 1.4.x after a memory leak 🚀

Julien Breux — Wed, 08 Jul 2020 12:27:28 +0000

Context

Recently I designed the new Ornikar platform using Kubernetes and Istio using Google Cloud.

Kubernetes, it's good, everyone knows and it's fantastic. OK.
But Kubernetes without service-mesh and without the super powers of Istio, it's a bit like Superman can only fly two hours a day, it's cool, but it's very limited.

Joking aside, so I set up Istio in version 1.3.x. The troubles started when I detected a memory leak on one of our gateways.

First memory leak detection

The first detection was really made thanks to external observability.
In fact, I use Pingdom to measure the uptime, availability and response time of my external endpoints.

Naturally, I first try to see if there were no false positives aside Pingdom.
But that was not it.

Memory leak detection confirmation

Then I easily found the information with our amazing stack Prometheus, Thanos and Grafana.

This graph is to be correlated with the uptime graph.
We can clearly see the memory leak.
At this moment, I must act!

Problem solve

To correct the problem, a simple upgrade to the higher version of Istio was enough.

Very simple with us because the whole infrastructure is "as-code" and with use Github and Codefresh to deploy.

To finish, I just had to restart each deployment to update Envoy in sidecar.

We can see on the graph above that the upgraded version of Istio did fix the memory leaks.

We also see that this solution is stable over time.

Some conclusions

First conclusion

When we talk to you about "observability", that should be taken seriously.
Many people tend to forget that observability is above all to have eyes on what we do.
In this case, if I had not set up Pingdom and Grafana I would never have been able to detect the failure.
And in a non-proactive way, it's the platform users who have reported the disturbances.

Second conclusion

Each time a component is installed, whether infrastructure or software.
Add the right metrics that let you know if the component is healthy or not.

Last conclusion

I like Istio, I like service mesh and I love Open Source. 🚀

Cover credit

julienberthier.org

DEV Community: Julien Breux