DEV Community

Cover image for From Istio 1.3.x to 1.4.x after a memory leak 🚀
Julien Breux
Julien Breux

Posted on

From Istio 1.3.x to 1.4.x after a memory leak 🚀

Context

Recently I designed the new Ornikar platform using Kubernetes and Istio using Google Cloud.

Kubernetes, it's good, everyone knows and it's fantastic. OK.
But Kubernetes without service-mesh and without the super powers of Istio, it's a bit like Superman can only fly two hours a day, it's cool, but it's very limited.

Joking aside, so I set up Istio in version 1.3.x. The troubles started when I detected a memory leak on one of our gateways.

First memory leak detection

The first detection was really made thanks to external observability.
In fact, I use Pingdom to measure the uptime, availability and response time of my external endpoints.

Pingdom problem

Naturally, I first try to see if there were no false positives aside Pingdom.
But that was not it.

Memory leak detection confirmation

Then I easily found the information with our amazing stack Prometheus, Thanos and Grafana.

Grafana problem

This graph is to be correlated with the uptime graph.
We can clearly see the memory leak.
At this moment, I must act!

Problem solve

To correct the problem, a simple upgrade to the higher version of Istio was enough.

Github PR

Very simple with us because the whole infrastructure is "as-code" and with use Github and Codefresh to deploy.


To finish, I just had to restart each deployment to update Envoy in sidecar.

Grafana fix graph daily

We can see on the graph above that the upgraded version of Istio did fix the memory leaks.

Grafana fix graph

We also see that this solution is stable over time.

Some conclusions

First conclusion

When we talk to you about "observability", that should be taken seriously.
Many people tend to forget that observability is above all to have eyes on what we do.
In this case, if I had not set up Pingdom and Grafana I would never have been able to detect the failure.
And in a non-proactive way, it's the platform users who have reported the disturbances.

Second conclusion

Each time a component is installed, whether infrastructure or software.
Add the right metrics that let you know if the component is healthy or not.

Last conclusion

I like Istio, I like service mesh and I love Open Source. 🚀

Cover credit

julienberthier.org

Top comments (3)

Collapse
 
renetillamenva07 profile image
Info Comment hidden by post author - thread only accessible via permalink
Rene Tillamenva

Hello Julien

Collapse
 
Sloan, the sloth mascot
Comment deleted
Collapse
 
renetillamenva07 profile image
Rene Tillamenva

How are you doing

Some comments may only be visible to logged-in visitors. Sign in to view all comments. Some comments have been hidden by the post's author - find out more