Context
Recently I designed the new Ornikar platform using Kubernetes and Istio using Google Cloud.
Kubernetes, it's good, everyone knows and it's fantastic. OK.
But Kubernetes without service-mesh and without the super powers of Istio, it's a bit like Superman can only fly two hours a day, it's cool, but it's very limited.
Joking aside, so I set up Istio in version 1.3.x
. The troubles started when I detected a memory leak on one of our gateways.
First memory leak detection
The first detection was really made thanks to external observability.
In fact, I use Pingdom to measure the uptime, availability and response time of my external endpoints.
Naturally, I first try to see if there were no false positives aside Pingdom.
But that was not it.
Memory leak detection confirmation
Then I easily found the information with our amazing stack Prometheus, Thanos and Grafana.
This graph is to be correlated with the uptime graph.
We can clearly see the memory leak.
At this moment, I must act!
Problem solve
To correct the problem, a simple upgrade to the higher version of Istio was enough.
Very simple with us because the whole infrastructure is "as-code" and with use Github and Codefresh to deploy.
To finish, I just had to restart each deployment to update Envoy in sidecar.
We can see on the graph above that the upgraded version of Istio did fix the memory leaks.
We also see that this solution is stable over time.
Some conclusions
First conclusion
When we talk to you about "observability", that should be taken seriously.
Many people tend to forget that observability is above all to have eyes on what we do.
In this case, if I had not set up Pingdom and Grafana I would never have been able to detect the failure.
And in a non-proactive way, it's the platform users who have reported the disturbances.
Second conclusion
Each time a component is installed, whether infrastructure or software.
Add the right metrics that let you know if the component is healthy or not.
Last conclusion
I like Istio, I like service mesh and I love Open Source. 🚀
Top comments (3)
Hello Julien
How are you doing
Some comments may only be visible to logged-in visitors. Sign in to view all comments. Some comments have been hidden by the post's author - find out more