Investigating a timeout issue in an Istio service mesh

Yann Soubeyrand — Thu, 07 May 2020 07:53:11 +0000

Last week, with my colleague Marc, we faced a timeout issue in an Istio service mesh. An idle PostgreSQL connection was shut down precisely one hour after it has been opened. During our investigations, I had to capture the network traffic entering and leaving the PostgreSQL client pod.

Ksniff

For this, I've been using Ksniff. This Kubernetes plugin tries to deploy a statically compiled tcpdump binary inside the pod which traffic you want to capture and then streams the captured packets to a Wireshark instance running on your workstation. This plugin is just awesome, thanks a lot to its author Eldad Rudich!

Envoy traffic interception

Istio works by injecting an Envoy proxy sidecar container inside every pod, which will intercept inbound and outbound network traffic. So when a process inside your container communicates with an external server, there are in fact two TCP connections: one between the process and Envoy, and one between Envoy and the distant server. However, in Wireshark, I saw only the packets between Envoy and the distant server and the packets from Envoy to the process but not the packets from the process to Envoy.

In the Wireshark screenshot below, 10.0.9.225 is the IP address of the process, 172.20.94.219 is the IP address of the virtual service the process communicates with, and 10.0.5.234 is the IP address of the distant (real) server backing the virtual service.

I was a bit surprised 🤔 So I searched how Istio diverted the network traffic through Envoy. It does so by adding iptables REDIRECT rules to send the traffic to port 15001 on localhost which Envoy is listening on. Indeed, I could see it in Wireshark:

But then, how can Envoy know where to forward the traffic if all the traffic it sees entering is destined for 127.0.0.1:15001? All my knowledge on IP network functioning was questioned! However, after some time spent looking for the answer, I finally found it!

When a packet hits an iptables REDIRECT rules, the kernel sets a socket option named SO_ORIGINAL_DST which contains the original packet destination. Envoy just has to read this option to decide what to do with this packet.

Conclusion

We've seen how network traffic is redirected in an Istio service mesh using iptables REDIRECT rules and SO_ORIGINAL_DST socket option.

Going back to our original issue, the investigations confirmed that the culprit was Envoy's idle timeout. From what I understand, it should be possible to configure it, but I didn't figure out yet how to do so.

Argo CD secrets management using Sops

Yann Soubeyrand — Mon, 04 May 2020 14:56:19 +0000

At Camptocamp, we've been using Argo CD for some time. It does a wonderful job of deploying Helm charts to Kubernetes clusters. But we lacked a way to manage our secrets so that we could safely commit them to our Git repositories.

Helm Secrets plugin

We knew about Helm Secrets, a Helm plugin which uses Sops under the hood to manage encrypted value files. We intended to use it with Argo CD but we faced several issues:

To render an Helm chart's manifests, Argo CD issues a helm template command. To use Helm Secrets, it would have to execute helm secrets template instead. However, there is no way to edit the Helm command used by Argo CD. One has to configure a custom plugin. This has the drawback of loosing Helm specific features (like specifying value files or parameters).
Helm Secrets works by decrypting secret value files to temporary files on disk before actually calling Helm. While not a big concern, it's not a security best practice. Secrets should never touch the disk when they are unencrypted.
Helm Secrets seemed unmaintained at the time.

Helm Sops

Since we only needed a small subset of Helm Secrets' functionalities, we decided to write a small Go binary which would transparently wrap Helm so that we could substitute the Helm binary with it in the Argo CD Docker image. Let's introduce Helm Sops!

camptocamp / helm-sops

Helm Sops

Helm Sops is a Helm wrapper which decrypts Sops encrypted value files before invoking Helm. It does so by using named pipes to pass cleartext value files to Helm in order to avoid secrets being written to disk in clear.

Installation

Prerequisites

Helm is needed for Helm Sops to work. Follow the instructions here to install it.

Getting Helm Sops binary

Helm Sops releases

Helm Sops released binaries can be downloaded from GitHub.

Building from sources

Helm Sops can be built using the go build command.

Deploying Helm Sops

Deploy Helm Sops executable (helm-sops) in a directory present in the PATH. When invoking Helm Sops, it will look for a Helm executable named helm in the PATH.

Alternatively, Helm Sops executable can be renamed helm before deploying it. When invoked as helm, Helm Sops will look for a Helm executable named _helm…

View on GitHub

Helm Sops is loosely inspired by Helm Secrets but uses named pipes instead of temporary files to pass decrypted secret value files to Helm.

To use it:

deploy our custom Argo CD Docker image integrating Helm Sops to your Kubernetes cluster,
install Sops on your workstation and create a .sops.yaml file to specify the encryption method to use (GPG key, AWS KMS key, etc),
use Sops to create a file named secrets.yaml and put your Helm secret values inside,
git commit,
git push,
finally, visit the Argo CD web interface to deploy your Helm application \o/
(optional) install Helm Sops on your workstation to be able to use the Argo CD CLI with encrypted value files.

Full step by step instructions can be found here.

Security Considerations

While Sops uses battle tested cryptography, the way we use it with Argo CD makes it vulnerable to the confused deputy problem. Someone which can deploy applications using Argo CD and has access to an encrypted value file but cannot decrypt it, can deploy a new application accompanied by the encrypted value file and make Argo CD decrypt it for them.

This limitation can be problematic in multi-tenant environments. A way to solve it is to make use of AEAD. AWS KMS offers encryption context to this end.

Sops already supports AWS KMS encryption context but it relies on the one specified in the metadata of the encrypted file. We would need Argo CD to be able to overwrite this encryption context when decrypting secret value files so that it cannot be confused anymore. We'd be happy to work with Sops maintainers to add this possibility 😉

DEV Community: Yann Soubeyrand

Investigating a timeout issue in an Istio service mesh

Ksniff

Envoy traffic interception

Conclusion

Argo CD secrets management using Sops

Helm Secrets plugin

Helm Sops

camptocamp / helm-sops

Helm Sops

Installation

Prerequisites

Getting Helm Sops binary

Helm Sops releases

Building from sources

Deploying Helm Sops

Security Considerations