Pod uses dynamic environment variable

TL; DR

This post is build-up on the Merge ConfigMap and Secrets post.It is another use of initContainers, templating and entrypoint to customize a container startup.

The premise

I worked on a Kubernetes architecture where the hosts of the cluster had several NIC Cards connected to different networks (one to expose the services, one for management, one for storage, etc.).

When to create and mount an NFS volume, the CSI driver for PowerScale/Isilon passes a client IP that is used to create the export array-side.The driver picks the IP return by the fieldRef¹ status.hostIP, as you can see here).

The problem is that IP is used to serve Kubernetes services (aka the Internal IP displayed by kubectl get node -o wide). So how to change that value to use the storage network-related IP ?

The implementation

In my setup, I know which NIC card connects to which network (in this case ens33).The patch to the native csi-isilon deployment aims to :

1. Have a simple way to get the IP address of a specific NIC card
2. Pass that information on the driver startup

The first piece of configuration is to create a custom entrypoint that will set the X_NODE_IP variable with the proper.

Here I use an ERB template in which I call the ip addr command in a subshell with %x@ @ syntax, then I extract the IP with the substring [/inet\s+(\d+(\.\d+){3})/,1]. If you use IPv6 or another NIC card you can easily adjust it at line 9 of the following snippet.

It is not displayed in the configuration above, but the ip addr command works because the Isilon Node Pod has access to the host network thanks to hostNetwork: true in its definition.

400: Invalid request

The second step is to add an initContainers to the DaemonSet to generate a new entrypoint, and then force the driver Pod to use the new entrypoint :

400: Invalid request

To apply the patch you can create the config map with :

kubectl create -f nodeip-configmap.yaml

And patch the Isilon daemon set with :

kubectl patch daemonset isilon-node -n isilon --patch "$(cat isilon-ds.patch)"

Wrap-up

The same tools (ERB, ConfigMap, initContainer, Entrypoint), can be use to tune pretty much any Kubernetes Pod deployments to customize or add extra-capabilities to your Pod startup (integration with Vault, tweak program startup, etc.).

The list of fieldRef possible values is documented here, ↩︎