DEV Community: coulof

Implement Tanzu on private networks with VyOS

coulof — Mon, 04 Jan 2021 08:00:00 +0000

TL; DR

With a virtual router (here VyOS), and proper routing table, it is possible to implement VMware Tanzu on private networks without NSX-T.

The premise

I needed to evaluate VMware Tanzu basic capabilities with CSI driver for PowerScale. To install Tanzu, also named Workload Management in vCenter, I followed that video and these 1, 2, 3 posts.

As a prerequisite, Tanzu needs two networks at least, one for the management cluster (also named Supervisor cluster) and one for the workload (i.e. the on-demand clusters to run workload).

In my lab, I only have one routable VLAN on the network with limited IPs. For other activities, like Anthos validation, I use a single private network behind a NAT ; unfortunately that configuration is not enough here.

For Tanzu, I decided to use three networks:

The frontend network (10.247.247.0/24), which is routable to the external world and will have the Load-Balancer VIP
The management network (10.0.0.0/24), for the vSphere management of the supervisor cluster
The workload network (10.0.1.0/24), to host all Tanzu clusters

The problem is, I do not have an NSX-T license to manage these private networks. Luckily for us, once again Linux is here to save the day.

The implementation

The trick here is to use a virtual machine that will act as a router for the different networks. My choice went to VyOS, a Debian-base distro design for routing. It can be used as a firewall, VPN, do QoS, etc. The configuration is done with commands similar to what you can have with Cisco IOS or other proprietary networking devices.

In this case, I use only the routing and NAT features. For free, VyOS will route between the connected interface, so after the NICs configuration I just had to configure the NAT for my two private networks.

The final configuration is pretty much:

400: Invalid request

With that configuration:

any VMs deployed with Tanzu can connect to the external world through the NAT via the Dell LAN segment
any VMs with a single NIC can talk to the same segment via the DSwitch or to the other network via the VyOS VM

Ping issue

Tanzu explicitly states that Supervisor and Workload clusters must be able to connect to the HA Proxy data plane.

On my setup I used VMware HA Proxy and deployed the image with three networks. The Data plane API management listens on the Management network and default port of 5556.

Supervisor cluster deployment went well but during the workload cluster creation I got the following error:

unexpected error while reconciling control plane endpoint for my-cluster: failed to reconcile loadbalanced endpoint for WCPCluster csi/my-cluster: failed to get control plane endpoint for Cluster csi/my-cluster: Virtual Machine Service LB does not yet have VIP assigned: VirtualMachine Service LoadBalancer does not have any Ingresses

As a matter of fact I could :

ping any IPs from the Load-Balancer
ping the Load-Balancer IP from a node on the same network
ping any IP Supervisor IP from the workload node and the other way around
but I could not ping a Management proxy IP which from a workload node

The problem here the ICMP echo requests goes though the router but the response is issued directly on the DSwitch :

To solve that, the trick is to make sure all the request to HAProxy are answered through the router. To do so there are three things to do :

Set the workload IP with a /32 mask to force to use this communication for this device only in : /etc/systemd/network/10-workload.network
Update the gateway of the Management NIC to for the two networks management & workload in : /etc/systemd/network/10-management.network
Add a default route for the external work with a higher weight in : /etc/systemd/network/10-frontend.network

Below are the NICs configuration you can edit then apply with a systemctl restart systemd-networkd

400: Invalid request

400: Invalid request

400: Invalid request

Now the ICMP echo reply goes through the same route and responds :

Conclusion

For basic features (like NAT, static or dynamic routing, NAT, DHCP, Firewall, etc.) VyOS is an excellent virtual router that can get you a long way if, like me, you miss NSX-T license.

Use cert-manager with for Karavi Observability

coulof — Tue, 29 Dec 2020 08:00:00 +0000

TL; DR

Check out how works Prometheus metrics & Grafana dashboard for CSI PowerFlex from that karavi observability video.

Learn more about karavi and karavi-observability from their respective repositories. And use these configurations to use cert-manager as a certificate source for the karavi components.

The premise

Dell Technologies launched the project Karavi in December 2020 with the objective to complement functionalities not covered by the CSI specification.

Karavi focuses in three domains :

Observability
Data mobility
Security

The first project brought to life as a tech-preview is the metrics & topology collection for PowerFlex Kubernetes volumes.

The video below will give you an introduction on the architecture, use-cases and what is coming in the future.

At the time of the publication of this post, the communication between the OpenTelemetry component and Prometheus server, and, the communication between the Karavi topology component and Grafana are secured with TLS by default :

As indicated in the documentation, you have to supply certificates to the helm installer :

helm install karavi-observability dell/karavi-observability -n karavi --create-namespace \
    --set-file karaviTopology.certificateFile=<location-of-karavi-topology-certificate-file> \
    --set-file karaviTopology.privateKeyFile=<location-of-karavi-topology-private-key-file> \
    --set-file otelCollector.certificateFile=<location-of-otel-collector-certificate-file> \
    --set-file otelCollector.privateKeyFile=<location-of-otel-collector-private-key-file> \
    --set karaviMetricsPowerflex.powerflexPassword=<base64-encoded-password> \
    --set karaviMetricsPowerflex.powerflexUser=<base64-encoded-username> \
    --set karaviMetricsPowerflex.powerflexEndpoint=https://<powerflex-endpoint>

This command assumes you generated the different certificates with correct DNS names. For example with openssl:

openssl req -new -newkey rsa:2048 -nodes -keyout karavi-topology.key -out karavi-topology.csr -subj '/CN=karavi-topology'
openssl x509 -req -in karavi-topology.csr -signkey karavi-topology.key -out karavi-topology.crt
openssl req -new -newkey rsa:2048 -nodes -keyout otel-collector.key -out otel-collector.csr -subj '/CN=otel-collector'
openssl x509 -req -in otel-collector.csr -signkey otel-collector.key -out otel-collector.crt

Obviously, anytime the certificate expires or you move the components to different namespace you have to manually generate new certificates.

So let us see how to take advantage of cert-manager to automate that process.

The implementation

cert-manager is the goto Kubernetes certificate management tool. It bundled with some distribution like GKE. If you need to install it few kubectl apply will do the job.

The first step to use certificate delivered by cert-manager is to configure an Issuer. The easiest is to use a SelfSigned without a dedicated certificate authority :

400: Invalid request

The next step is to define the parameters for your certificate (size, rotation, DNS names, etc.) ; here is, for example, what we have for the otel-collector :

400: Invalid request

The same for karavi-topology can be downloaded here.

The last step is to patch the deployment (otel-collector and karavi-topology) to use the new certificate stored as a secret.

You can choose to do it directly in the karavi-metrics-powerflex & karavi-topology Helm charts ; or patch it in Kubernetes if already deployed.

For example, with the otel-collector we have:

400: Invalid request

For karavi-topology, the patch looks like this:

400: Invalid request

The last step is to apply the patch to the deployments:

kubectl patch deployments.apps -n karavi otel-collector --patch "$(cat otel-cert_manager.patch)"
kubectl patch deployments.apps -n karavi karavi-topology --patch "$(cat karavi-topology-cert_manager.patch)"

Go further

If you want to learn more about Karavi observability, you can check that other blog post or ask for help on the Dell container community forum.

PersistentVolume static provisioning

coulof — Sun, 01 Nov 2020 07:00:00 +0000

TL; DR

In this article, we will discuss and present a script (ingest-static-pv.sh) for Persistent Volume static provisioning of Dell CSI Drivers for PowerMax, PowerStore, PowerScale, PowerFlex, and Unity.

The premise

As part of an OpenShift migration project from one cluster to a new one, we wanted to ease the transition by loading the existing persistent storage in the new cluster.

TL; DR
The premise
Concepts for static provisioning
- PersistentVolume static provisioning
- reclaimPolicy
- Provisioner
- volumeHandle
- PowerMax
- PowerStore
- PowerScale
- PowerFlex
- Unity
ingest-static-pv.sh ; step by step with PowerMax
- Get the volume details
- Confirm the Provisioner and StorageClass
- Prepare the volumeHandle value
- Dry-run
- And run it for real !
- Map the PVC to a Pod
Conclusion

Concepts for static provisioning

Before we dive into the implementation, let us review some Kubernetes concepts that back the static provisioning.

`PersistentVolume` static provisioning

The static provisioning, as opposite to dynamic provisioning, is the action of creating PersistentVolume upfront to they are ready to be consumed later by a PersistentVolumeClaim.

`reclaimPolicy`

Each StorageClass has a reclaimPolicy that tells Kubernetes what to do with a volume once it is released from it usage. Every Dell drivers support the Retain and Delete policies.

If set to Delete, the Dell drivers will remove the volume and other objects (like the exports, quotas, etc.) on the backend array. If set to Retain, only the Kubernetes objects (PersistentVolume, VolumeAttachment) will be deleted. With Retain, it is up to the storage administrator to cleanup or not.

The reclaimPolicy is inherited in the PersistentVolume under the attribute persistentVolumeReclaimPolicy. It is possible to change a persistentVolumeReclaimPolicy at any point in time with a kubectl edit pv [my_pv] or command like :

kubectl patch pv [my_pv] -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

`Provisioner`

The provisioner gives the driver to be used for the volume provisioning. The Dell drivers defaults are:

NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
isilon (default) csi-isilon.dellemc.com Delete Immediate true 15d
powermax (default) csi-powermax.dellemc.com Delete Immediate true 46d
powermax-xfs csi-powermax.dellemc.com Delete Immediate true 46d
powerstore (default) csi-powerstore.dellemc.com Delete WaitForFirstConsumer true 22h
powerstore-nfs csi-powerstore.dellemc.com Delete WaitForFirstConsumer true 22h
powerstore-xfs csi-powerstore.dellemc.com Delete WaitForFirstConsumer true 22h
unity (default) csi-unity.dellemc.com Delete Immediate true 20d
unity-iscsi csi-unity.dellemc.com Delete Immediate true 20d
unity-nfs csi-unity.dellemc.com Delete Immediate true 20d
vxflexos (default) csi-vxflexos.dellemc.com Delete WaitForFirstConsumer true 29d
vxflexos-xfs csi-vxflexos.dellemc.com Delete WaitForFirstConsumer true 29d

csi-powerstore and csi-powermax drivers allow you to tweak the provisioner. It enables you to spin multiple instances of the driver in the same cluster and, therefore, connect to different array or create more granular StorageClass.

To do so you can, during the installation, edit the variable driverName for PowerStore and the variable customDriverName for PowerMax.

`volumeHandle`

The volumeHandle is the unique identifier of the volume created on the storage backend and returned by the CSI driver during the volume creation. Any call by the CSI drivers on a volume will reference that unique id.

What follows is probably the most important piece of this article. We will list for each Dell driver how the volumeHandle is construct and how to load an existing volume as a PV.

PowerMax

The PowerMax volumeHandle consists of:

    csi-<Cluster Prefix>-<Volume Prefix>-<Volume Name>-<Symmetrix ID>-<Symmetrix Vol ID>
    1 2 3 4 5 6

csi- is an hardcoded value
Cluster Prefix , is the value given during the installation of the driver in the variable named clusterPrefix.
Volume Prefix , is the value given during the installation of the driver in the variable named volumeNamePrefix.
Volume Name , is a random UUID given by the controller sidecar.
Symmetrix ID is the twelve characters long PowerMax identifier.
Symmetrix Vol ID is the LUN identifier.

The volumeHandle can be found in Unisphere for PowerMax under Storage > Volumes. Highlighting the volume will populate the window on the left showing this information.

The construction of the volumeHandle is given here and you can test your volumeHandle is valid on rubular or directly from STORAGECLASS=powermax VOLUMEHANDLE=help ingest-static-pv.sh.

PowerStore

The PowerStore volumehandle is just volume’s or NFS share ID. For example:

    volumeHandle: 880fb26c-9a94-4565-9e6e-c0bf2b029ecc

The easiest to get the identifier, is to connect to the WebUI and copy the UUID from the URL like here:

You can test your volumeHandle is valid a valid UUID on rubular or directly from STORAGECLASS=powerstore VOLUMEHANDLE=help ingest-static-pv.sh.

PowerScale

The PowerScale/Isilon volumeHandle is the volume name, the Export ID and the Access Zone separated by =_=_=. For example:

    volumeHandle: PowerScaleStaticVolTest=_=_=176=_=_=System

This information can be found in the PowerScale OneFS GUI under Protocols > Unix sharing (NFS) > NFS Exports

The construction of the volumeHandle is given here and you can test your volumeHandle is valid on rubular or directly from STORAGECLASS=isilon VOLUMEHANDLE=help ingest-static-pv.sh.

PowerFlex

The PowerFlex/VxFlexOS volumehandle is just volume’s ID. For example:

    volumeHandle: ecdbd5bd0000000a

The volume’s ID can be found in the VxFlex OS GUI under Frontend > Volumes. Click on the volume then the Show property sheet icon. The ID will be listed under Identity > ID ; or you can check it with : scli --query_all_volumes

The volume ID is missing from the 3.5 Web UI.

Unity

The Unity volumeHandle is the volume name or filesystem name, the protocol (iSCSI, FC or NFS), and the CLI ID separated by dashes -.

    volumeHandle: csiunity-fde5df688a-iSCSI-fnm00000000000-sv_16
    volumeHanble: csiunity-46d0385efd-FC-fnm00000000000-sv_938
    volumeHandle: csiunity-a7bb9ee130-NFS-fnm00000000000-fs_5

This information can be found in the Unisphere for Unity ; under Storage then Block or File. The CLI ID may not be visible by default. To view the CLI ID click Customize your view > Columns check the CLI ID check box.

The construction of the volumeHandle is given here and you can test your volumeHandle is valid on rubular or directly from:

STORAGECLASS=unity VOLUMEHANDLE=help ./ingest-static-pv.sh

`ingest-static-pv.sh` ; step by step with PowerMax

The ingest-static-pv.sh script (available on https://github.com/coulof/dell-csi-static-pv) will ease the loading of existing volume as a PV or both PV and PVC. It is designed to take the minimal required fields by the provisioner to work.

The example below will go through every step to statically load a PowerMax LUN in Kubernetes.

Get the volume details

For the first step you can connect to Unisphere and get the Volume Identifier (here csi-fdg-pmax-9e954fcdfa), the Symmetrix ID (000197900704), the Symmetrix Vol ID (0017C) and the Capacity (8GB) which are mandatory to provision the PV.

Confirm the Provisioner and StorageClass

You can validate the provisioner for your driver with kubectl get sc -o custom-columns=NAME:.metadata.name,PROVISIONER:.provisioner

NAME PROVISIONER
powermax csi-powermax.dellemc.com
powermax-xfs csi-powermax.dellemc.com

In this case, we will use the default one given by the StorageClass name.

Prepare the `volumeHandle` value

As mentioned in the chapter above, the construction of volumeHandle is the key piece to map a Kubernetes PersistentVolume to its actual volume in the backend storage array.

To get help on the volumeHandle for the storage system you need to load, you can use : STORAGECLASS=powermax VOLUMEHANDLE=help ./ingest-static-pv.sh ; or here, refer to the PowerMax chapter of this blog.

For the help on all the parameters, run ./ingest-static-pv.sh without any environment variable.

Thanks to the Unisphere values we found above, we must define the value of VOLUMEHANDLE as csi-fdg-pmax-9e954fcdfa-000197900704-0017C.

Tip: in case you need to create a volume from scratch and you need to cook a PV name with a UUID you can use the following command: UUID=$(uuidgen) && echo ${UUID: -10}

Dry-run

The script will output the definitions on stdout only by default.

In the example below, we decided to create both the PV and the PVC (cf. PVCNAME variable) in the default namespace.

To do so we execute:

STORAGECLASS=powermax VOLUMEHANDLE=csi-fdg-pmax-9e954fcdfa-000197900704-0017C PVNAME=pmax-9e954fcdfa SIZE=8 PVCNAME=testpvc ./ingest-static-pv.sh

which results in :

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pmax-9e954fcdfa
spec:
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: powermax
  volumeMode: Filesystem
  csi:
    driver: csi-powermax.dellemc.com
    volumeHandle: csi-fdg-pmax-9e954fcdfa-000197900704-0017C
    fsType: ext4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: testpvc
  namespace: default
spec:
  volumeName: pmax-9e954fcdfa
  storageClassName: powermax
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi

If you run the script without PVNAME the PersistentVolume definition only will be displayed.

And run it for real !

To load the definition in your Kubernetes instance you must specify DRYRUN=false.

Map the PVC to a Pod

Now that the PVC is Bound to the PV, we can use it in a Pod.

If the CSI driver succeeds to mount the volume to the Pod, we will have VolumeAttachment object displayed with kubectl get volumeattachments:

NAME ATTACHER PV NODE ATTACHED AGE
csi-39d511b28dc4490d47ede7f573d7616ae05addda6db9f2a8b6aecf7649a00722 csi-powermax.dellemc.com pmax-9e954fcdfa lsisfty94.lss.emccom true 82s

And the events attached to the Pod will show a SuccessfulAttachVolume message like below :

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 28s default-scheduler Successfully assigned default/testpod-script to lsisfty94.lss.emc.com
Normal SuccessfulAttachVolume 18s attachdetach-controller AttachVolume.Attach succeeded for volume "pmax-9e954fcdfa"

Conclusion

The static provisioning proved to be very useful for migration projects.

The script ingest-static-pv.sh is planned to be used in a couple of projects. One for a migration from an existing OpenShift cluster to a new one ; the second for a migration of an OpenShift storage backend from Pure storage to PowerMax.

Feel free to try it and open tickets to the repo .

Home dir automation with Ansible PowerScale / Isilon

coulof — Sat, 29 Aug 2020 07:00:00 +0000

TL;DR

ansible-isilon eases the admin tasks on Isilon / PowerScale ; watch how cool it can be on Youtube and how to use it below.OverlayFS is great but has some limitations for some use-cases ; UnionFS is not dead !

The premise
The implementation
- Install Ansible modules for PowerScale/Isilon
- The files
- List usage in Ansible
- UnionFS
- File system removal
Video

The premise

In my old days at the university, I used to work Sun Ray thin client (imagine the evolution between VT100 to modern VDI). Students and teachers were all connected to the same SPARC server to work. Each of us had its own home directory accessible from the NFS server.

More than 15 years later, enterprises of any size still use home directories on NFS for their users !

In the following article, we will show how to use Ansible to manage home directories hosted on a PowerScale array in a university.The predicate is that Active Directory is the reference for the userbase. Each LDAP user can be either in the student group or the teacher group.Any student or teacher in AD must have his homedir in PowerScale and be accessible via NFS exports. Any student who is no longer enrolled and not in AD will have thier homedir removed.

The ansible playbook will :

get the list of students and teachers from AD
create a unix home directory in PowerScale/Isilon for each user
set different quotas if the user is a student or a teacher
have daily snapshots of the home directories with varying policies of retention if for the students and teachers
mount the home directories in a list of Unix server
cleanup the home directories of students that are not in the AD anymore

The implementation

In this chapter I will not detail all the tasks as most of them are self-explanatory, but, describe a few tips & tricks that can be reused in other playbooks.

Install Ansible modules for PowerScale/Isilon

The Product Guide documents the module installation and usage (equivalent to ansible-doc dellemc_isilon_[module]).

This example comes with a Dockerfile that has the required dependencies to run the playbook.

As the ansible-isilon is very specific about Isilon SDK version, the most important line is :

RUN pip3 install isi-sdk-8-1-1 pywinrm && \
    git clone https://github.com/dell/ansible-isilon.git

Once docker build-ed, you can execute the playbook with it with :

podman run --security-opt label=disable -e ANSIBLE_HOST_KEY_CHECKING=False \
           -v ~/.ssh/id_rsa.emc.pub:/root/.ssh/id_rsa.pub -v ~/.ssh/id_rsa.emc:/root/.ssh/id_rsa \
           -v "$(pwd)"/homedir/:/ansible-isilon \
           -ti docker.io/coulof/ansible-isilon:1.1.0 ansible-playbook \
           -i /ansible-isilon/hosts.ini /ansible-isilon/create_homedir_for_ad_users_in_isilon.yml

Note that on my Fedora 32 machine, the --security-opt label=disable is mandatory to be able to mount the volumes.

The files

To use the playbook, you will have to update a couple of files:

hosts.ini ; which has the inventory of Unix and Domain Controller
credentials-isi.yml ; which has the details of the PowerScale
create_homedir_for_ad_users_in_isilon.yml ; which is the playbook with all the tasks, where you have to update the variables base_path and nfs_server_ip in several sections to point to the PowerScale path and IP.

List usage in Ansible

The first tip is in task Get userbase from Active Directory with :

    - set_fact:
        students_list: "{{members_students_group.members | list}}"
        teachers_list: "{{members_teachers_group.members | list}}"

The set_fact creates two lists of users that will be reused across the playbook.With the object list, we can loop through and execute the same task for each user as done in the FS creation :

      dellemc_isilon_filesystem:
        <<: *isi_connection_vars
        path: "{{base_path}}/students/{{item}}"
        ...
        state: 'present'
      loop: "{{ hostvars['devconad.com']['students_list'] }}"

Or make it easy to find orphan homedirs by playing with list operations when listing unix mounted dirs :

    - name: Capture files in path and register
      shell: >
        ls -1 /mnt/nfs_students
      register: students_home_dir
      run_once: True
    - set_fact:
        orphan_home_dirs: "{{students_home_dir.stdout_lines | list | difference(hostvars['devconad.com']['students_list'])}}"

UnionFS

To stick with the usual /home/<username> file system hierarchy, I wanted to mount the students and teachers sub-dirs within the same /home and keep the write in the lower dirs as follow :

/mnt/nfs_teachers/ /mnt/nfs_students/ /home
├── alice ├── carol ├── alice
└── bob └── dan ├── bob
                                                ├── carol
                                                └── dan

The capability of writing in lowerdirs live is available in AuFS and UnionFS but not in the very popular OverlayFS.

As stated by the Kernel documentation:

Changes to the underlying filesystems while part of a mounted overlay filesystem are not allowed.

There are plenty of discussions about that topic on Stackoverflow.

To achieve it I used unionfs-fuse which is available from Ubuntu repo or CentOS third-party repo. The obvious advantage of Filesystem in Userspace is that I won’t need to recompile the Linux kernel to use it. In the /etc/fstab we can use unionfs# to mount a FUSE filesystem :

        line: "unionfs#/mnt/nfs_students=RW:/mnt/nfs_teachers=RW /home/ fuse cow 0 0"

File system removal

It is possible to remove PowerScale/Isilon file system with the Ansible directive :

    - name: Remove Filesystem and Quota for missing students from AD
      dellemc_isilon_filesystem:
        path: "{{base_path}}/students/{{item}}"
        quota:
          quota_state: absent
        state: absent

Note that by design, the Ansible module will only remove the directory if empty.If you need to remove a non-empty directory, you have to issue REST call directly.

Video

For a live demo, check the video here: https://www.youtube.com/watch?v=RF5WoeRry1k&list=PLbssOJyyvHuVXyKi0c9Z7NLqBiDiwF1eA&index=2

K8s mount PV with SELinux

coulof — Tue, 14 Jul 2020 07:00:00 +0000

TL; DR

If you want to use Kubernetes with SELinux and mount PersistentVolume, you have to make sure your mounted FS has labels. You can do it with the mountOptions -o context="system_u:object_r:container_var_lib_t:s0" and if your driver doesn’t support it, you can write an SELinux policy like this one.

The premise

The problem came from a customer who is using Docker Enterprise with CSI Driver for VxFlexOS (now PowerFlex), and SELinux enforced on the nodes.

Anytime a Pod tried to write data on the PersistentVolume, we had Permission denied error from the OS.

SELinux relies on file label (sometimes call context) in the file extended attributes to apply its policies. For examples below :

ls -lZd /root /home /etc
drwxr-xr-x. root root system_u:object_r:etc_t:s0 /etc
drwxr-xr-x. root root system_u:object_r:home_root_t:s0 /home
dr-xr-x---. root root system_u:object_r:admin_home_t:s0 /root

By default, on a newly formatted and mounted FS the files are unlabeled :

mkfs.ext4 /dev/loop0
mount -o loop /dev/loop0 /media/xxx
ls -Z
drwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 xxx

On the customer setup, it was forbidden to do any action on an unlabeled file.

How to get the new FS labeled ?

The typical way to relabel a directory is to use the command restorecon against the mount point, which will restore the security context and label.Because we are in a containerized environment with volumes dynamically mounted by Kubernetes, this is not realistic to expect the restorecon command to be executed each time a volume is mounted.

Another option, especially suitable to big FileSystems with many files, is to can create a file named .autorelabel at the root level, so it forces a relabel on the mountpoint. Slightly better but, still, not feasible in a dynamic environment like ours.

A better option is to mount the FS with the context option. That option is my favorite ❣

Unfortunately, the DellEMC CSI drivers don’t have the mountOption capability at the time of that post. That feature is on the roadmap, but in the meantime, I needed a plan B.

The last possibility and the one we implemented is to write a specific policy to allow containers to manipulate unlabeled files and directories.

Since SELinux follows a model of the least-privilege (aka you can’t do anything except if explicitly allowlisted), the challenge was to have all the syscalls a container needs to do their job.

To get inspiration, I hijacked policies given in the SELinux Project and came up with that list : class file { create open getattr setattr read write append rename link unlink ioctl lock };

The full policy is :

module vxflexos-cni 1.0;

require {
    type unlabeled_t;
    type container_t;
    class file { create open getattr setattr read write append rename link unlink ioctl lock };
    class dir { add_name create getattr ioctl link lock open read remove_name rename reparent rmdir search setattr unlink write };
}

#!!!! WARNING: 'unlabeled_t' is a base type.
allow container_t unlabeled_t:dir { create open getattr setattr read write link unlink rename search add_name remove_name reparent rmdir lock ioctl };
allow container_t unlabeled_t:file { create open getattr setattr read write append rename link unlink ioctl lock };

To compile it, you will need the SELinux Devel package (in Fedora : dnf install selinux-policy-devel.noarch will do) and then compile the policy with : make -f /usr/share/selinux/devel/Makefile. To install the newly compiled policy run : semodule -i vxflexos-cni.pp.

Wrap-up

mountOptions capability is coming with every Dell Technologies CSI driver in the incoming months.

The Gentoo website is a gold mine of information for SELinux ! To understand better the issue, I mostly read these pages on Labels, the Tutorial to create a policy, and SELinux Project Policies.

Pod uses dynamic environment variable

coulof — Fri, 19 Jun 2020 07:00:00 +0000

TL; DR

This post is build-up on the Merge ConfigMap and Secrets post.It is another use of initContainers, templating and entrypoint to customize a container startup.

The premise

I worked on a Kubernetes architecture where the hosts of the cluster had several NIC Cards connected to different networks (one to expose the services, one for management, one for storage, etc.).

When to create and mount an NFS volume, the CSI driver for PowerScale/Isilon passes a client IP that is used to create the export array-side.The driver picks the IP return by the fieldRef¹ status.hostIP, as you can see here).

The problem is that IP is used to serve Kubernetes services (aka the Internal IP displayed by kubectl get node -o wide). So how to change that value to use the storage network-related IP ?

The implementation

In my setup, I know which NIC card connects to which network (in this case ens33).The patch to the native csi-isilon deployment aims to :

Have a simple way to get the IP address of a specific NIC card
Pass that information on the driver startup

The first piece of configuration is to create a custom entrypoint that will set the X_NODE_IP variable with the proper.

Here I use an ERB template in which I call the ip addr command in a subshell with %x@ @ syntax, then I extract the IP with the substring [/inet\s+(\d+(\.\d+){3})/,1]. If you use IPv6 or another NIC card you can easily adjust it at line 9 of the following snippet.

It is not displayed in the configuration above, but the ip addr command works because the Isilon Node Pod has access to the host network thanks to hostNetwork: true in its definition.

400: Invalid request

The second step is to add an initContainers to the DaemonSet to generate a new entrypoint, and then force the driver Pod to use the new entrypoint :

400: Invalid request

To apply the patch you can create the config map with :

kubectl create -f nodeip-configmap.yaml

And patch the Isilon daemon set with :

kubectl patch daemonset isilon-node -n isilon --patch "$(cat isilon-ds.patch)"

Wrap-up

The same tools (ERB, ConfigMap, initContainer, Entrypoint), can be use to tune pretty much any Kubernetes Pod deployments to customize or add extra-capabilities to your Pod startup (integration with Vault, tweak program startup, etc.).

The list of fieldRef possible values is documented here, ↩︎

Gitlab CI/CD with CSI PowerMax

coulof — Fri, 29 May 2020 07:00:00 +0000

TL; DR

Watch the basic deployment & snapshot-based deployment videos on Youtube and check the .gitlab-ci-cd.yaml on Gitlab.

The premise

For the first release of the CSI Driver for PowerMax we wanted to show how the PV dynamic provisioning and snapshot capabilities.

To present a realistic scenario, we used Gitlab CI/CD, its Kubernetes runner, and the CSI Driver off course.

The application itself is a fork of the VueJS example app TODO, which we modified to use Sinatra as an API provider and SQLite to store the TODOs.

The implementation

The concept is:

the master branch corresponds to the latest image and is the production environment
anytime we push a new branch to GitLab we:
- build the image
- take a snapshot of PV from production
- create an environment to access the new app
new commits on the branch will keep using their own environment with an independent PV
on branch merge:
- the dedicated environment and related PV are deleted
- the production is redeployed with the latest image

Most of the magic on the storage layer happens in the PVC, Snap definitions, and with the Helm variables.

we can see that, if the branch is the latest, we deploy a dedicated volume (i.e. only the first time). For every other branches we start from a snapshot taken at the time of the first branch creation.

Under the scene, we will have two independent volumes in PowerMax. For more deep-dive on PowerMax SnapVX (i.e. PowerMax local replicas) you can check that white paper.

400: Invalid request

To avoid, the storage box to be bloated by the project, we also defined a resource Quota on the namespace.

400: Invalid request

Limitations

The current version of the CSI driver (v1.2) the snapshot API is v1alpha1 and not compatible with Kubernetes v1.17 and beyond.

A snapshot is only accessible from the same namespace and cannot restore a volume on a different namespace.

Other tips

One of the tricks is is to put the Gitlab variable CI_COMMIT_SHORT_SHA in the helm template ; that way, we make sure it is re-proceed and therefore redeployed with the latest build by Helm.

Finally, to save some time in building the images, I used local gems.

Videos

For a live demo, check the videos here:

Merge ConfigMap and Secrets

coulof — Thu, 28 May 2020 07:00:00 +0000

TL; DR

To use a Secret value within a ConfigMap you can use an initContainer to call a template engine.

The premise

In the previous post, I presented how to use kubernetes-event-exporter with Elasticsearch.

One of the problems I faced is that the tool doesn’t follow the configuration guidelines from the 12-factor app methodology.

That is to say, we have to put the credentials in the YAML configuration rather than in environment variable :-(

As for Kubernetes, it doesn’t allow us to mix Secret values within ConfigMap.

The solution

To solve that issue, we have 3 components:

the Secret as-is
the ConfigMap which will have the configuration as a template
the initContainer that will merge the two

SecretMap

The secret comes from the ECK Operator ; we can get it with kubectl get secrets quickstart-es-elastic-user -o yaml :

apiVersion: v1
kind: Secret
data:
  elastic: YU80bnc4NzZWMXBWMThOZThqOFlnOE1r

ConfigMap

In the case of k8s-events-reporting the ConfigMap looks like this:

400: Invalid request

The important piece is the last line, <=% %> is the erb syntax to call ruby code, and ENV is a hash to acces the environment variables.

Why ERB?

Because I ♥ Ruby !
Because the erb command line comes with the ruby docker official image (there is no need for a custom Dockerfile and therefore no maintenance)

initContainer

Last but not least, here is the Deployment with the initContainer config that will craft the config file from both the Secret passed as an environment variable and the ConfigMap template. The event-exporter container can later use that file.

400: Invalid request

K8s events monitoring

coulof — Wed, 27 May 2020 07:00:00 +0000

TL; DR

For events monitoring you can use this ready-to-go elastic + kibana + k8s-event-exporter stack from : k8s-events-reporting

The premise

As part of an internal project at Dell, I needed to measure the time between the different PVC and PV states (Pending, Bound, Mounted, etc.) through their lifecycles.

For example, how long it takes for a PVC to be bounded to is volume? The same question between a volume request and mounted to a Pod.

The idea is to measure the performance of our different drivers (csi-powermax, csi-isilon, csi-vxflexos, csi-powerstore) in various scenarios (e.g. one pod needs one hundred volumes, one hundred pods writing to the same volume, evaluate the impact of the volume size, etc.).

The PV/PVC/Pod status is available with kubectl get pv,pvc,po and you can track the lifecycle through Kubernetes events with kubectl get events.

How to get the events?

A simple search on kubernetes and monitoring will return tons of resources from open-source or proprietary projects to collect metrics and logs from Kubernetes. Unfortunately, they mostly focus on metrics collection and container logs.

For my little project, I just needed to get the details of the events.

My first idea was to dump the events kubectl get events -A -o json and load them somewhere like SQLite or excel. Being a nerd, I think we can do better.

The Kubernetes events reporting stack is composed of :

kubernetes-event-exporter for the event collection
elasticsearch for the database
kibana for the reporting engine

kubernetes-event-exporter

This utility developed by OpsGenie will basically dump the events and forward them to different destinations (sink in their terminology). It has extra features like filtering the type of events or choose the fields you want to forward.

The deployment of that component has been tweaked and templatized in a helm chart like the rest of the stack.

The magic to connect that component to elasticsearch will be discussed in the dedicated post for ConfigMap and Secrets.

ElasticSearch & Kibana

The rest of the stack uses well-known components from Elastic. The deployment uses the Elastic Cloud on Kubernetes.

Thanks to the operator framework, we can very easily configure and deploy a secured version of ElasticSearch and Kibana. This is done by the install.sh with :

kubectl apply -f https://download.elastic.co/downloads/eck/1.1.1/all-in-one.yaml

After you run and successful launch the install.sh and your stack is ready ; you can load the kibana dashboard with :

curl -X POST "localhost:5601/api/kibana/dashboards/import?exclude=index-pattern" -H 'Content-Type: application/json' -d @kibana-dashboard.json

Of course, you have to adjust the URL & credentials; the result will look like this :

From now on, you can visualize the kubernetes events and keep them longer than the 1-hour default in your cluster. The provided dashboard is PV/PVC/Pods centric but any events are collected so you can tweak and hack the Kibana dashboard ;-)

Wrap-up

That Kubernetes events monitoring stack have been used and tested for one-shot statistics and analytics. Nevertheless, the component and approach can fit other use-cases like observability, alerting, and monitoring.

There is more to say about the kubernetes-even-exporter configuration in the context of Kubernetes. It will be addressed in more detail in the ConfigMap and Secrets post.

DEV Community: coulof

Implement Tanzu on private networks with VyOS

TL; DR

The premise

The implementation

Ping issue

Conclusion

Use cert-manager with for Karavi Observability

TL; DR

The premise

The implementation

Go further

PersistentVolume static provisioning

TL; DR

The premise

Concepts for static provisioning

PersistentVolume static provisioning

reclaimPolicy

Provisioner

volumeHandle

PowerMax

PowerStore

PowerScale

PowerFlex

Unity

ingest-static-pv.sh ; step by step with PowerMax

Get the volume details

Confirm the Provisioner and StorageClass

Prepare the volumeHandle value

Dry-run

And run it for real !

Map the PVC to a Pod

Conclusion

Home dir automation with Ansible PowerScale / Isilon

TL;DR

Table of Contents

The premise

The implementation

Install Ansible modules for PowerScale/Isilon

The files

List usage in Ansible

UnionFS

File system removal

Video

K8s mount PV with SELinux

TL; DR

The premise

How to get the new FS labeled ?

Wrap-up

Pod uses dynamic environment variable

TL; DR

The premise

The implementation

Wrap-up

Gitlab CI/CD with CSI PowerMax

TL; DR

The premise

The implementation

Other tips

Videos

Merge ConfigMap and Secrets

TL; DR

The premise

The solution

SecretMap

ConfigMap

Why ERB?

initContainer

K8s events monitoring

TL; DR

The premise

How to get the events?

kubernetes-event-exporter

ElasticSearch & Kibana

Wrap-up

`PersistentVolume` static provisioning

`reclaimPolicy`

`Provisioner`

`volumeHandle`

`ingest-static-pv.sh` ; step by step with PowerMax

Prepare the `volumeHandle` value