DEV Community: Carlos Camacho

KubeInit External access for OpenShift/OKD deployments with Libvirt

Carlos Camacho — Tue, 25 Aug 2020 00:00:00 +0000

In this post, it will be described the basic network architecture when OKD is deployed using KubeInit in a KVM host.

TL;DR;

We will describe how to extend the basic network configuration to provide external access to the cluster services by adding an external IP to the service machine.

Initial hypervisor status

We check both the routing table and the network connections in the hypervisor host.

[root@nyctea ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default _gateway 0.0.0.0 UG 100 0 0 eno1
10.19.41.0 0.0.0.0 255.255.255.0 U 100 0 0 eno1
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0


[root@nyctea ~]# nmcli con show
NAME UUID TYPE DEVICE
System eno1 162499bc-a6fa-45db-ba76-1b45f0be46e8 ethernet eno1   
virbr0 4ba12c69-3a8b-42e8-a9dd-bc020fdc1a90 bridge virbr0
eno2 e19725f2-84f5-4f71-b300-469ffc99fd99 ethernet --     
enp6s0f0 7348301f-8cae-4ab1-9061-97d7a344699c ethernet --     
enp6s0f1 8a96c226-959a-4218-b9f7-c3ab6ee3d02b ethernet --

As it is possible to see there are two physical network interfaces (eno1, and eno2) for which only one is actually connected.

Initial network architecture

The following picture represents the default network layout for a usual deployment.

The default deployment will install a multi-master cluster, with one worker node (up to 10). From the above figure is possible to see:

All cluster nodes are connected to the 10.0.0.0/24 network. This will be the cluster management network, and the one will use to access the nodes within the hypervisor.

The 10.0.0.0/24 network is defined as a Virtual Network Switch implementing both NAT and DCHP for any interface connected to the kimgtnet0 network.

All bootstrap, master, and worker nodes are installed with Fedora CoreOS as is the required OS for OKD > 4.

The services machine has installed CentOS 8 with BIND, HAProxy, and NFS.

Using DHCP, we assign the following IP mapping based on the MAC address of each node (defined in the Ansible inventory).

 # Master
 okd-master-01 ansible_host=10.0.0.1 mac=52:54:00:aa:6c:b1
 okd-master-02 ansible_host=10.0.0.2 mac=52:54:00:59:0e:e4
 okd-master-03 ansible_host=10.0.0.3 mac=52:54:00:b4:39:45

 # Worker
 okd-worker-01 ansible_host=10.0.0.4 mac=52:54:00:61:22:5a
 okd-worker-02 ansible_host=10.0.0.5 mac=52:54:00:21:fd:fd
 okd-worker-03 ansible_host=10.0.0.6 mac=52:54:00:4c:0a:81
 okd-worker-04 ansible_host=10.0.0.7 mac=52:54:00:54:ff:ac
 okd-worker-05 ansible_host=10.0.0.8 mac=52:54:00:4a:6b:f6
 okd-worker-06 ansible_host=10.0.0.9 mac=52:54:00:40:22:52
 okd-worker-07 ansible_host=10.0.0.10 mac=52:54:00:6c:0a:03
 okd-worker-08 ansible_host=10.0.0.11 mac=52:54:00:0b:14:f8
 okd-worker-09 ansible_host=10.0.0.12 mac=52:54:00:f5:6e:e5
 okd-worker-10 ansible_host=10.0.0.13 mac=52:54:00:5c:26:4f

 # Service
 okd-service-01 ansible_host=10.0.0.100 mac=52:54:00:f2:46:a7

 # Bootstrap
 okd-bootstrap-01 ansible_host=10.0.0.200 mac=52:54:00:6e:4d:a3

The previous deployment can be used for any purpose but it has one limitation, this limitation is that the endpoints do not have external access. This means that i.e. https://console-openshift-console.apps.watata.kubeinit.localcan not be accessed from anywhere instead of the hypervisor itself.

Extending the basic network layout

Now it will be described a simple way to provide external access to the cluster public endpoints published in the service machine.

Requirements

An additional IP address to be mapped to the services machine from an external location.
Creating a network bridge to slave the interface used for external access.

If a user has one extra IP (public or private) it will be enough to configure remote access to the cluster endpoints.

As long as we have an extra IP it does not matter how many physical interfaces we have, as we can have multiple IP addresses configured using a single physical NIC.

New network layout

This is the resulting network architecture to access remotely our freshly installed OKD cluster.

As is visible in the above figure there is an extra connection to the service machine, connected directly to the virtual bridge slaving a physical interface.

Our development environment has only one network card connected, in this case after we create the main switch and slave the network device, it will lose the assigned IP automatically. Do not try this using a shell as you will get dropped.

How to enable the external interface

To deploy this architecture please follow the next steps:

Create a virtual bridge slaving the selected physical interface.
Adjust the deployment command.
Run KubeInit.
Adjust your local Domain Name System (DNS) resolver.

Step 1 (creating the virtual bridge)

Using CentOS8 cockpit

We create an initial bridge using the CentOS cockpit, after losing the IP it will be recovered/reconfigured automatically(don’t try this from the CLI as you will lose access).

In this case,

Create a bridge called kiextbr0 connected to eno1:

Click on: Networking -> Add Bridge

Then adjust the bridge configuration options (bridge name and the interface to slave).

Write: kiextbr0 as the bridge name, and select your network interface eno1.

Go to the dashboard and verify that everything is OK.

Check that the bridge is created correctly and has the IP configured correctly.

Manual bridge creation

As an example, you can run these steps by the CLIadjusting your interface and bridge names accordingly.

nmcli connection add ifname br0 type bridge con-name br0
nmcli connection add type bridge-slave ifname enp0s25 master br0
nmcli connection modify br0 bridge.stp no
nmcli connection delete enp0s25
nmcli connection up br0

NOTE: If you have only one interface the connection will be dropped and you will lose connectivity.

Checking the system status

We check again the system status:

[root@nyctea ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default _gateway 0.0.0.0 UG 425 0 0 kilocbr0
10.19.41.0 0.0.0.0 255.255.255.0 U 425 0 0 kilocbr0
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0

[root@nyctea ~]# nmcli con show
NAME UUID TYPE DEVICE  
kilocbr0 1c4d60a3-06a7-429f-a689-ffba5a49efbb bridge kilocbr0
System eno1 162499bc-a6fa-45db-ba76-1b45f0be46e8 ethernet eno1    
virbr0 4ba12c69-3a8b-42e8-a9dd-bc020fdc1a90 bridge virbr0  
eno2 e19725f2-84f5-4f71-b300-469ffc99fd99 ethernet --      
eno3 65be9380-980b-4237-b27c-2479e8f8535d ethernet --      
eno4 9f5afe2d-6166-4197-a23f-e64c3b1b5ab2 ethernet --      
enp6s0f0 7348301f-8cae-4ab1-9061-97d7a344699c ethernet --      
enp6s0f1 8a96c226-959a-4218-b9f7-c3ab6ee3d02b ethernet --

We can see we have the new bridge created successfully and it has the IP address also configured correctly.

Step 2 (adjusting the deployment command)

There are a few variables that need to be adjusted in order to successfully configure the external interface.

These variables are defined in the okd playbook (the location of these variables will change)but not their name.

The meaning of the variables are:

kubeinit_bind_external_service_interface_enabled: true - This will enable the Ansible configuration of the external interface, the BIND update, and the additional interface in the service node.

kubeinit_bind_external_service_interface.attached: kiextbr0 - This is the virtual bridge where we will plug the eth1 interface of the services machine. The bridge MUST be created first and slaving the physical interface we will use.

kubeinit_bind_external_service_interface.dev: eth1 - This is the name of the external interface we will add to the services machine.

kubeinit_bind_external_service_interface.ip: 10.19.41.157 - The external IP addressof the services machine.

kubeinit_bind_external_service_interface.gateway: 10.19.41.254 - The gateway IP address of the services machine.

kubeinit_bind_external_service_interface.netmask: 255.255.255.0 - The network mask of the external interface of the services machine.

After we configure correctly the previous variables we can proceed to run the deployment command.

Step 3 (run the deployment command)

Now we deploy as usual KubeInit:

Remember that you can execute this deployment command before creating the bridge with the CentOS cockpit, the bridge creation has no impact on how we deploy KubeInit.

ansible-playbook \
    -v \
    --user root \
    -i ./hosts/okd/inventory \
    --become \
    --become-user root \
    -e "{ \
      'kubeinit_bind_external_service_interface_enabled': 'true', \
      'kubeinit_bind_external_service_interface': { \
        'attached': 'kiextbr0', \
        'dev': 'eth1', \
        'ip': '10.19.41.157', \
        'gateway': '10.19.41.254', \
        'netmask': '255.255.255.0' \
      } \
    }" \
    ./playbooks/okd.yml

Step 4 (adjust your resolv.conf)

You must reach the cluster external endpoints by DNS, this means, the dashboard and any other application deployed(you can add entries for any registry pointing to the service machine but this can be cumbersome).

For example, configure your local DNS resolver to point to 10.19.41.157

 [ccamacho@localhost]$ cat /etc/resolv.conf
 nameserver 10.19.41.157
 nameserver 8.8.8.8

After that, you should be able to access the cluster without any issue and use it for any purpose you have.

Voilà!

Final considerations

Some of the very interesting changes in BIND is how we manage both external and internal views.

In this case, we have an internal and external view that will behave differently depending on where the requests are originated from.

If a DNS request is created trough the cluster’s external interface, the reply will be created based on the external view, in this case we only reply with the external HAProxy endpoints related to the services node, thus, we will only reply with 10.19.41.157 as it is the only that needs to be presented externally.

The end

If you like this post, please try the code, raise issues, and ask for more details, features, or anything that you feel interested in. Also, it would be awesome if you become a stargazer to catch updates and new features.

This is the main project repository.

Happy KubeIniting!

Updated 2020/08/25: First version (draft).

Updated 2020/08/26: Published.

A review of the MachineConfig operator

Carlos Camacho — Sun, 16 Aug 2020 00:00:00 +0000

The latest versions of OpenShift rely on operators to completely manage the cluster and OS state,this state includes for instance, configuration changes and OS upgrades.For example, to install additional packages or changing any configuration file to execute whatever task isrequired, the MachineConfig operator should be the one in charge of applying these changes.

These configuration changes are executed by an instance of the‘openshift-machine-config-operator’ pod, which after this new state is reachedthe updated nodes will be automatically restarted.

There are several mature and production-ready technologies allowing to automate and applyconfiguration changes to the underlying infrastructure nodes, like, Ansible,Helm, Puppet, Chef, and many others, yet, the MachineConfig operator forceusers to adopt this new method and pretty much discard any previously developedautomation infrastructure.

The MachineConfig operator is more than installing packages and updating configuration files

I see this MachineConfig operator as a finite state machine where it’s representeda cluster-wide specific sequential logic to ensure that the cluster’s state is preserved and consistent.This notion has several and very powerful benefits, like making the cluster resilient to failuresdue to unfulfilled conditions in each of the sub-stages part of this finite state machine workflow.

For instance, let the following example show a practical and objective applicationof the benefits of this approach.

We assume the following architecture reference:

3 master nodes.
1 worker node.
The master nodes are not schedulable.

This is a quite simple multi-master deployment with a single worker nodefor development purposes, before the cluster was deployed themaster nodes were set as “mastersSchedulable: False” by running"sed -i 's/mastersSchedulable: true/mastersSchedulable: False/' install_dir/manifests/cluster-scheduler-02-config.yml".Now, after deploying the cluster and executing a configuration change in the worker node it willfail, and let investigate why.

The following yaml file will be applied, which it’s content is correct and it should work out of the box:

cat << EOF > ~/99_kubeinit_extra_config_worker.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  creationTimestamp: null
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-kubeinit-extra-config-worker
spec:
  osImageURL: ''
  config:
    ignition:
      config:
        replace:
          verification: {}
      security:
        tls: {}
      timeouts: {}
      version: 2.2.0
    networkd: {}
    passwd: {}
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,IyEvdXNyL2Jpbi9iYXNoCnNldCAteAptYWluKCkgewpzdWRvIHJwbS1vc3RyZWUgaW5zdGFsbCBwb2xpY3ljb3JldXRpbHMtcHl0aG9uLXV0aWxzCnN1ZG8gc2VkIC1pICdzL2VuZm9yY2luZy9kaXNhYmxlZC9nJyAvZXRjL3NlbGludXgvY29uZmlnIC9ldGMvc2VsaW51eC9jb25maWcKfQptYWluCg==
          verification: {}
        filesystem: root
        mode: 0755
        path: /usr/local/bin/kubeinit_kubevirt_extra_config_script
EOF
oc apply -f ~/99_kubeinit_extra_config_worker.yaml

The defined MachineConfig object will create a file in/usr/local/bin/kubeinit_kubevirt_extra_config_script that once executed it willinstall a package and disable SElinux in the worker nodes.

Now, let’s check the state of the worker machine config pool.

oc get machineconfigpool/worker

This is the result:

NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
worker rendered-worker-a9.. False True True 1 0 0 1 12h

Now it is possible to depict that the operator state is degraded and there is not much more information about it.Let’s get the status of the machine-config pods.

kubectl get pod -o wide --all-namespaces | grep machine-config

It is possible to see that all pods are running without issues.

openshift-machine-config-operator etcd-quorum-guard-7bb76959df-5bj7g 1/1 Running 0 11h 10.0.0.2 okd-master-02 <none> <none>
openshift-machine-config-operator etcd-quorum-guard-7bb76959df-jdtbv 1/1 Running 0 11h 10.0.0.3 okd-master-03 <none> <none>
openshift-machine-config-operator etcd-quorum-guard-7bb76959df-sndb2 1/1 Running 0 11h 10.0.0.1 okd-master-01 <none> <none>
openshift-machine-config-operator machine-config-controller-7cbb584655-bfjmh 1/1 Running 0 11h 10.100.0.20 okd-master-01 <none> <none>
openshift-machine-config-operator machine-config-daemon-ctczg 2/2 Running 0 12h 10.0.0.3 okd-master-03 <none> <none>
openshift-machine-config-operator machine-config-daemon-m82gz 2/2 Running 0 12h 10.0.0.2 okd-master-02 <none> <none>
openshift-machine-config-operator machine-config-daemon-qfc82 2/2 Running 0 12h 10.0.0.1 okd-master-01 <none> <none>
openshift-machine-config-operator machine-config-daemon-vwh4d 2/2 Running 0 11h 10.0.0.4 okd-worker-01 <none> <none>
openshift-machine-config-operator machine-config-operator-c98bb964d-5vnww 1/1 Running 0 11h 10.100.0.21 okd-master-01 <none> <none>
openshift-machine-config-operator machine-config-server-g75x5 1/1 Running 0 12h 10.0.0.2 okd-master-02 <none> <none>
openshift-machine-config-operator machine-config-server-kpwqb 1/1 Running 0 12h 10.0.0.3 okd-master-03 <none> <none>
openshift-machine-config-operator machine-config-server-n9q2r 1/1 Running 0 12h 10.0.0.1 okd-master-01 <none> <none>

Let’s check the logs of the machine-config-daemon pod in the worker node.This pod has two containers, machine-config-daemon, and oauth-proxy.

kubectl logs -f machine-config-daemon-vwh4d -n machine-config-operator

Now, it is possible to see the actual error in the container execution:

I0816 06:58:42.985762 3240 update.go:283] Checking Reconcilable for config rendered-worker-a9681850fe39078ea0f42bd017922eb7 to rendered-worker-7131e04f110c489a0ad171e719cedc24
I0816 06:58:43.849830 3240 update.go:1403] Starting update from rendered-worker-a9681850fe39078ea0f42bd017922eb7 to rendered-worker-7131e04f110c489a0ad171e719cedc24: &{osUpdate:false kargs:false fips:false passwd:false files:true units:false kernelType:false}
I0816 06:58:43.852961 3240 update.go:1403] Update prepared; beginning drain
E0816 06:58:43.911711 3240 daemon.go:336] WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-48g5s, openshift-dns/dns-default-h2lt5, openshift-image-registry/node-ca-9z9zt, openshift-machine-config-operator/machine-config-daemon-vwh4d, openshift-monitoring/node-exporter-m5p2n, openshift-multus/multus-lnsng, openshift-sdn/ovs-5xzqs, openshift-sdn/sdn-vplps
.
.
.
I0816 06:58:43.918261 3240 daemon.go:336] evicting pod openshift-ingress/router-default-796df5847b-9hxzx
E0816 06:58:43.928176 3240 daemon.go:336] error when evicting pod "router-default-796df5847b-9hxzx" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0816 07:08:44.981198 3240 update.go:172] Draining failed with: error when evicting pod "router-default-796df5847b-9hxzx": global timeout reached: 1m30s, retrying
E0816 07:08:44.981273 3240 writer.go:135] Marking Degraded due to: failed to drain node (5 tries): timed out waiting for the condition: error when evicting pod "router-default-796df5847b-9hxzx": global timeout reached: 1m30s

The log shows that the machine config operator failed to drain the worker node before applying theconfiguration and executing the restart, as the router-default pod was not able to berescheduled in another node. Not being able to schedule again this podviolates the pod's disruption budget, thus, the operator is now degraded.

Let’s check the router-default pod status:

kubectl get pod -o wide --all-namespaces | grep "router-default"

It is possible to see that the pod is pending to be scheduled.

openshift-ingress router-default-796df5847b-9hxzx 1/1 Running 0 12h 10.0.0.4 okd-worker-01 <none> <none>
openshift-ingress router-default-796df5847b-h8bm4 0/1 Pending 0 12h <none> <none> <none> <none>

Let’s check it’s status:

oc describe pod router-default-796df5847b-h8bm4 -n openshift-ingress

Now, it is possible to confirm that the pod is Pending as there is not any available node to schedule it back again.

Name: router-default-796df5847b-h8bm4
Namespace: openshift-ingress
.
.
.
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning FailedScheduling <unknown> default-scheduler 0/4 nodes are available: 1 node(s) were unschedulable, 3 node(s) didn't match node selector.

We check the nodes status

oc get nodes

Again, it is possible to see that the MachineConfig operator tried to drain the node but it failedwhen rescheduling its pods.

NAME STATUS ROLES AGE VERSION
okd-master-01 Ready master 12h v1.18.3
okd-master-02 Ready master 12h v1.18.3
okd-master-03 Ready master 12h v1.18.3
okd-worker-01 Ready,SchedulingDisabled worker 12h v1.18.3

Why did this happen?

Master nodes are not schedulable to handle workloads, this was configured when the cluster was deployed.So pretty much the operator didn’t have enough room to reschedule the pods in other nodes.

The benefits of this approach (using the MachineConfig operator) are uncountable as the operator is smart enough toavoid services breaking when it finds that a configuration change is not able toget the system back to a consistent state.

But, not everything is as perfect as it sounds…

Ignition files are used to apply these configuration changes and it’s json representation is not human-readable at all, for this we use Fedora CoreOS Configuration (FCC) files in YAML format, then internally these yaml files are converted into an Ignition (JSON) file by the Fedora CoreOS Config Transpiler, which is a tool that produces a JSON Ignition file from the YAML FCC file.

There is a huge limitation in the resources that can be defined, it is only supported storage, system.d services, and users, so, for executing anything the user will have to render a script that must be called once by a systemd service after the node restarts. This, after using for many many years technologies like Ansible, Puppet, or Chef, it looks like a hacky and dirty approach for users to apply their custom configurations.

Another thing, debugging, if there is a problem with your MachineConfig object you might see only this degraded state, forcing you to dig into the containers logs and hopefully find the source of any issue you might have.

I believe there is a lot of room for improvements in the MachineConfig operator, I would love to see an Ansible interface to be able to plug-in my configuration changes by the openshift-machine-config-operator pod. Also, it was showed that the operator improves the system’s resiliency by impeding that a configuration change breaks what we defined as a cluster’s consistent state.

The easiest and fastest way to deploy an OKD 4.5 cluster in a Libvirt/KVM host

Carlos Camacho — Fri, 31 Jul 2020 00:00:00 +0000

Long story short… A single command to deploy an OKD 4.5 cluster in ~30 minutes (3 controllers, 1 to 10 workers, 1 service, and 1 bootstrap node), forget about following endless and outdated documentation.

I wrote so much automation in the meantime I worked/learned/practiced in OpenStack/RHOSP/Kubernetes/Openshift/OKDin the last 2Y, but suddenly I “lost” the machine where I hosted all these valuable code snippets.

With all this… I had to quickly invest some time to put together all that code. The first part is related to K8s/OKDand I created a small project called KubeInit “The KUBErnetes INITiator” to share it with the world.

The first (and only for now) playbook will deploy in a single command a fully operational OKD 4.5 cluster with 3master nodes, 1 compute nodes (configurable from 1 to 10 nodes), 1 services node, and 1 dummy bootstrap node. The services node has installed HAproxy, Bind, Apache HTTPd, and NFS to host some of the external required cluster services.

Requirements

A huge amount of RAM (the smallest amount I was able to deploy is with 64GB with a smaller configuration), configure this in the inventory file.
Be able to log in as root in the hypervisor node.
Reach the hypervisor node using the hostname nyctea you can change this in the inventory or add an entry in your /etc/hosts file.

That’s it, super simple…

git clone https://github.com/kubeinit/kubeinit.git
cd kubeinit
ansible-playbook \
    --user root \
    -v -i ./hosts/okd/inventory \
    --become \
    --become-user root \
    ./playbooks/okd.yml

You should get something like:

[ccamacho@wakawaka kubeinit]$ time ansible-playbook \
                                  --user root \
                                  -i ./hosts/okd/inventory \
                                  --become \
                                  --become-user root \
                                  ./playbooks/okd.yml

Using /etc/ansible/ansible.cfg as config file
PLAY [Main deployment playbook for OKD] ********************************************
TASK [Gathering Facts] *************************************************************
ok: [hypervisor-01]
.
.
.
"NAME STATUS ROLES AGE VERSION",
"okd-master-01 Ready master 16m v1.18.3",
"okd-master-02 Ready master 15m v1.18.3",
"okd-master-03 Ready master 12m v1.18.3",
"okd-worker-01 Ready worker 6m12s v1.18.3"
]}]}}

PLAY RECAP *************************************************************************
hypervisor-01: ok=83 changed=39 unreachable=0 failed=0 skipped=6 rescued=0 ignored=3   

real 33m49.483s
user 2m30.920s
sys 0m19.678s

A ready to use OKD 4.5 cluster in ~30 minutes, yeah!

What you can do now is log-in into your hypervisor and check the cluster status from the service machine.

ssh root@nyctea
ssh root@10.0.0.100
# This is now the service node (check the Ansible inventory for IPs and other details)
export KUBECONFIG=~/install_dir/auth/kubeconfig
oc get pvc -n openshift-image-registry
oc get pv
oc get clusteroperator image-registry
oc get nodes

The root password of the services machine is defined as a variable in the playbook, but the public key of the hypervisor root user is deployed across all the cluster nodes, so, you should be able to connect to any node from the hypervisor machine using certs-based authentication. Connect as the root user for the services machine (because is CentOS based) or as the core user to any other node (CoreOS based), using the IP addresses defined in the inventory file.

There is some reasoning for this password-based access to the services node. Sometimes we need to connect to the services machine when we deploy for debugging purposes, in this case, if we don’t set a password for the user we won’t be able to login using the console. Instead, for all the CoreOS nodes, once they are bootstrapped correctly/automatically there is no need to log-in using the console, just wait until they are deployed to connect to them using SSH.

The code is not perfect by any means but is a good example of how to use a libvirt host to run your OKD cluster and it’s incredibly easy to improve and add other roles and scenarios.

Next steps, I’ll clean all the lint nits around…

This is the GitHub repository https://github.com/kubeinit/kubeinit/.

Please if you like it, add some comments, test it, use it, hack it, break it, or become a stargazer ;)

Badgeboard - GitHub actions, where is my CI dashboard!

Carlos Camacho — Wed, 04 Dec 2019 00:00:00 +0000

A widely used term in the agile world is the information radiator, which refers to display the project’s critical information as simple as possible. These information radiators improve the team’s communication by amplifying pieces of data to get a better notion of self-awareness.

TL;DR;

If you just want to go straight to the solution of how to convert SVG badges to a widget-based CI dashboard, just go to the Badgeboardrepository or open the demo.

Otherwise, continue reading.

If you are beginning to apply agile methodologies in your team, a good information radiator can be for example a CI status dashboard.

The purpose of this information radiators, as the name implies, is to radiate information. It is something that people know about it and can see it easily. Keep in mind that a good information radiator will adapt to the needs of the project throughout its life, so try not to invest too much time in its initial design, and make sure that it can be easily changed/fixed/used/improved.

Some features of these information radiators:

It reflects the now : Information radiators always show what is going on (if things are going north or south). They help us see what matters now to the team and what to focus on in most of the cases when we hit i.e. regressions.
Minimum maximum value information : Simple and highly valuable. The more information, the less focus on important information, and more effort to maintain the panel.
Must be alive : This information artifact should be updated each time. As soon as reality changes, the artifact status should also change.

CI Dashboards

CI dashboards are a graphical representation of the continuous integration test results, usually HTML based and displaying in colors (red, yellow and green) the actual tests running results.

GitHub badges

We can see the status badges as a brief summary of the CI pipeline status. Badges1 are a unified way to present condensed pieces of information about your projects. They are also considered as any visual token of achievement, affiliation, authorization, or other trust relationship.

They consist of a small image and additionally a URL that the image points to. Examples for badges can be the pipeline status, test coverage, or ways to contact the project maintainers.

What now?

We introduce a tool to convert SVG badges to CI dasboards (Badgeboard).

GitHub actions -> No CI dashboard by default :(

I really liked the big dashboard view printed on a big screen so everyone can see it in a quick and easy manner. So, if we start using i.e. GitHub actions we lose the ability to have this graphical representation towards a badge based view.

Yehi!!! Here we have badgeboard!!!

Badgeboardis an awesome information radiator to show the status of the badges you have in your project as a widget-based dashboard, in particular it’s the main CI dashboard ofPystol.

Is a very simple tool that converts the information inside any SVG badge you define from any source in a widget-based dashboard.

Demo

Just open the index.htmlfile and see how the dashboard is rendered.

Requirements

None! Just clone the repo and open the index.html file in your favorite browser.

Once you have a copy, make the adjustments to the configuration file located in assets/data_source/badges_list.js to use your own badges.

Note: Due to CORS restrictions, badgeboard uses a proxyto add cross-origin headers when building the widgets panel. Check additional information about the CORS proxy on NPM.

How it works

We capture the badges list (SVG files) and we read the color information from a single pixel, from there, depending on the color of the pixel the widget is painted with its corresponding color.

This would be the usual view of the project badges.

Adding your badges and colors

Use the coordinates_testing.html file to determine based on the SVG coordinates the RGB color to be used in the JS configuration file.

To do so, copy the link to your badge, find the badge example in the file, replace it with yours, open the file in a browser, get the console logs and move around the mouse over the badge to see the coordinates and the RBG color that matches it.

Adding custom color badges

To add new colors, edit the assets/css/custom.css file and add new color definitions for the widgets. Once you define the new color, in the configuration file called assets/data_source/badges_list.js use the new color like in the following example.

colors:[['<new_color_definition','<matching_rgb_from_the _badge>'],['status-good','48,196,82']],

Troubleshooting

If the board does not render correctly (No widgets at all) it’s for sure that you refreshed too many times the page. We use a CORS proxy to add cross-origin headers when building the widgets panel.

The requests it can handled are limited in order to avoid crashing the container, so we can all use it.

Please read the requirements and use your own NPM proxyso these restrictions go away.

References

We use both smashingand gridsterto create the dashboard and its widgets.

License

Badgeboard is part of Pystoland Pystol is open source software licensed under the Apache license.

Next steps

It would be awesome to get some feedback around the tool, so, please feel free to file issues, pull requests or comments in this post or in the Badgeboardrepository.

List of TO-DOs

There are still some bits to fix in Badgeboardfor example:

~~Make the link from the widgets to work.~~
Move common hardcoded bits into variables for an easier update.
Improve the documentation.

Updated 2019/12/04: Initial version.

Oil painting and Minikube - Installing Minikube in Centos 7

Carlos Camacho — Sun, 13 Oct 2019 00:00:00 +0000

Today I got some time to do some oil painting and reading about techy stuff :)

This post is a brief summary of the deployment steps for installing Minikube in a Centos 7 baremetal machine, and, to show you my painting (check the fedora!).

The following steps need to run in the Hypervisor machine in which you will like to have your Minikube deployment.

You need to execute them one after the other, the idea of this recipe is to have something just for copying/pasting.

The usual steps are:

01 - Prepare the hypervisor node.

Now, let’s install some dependencies. Same Hypervisor node, same root user.

# In this dev. env. /var is only 50GB, so I will create
# a sym link to another location with more capacity.
sudo mkdir -p /home/libvirt/
sudo ln -sf /home/libvirt/ /var/lib/libvirt

sudo mkdir -p /home/docker/
sudo ln -sf /home/docker/ /var/lib/docker

# Install some packages
sudo yum install dnf -y
sudo dnf update -y
sudo dnf groupinstall "Virtualization Host" -y
sudo dnf install libvirt qemu-kvm virt-install virt-top libguestfs-tools bridge-utils -y
sudo dnf install git lvm2 lvm2-devel -y
sudo dnf install libvirt-python python-lxml libvirt curl-y
sudo dnf install binutils qt gcc make patch libgomp -y
sudo dnf install glibc-headers glibc-devel kernel-headers -y
sudo dnf install kernel-devel dkms bash-completion -y
sudo dnf install nano wget -y
sudo dnf install python3-pip -y

02 - Check that the kernel modules are OK.

# Check the kernel modules are OK
sudo lsmod | grep kvm

03 - Enable libvirtd, disable SElinux xD and firewalld.

# Enable libvirtd
sudo systemctl start libvirtd
sudo systemctl enable libvirtd

# Disable selinux & stop firewall as needed.
setenforce 0
perl -pi -e 's/SELINUX\=enforcing/SELINUX\=disabled/g' /etc/selinux/config
systemctl stop firewalld
systemctl disable firewalld

04 - Install Minikube.

#Install minikube
/usr/bin/curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube
cp -p minikube /usr/local/bin && rm -f minikube

# Create the repo for kubernetes
cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

# Install kubectl
sudo dnf install kubectl -y
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

05 - Create the toor user (from the Hypervisor node, as root).

sudo useradd toor
echo "toor:toor" | sudo chpasswd
echo "toor ALL=(root) NOPASSWD:ALL" \
  | sudo tee /etc/sudoers.d/toor
sudo chmod 0440 /etc/sudoers.d/toor
sudo su - toor

cd
mkdir .ssh
ssh-keygen -t rsa -N "" -f .ssh/id_rsa

Now, follow as the toor user and prepare the Hypervisor node for Minikube.

06 - Install Docker.

We will like to also use docker in the Hypervisor node for creating images and debugging purposes.

# Install docker
sudo dnf install docker -y
sudo usermod --append --groups dockerroot toor
sudo tee /etc/docker/daemon.json >/dev/null <<-EOF
{
    "live-restore": true,
    "group": "dockerroot"
}
EOF
sudo systemctl start docker
sudo systemctl enable docker

07 - Finish the Minikube configuration.

# Add to bashrc in toor user
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

# We add toor to the libvirtd group
sudo usermod --append --groups libvirt toor

08 - Start Minikube.

minikube start --memory=65536 --cpus=4 --vm-driver kvm2
export no_proxy=$no_proxy,$(minikube ip)
nohup kubectl proxy --address='0.0.0.0' --port=8001 --disable-filter=true &
sleep 30
minikube addons enable dashboard
nohup minikube dashboard &
minikube addons open dashboard

The Minikube instance should be reachable from the following URL:

http://machine\_ip:8001/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/

# To stop/delete
kubectl delete deploy,svc --all
minikube stop
minikube delete

09 - Minikube cheat sheet.

# set & get current context of cluster
 kubectl config use-context minikube
 kubectl config current-context

# fetch all the kubernetes objects for a namespace
 kubectl get all -n kube-system

# display cluster details
 kubectl cluster-info

# set custom memory and cpu 
 minikube config set memory 4096
 minikube config set cpus 2

# fetch cluster ip
 minikube ip

# ssh to the minikube vm
 minikube ssh

# display addons list and status
 minikube addons list

# exposes service to vm & retrieves url 
 minikube service elasticsearch
 minikube service elasticsearch --url

Updated 2019/10/13: Initial version.

Updated 2019/10/15: Install also docker in the hypervisor.